SolarThing Web Enhacements

SolarThing Web? What are you talking about? Well, conveniently attached to the SolarThing GraphQL program (AKA SolarThing Server), is a web interface. At the moment, it is very basic, but it now has some features worth talking about. Note that this is different than the Legacy solarthing-web project, but does include the "classic" view from that project.

SolarThing Web

• Added a navigation bar
• Added /classic endpoint
• Added /authorized endpoint
  • You can authorize and deauthorize senders

Release Process Changes

• Moving forward, artifacts being attached to releases will be attached by GitHub actions. This will automate things and could even help sign jars in the future for extra security. Documentation PDFs and epubs will also be published, but it is recommended to go to https://solarthing.readthedocs.io for the latest documentation
• EDIT: solarthing-2022.3.1.jar was not automatically updated, as I had to manually fix it ☹️

Small updates with minor breaking changes

This release has a bunch of small changes from the past few months.

Breaking changes

• run subcommand is now required when running the jar file. No one should notice this change
• Some of the old send commands are no longer supported such as sendcommand. You must now use sendopen.

GraphQL

• Added a React frontend that is served on the root path. This is a work in progress, but for now it displays the battery voltage for the default source ID
• It's worth mentioning that the GraphQL program is now more of a server program for SolarThing. Expect the name of this program to change in the future to something like "SolarThing Server"

SolarThing Check

• The solarthing check command now has a --debug flag to spit out more information. This will help with issues like #55.

Other changes (part of the repo, not jars)

• When compiling yourself, jar files will be put in one of two spots, which will help avoid writing over the jar file being used to run SolarThing when developing

Command receiving changes, removed legacy CouchDB view,

This release has many small changes

All

• Packet IDs default to using a shorter variant

Modbus (rover and tracer)

• Added configurable initial_timeout and end_timeout parameters

Commands

• Command packet collections must only have 3 packets: LARGE_INTEGRITY, SOURCE, FRAGMENT_INDICATOR
• Upon the receiving of a command, an accept or reject packet will be uploaded to the events database

GraphQL

• Added battery record caches
• Subsequent queries that are identical to currently processing queries will wait on the result of the query in progress

Automation

• Deprecated sendcommand and sendrequestheartbeat
• Scheduling commands now allows "and" to be present in relative duration requests
• The alter manager action requires a fragment ID to be provided. This is a breaking change to configs if anyone was using this feature.

CouchDB

• The setup program will now no longer create the millis view, which is used by old versions of SolarThing
• The manager user will have permission to upload to solarthing_events.

Tracer and Rover events, Better event reupload, GraphQL program logging

This release has a lot of small changes that will go unnoticed.

New documentation available on https://solarthing.readthedocs.io. A PDF of the documentation will also be attached to each release for preservation of the state of the documentation at a given time. However you should use https://solarthing.readthedocs.io for up to date documentation usually.

All

• Fixed CouchDB uploading bug that caused one or two errors to be spit out every time a new document ID was generated (happens about once or twice every 2 minutes depending on what your unique is set to)

Rover

• rover charging state change event
• rover error mode change event

Tracer

• Charging equipment status change event

GraphQL

• Logs are now written to the logs directory. You should use less -R to view those logs, as they will contain color and do not look good when viewed with certain editors.

Automation

• Added heartbeat command to chatbot
• Internally added ResourceManager

Internal changes:

• Got rid of InstantType.
• Reuploading event packets is not reliant on new events being processed.
• More debugging around mate packets being ignored for being stale or duplicated

Better CLI Experience, Security Updates

The biggest change in this release is the addition of the solarthing check command. The command line experience is also more complete with the addition of solarthing version. SolarThing will also log the date that the jar file was last modified.

Updated to log4j version 2.17.0 and many other version bumps.

The solarthing check command can be used to scan for all types of devices.

Security Fix, withlock action, flags, graphql alter query, rest command/run endpoint

Security Fix: This release has an important security fix described in #49. It is recommended everyone upgrade, although it is unlikely the vulnerability can be exploited for most SolarThing set ups.

All

• millisNull view is (in automation program and in graphql program)
  Actions
• Added a withlock action, which is recommended whenever you are dealing with an action that should never parallel itself.
  • As of right now, many actions that using locks in config_templates have bugs in them that cause them to unlock even when there's already a lock on them
• log action can send to the summary log if desired
• Flags are a new feature that can be set using an authorized client (currently only slack chat bots). Flags can be utilized in actions for checking if boolean configuration values are set/unset.
• New heartbeat packets. This is a good way to make sure commands stay working. You can set up alerts with the message-sender action part of the automation program

GraphQL/REST Program

• New GraphQL queries for alter database related stuff
• /commend/run endpoint for executing actions. An API key is required. You can make your own API keys and put them in the configuration
  • This feature is undocumented as of now. I hope to add documentation for it in the future.
  • This feature sets CORs headers that allow it to be called from a browser.

Alter Database, CouchDB Setup Fixes, Feedback Actions, GraphQL program at Java 11

Update 2021-12-11: This release (and previous releases) contains the CVE-2021-44228 vulnerability. See #49 for more info.

This release has a LOT of changes. Many will go unnoticed, and many new features will remain undocumented until I get around to well, documenting them.

These changes likely won't mean anything for most users of SolarThing, so updating likely will not fix any bugs or add any new featuers, however I always recommend updating so that if there is a problem, you can let me know!

If you aren't going to update, that's OK, however I do recommend that you run the CouchDB setup program again: instructions. If you have already ran CouchDB setup once before, give it the same usernames as before, and it will not prompt to create a new user. Running the CouchDB program again will make your database more secure from unauthorized changes.

Alter:

• Added alter database
• Scheduled commands now work

Automation

• Added action to handle the alter database
• Slack chatbot
  • Can now request to schedule commands and cancel those commands

Commands:

• The JSON of requesting commands has some additions. Notably, the execution reason, which will commonly contain a source from the open database
• While a command is being executed, an action can now send feedback to solarthing_events to indicate that something has happened.
• Clients that are requesting commands must be updated in order to work. The expected (root) dateMillis and encrypted dateMillis must now be the same in order for a command to be processed.
• Even unauthorized commands are now only processed once, instead of being processed repeatedly during the 5 minutes they are queried.

GraphQL

• Bug fix for CouchDB queries that affected solcast data queries

Internal Changes:

• GraphQL program requires Java 11
• Updated to gradle 7

Slack chatbot, better GraphQL solcast, removed convenience fields

This release contains a bunch of stuff that will be left undocumented for a while probably.

This release adds a chatbot, which currently can only be used with slack. There is no documentation on how to set this up, but it is an action that you put in a config for solarthing-automation.

All

• Option to shorten document ID. This helps a tiny bit to reduce the amount of storage CouchDB uses.
• [Action] Added wait action, which takes an ISO duration rather than milliseconds
• Added a way to start a server and send log messages to anyone netcatting the server. This functionality is off by default.

Mate:

• FX and MX packets no longer contain convenience fields. This is to help reduce the amount of space CouchDB uses, as we don't actually need to store the name of the mode these devices are in.

GraphQL:

• Solcast query now uses actual past data rather than an API call for past data.

Bug Fixes:

• PVOutput identifier selection is no longer broken
• Available commands packets no longer are added if there are no preceding packets

Tracer release

SolarThing now supports the EPEver Tracer charge controller!

All:

• Fixed bug when retrieving authorized users for commands
• If you use the SNAPSHOT program and update while SolarThing is running, SolarThing may crash. If it does, it will automatically restart assuming the new systemd service is installed.

Request

• Added tracer support

Mate:

• The IO now automatically tries to reload if it fails. --This is mostly untested

CouchDB Setup

• solarthing_cache database is automagically created

Rover

• Added undocumented configuration option to create a server that can be connected to by a simple client such as netcat (nc)
  • This is also part of the new tracer feature and will be documented in the future

Rover program changes to request

This release makes using the rover program obsolete. Although existing configurations will continue to work as before, it is recommended to eventually upgrade your rover program to use the request program instead. The rover quickstart now shows an up to date way on how to configure the request program to be identical to your old rover configuration.

These changes also make it possible to monitor multiple rovers in the same program. If this is useful to anyone, please make an issue on the issues page and I will update the documentation for it.

The rover-setup program still exists. If you want to use the "dummy" rover configuration, you now must make a "dummy" io configuration. A base template for that is provided in the config_templates/io directory.

Rover:

• New way to configure using the request program

GraphQL

• Undocumented /cache endpoint has been added, making the program more of a "RESTful" program. This along with solarthing_cache may by updated with documentation in the future

Message:

• Mattermost is no longer supported. If anyone was actually using this, please let me know. It's pretty easy to add back.

I realize that documentation may be lacking for a few of the added features, so if you find anything in particular lacking, please make an issue about it!