Setup Fly.io

.dockerignore

@@ -8,3 +8,6 @@ docassemble-os/**
 target/**
 bin/**
 *.hprof
+*.local
+*.log
+*.zip

.github/workflows/fly.yml

@@ -0,0 +1,21 @@
+# Example fly.yml
+# You can use this as a template for the Fly.io continuous deployment GitHub workflow config file.
+# Copy into .github/workflows/fly.yml for GitHub to see this file.
+# For more details, check out:
+# https://fly.io/docs/app-guides/continuous-deployment-with-github-actions/
+name: Fly Deploy
+on:
+  push:
+    branches:
+      - main
+jobs:
+  deploy:
+    name: Deploy app
+    runs-on: ubuntu-latest
+    concurrency: deploy-group    # optional: ensure only one action runs at a time
+    steps:
+      - uses: actions/checkout@v4
+      - uses: superfly/flyctl-actions/setup-flyctl@master
+      - run: flyctl deploy --remote-only
+        env:
+          FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}

.github/workflows/formatting.yml

@@ -7,9 +7,9 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - name: Set up JDK 17
-        uses: actions/setup-java@v3
+        uses: actions/setup-java@v4
         with:
           java-version: '17'
           distribution: 'adopt'

.gitignore

@@ -45,6 +45,7 @@ hs_err_pid*
 
 # Contains key info
 client_sign.properties
+client_sign.properties.local
 *.pfx
 .env*
 .da_env

Dockerfile

@@ -6,7 +6,7 @@ COPY pom.xml LICENSE client_sign.propertie[s] quartz.properties Suffolk.pf[x] ac
 RUN mvn -f /usr/src/app/pom.xml -DskipTests clean dependency:resolve dependency:go-offline package && mvn -f /usr/src/app/pom.xml test
 COPY src /usr/src/app/src
 RUN mvn -f /usr/src/app/pom.xml -DskipTests package dependency:build-classpath -Dmdep.outputFile=cp.txt -PnoDockerTests
-COPY docker_run_script.sh docker_integration_test.sh /usr/src/app/
+COPY docker_run_script.sh docker_integration_test.sh fly_startup_script.sh /usr/src/app/
 
 EXPOSE 9000
 

docker_run_script.sh

@@ -1,7 +1,14 @@
 #! /bin/sh
 
 set -ex
+
 cd /usr/src/app
+
+# Customize startup if running on Fly.io.
+if [ -n "$FLY_MACHINE_ID" ]; then
+  ./fly_startup_script.sh
+fi
+
 # Add this before the `-cp` line if needed to get exact SOAP envelopes being sent
 # -javaagent:extract-tls-secrets-4.0.0.jar=/tmp/secrets/secrets.log \
 # Add this when we figure out ThreadPools and Unmarshalling (see #111)

docs/adr/010-use-a-platform-and-database-as-services-to-improve-operations.md

@@ -0,0 +1,46 @@
+# Use a Platform and Database as Services to Improve Operations
+
+## Context
+
+The existing system is deployed to Lightsail. Both the Java application and Postgres database are Dockerized and run within the Lightsail VPS. Many tasks require connecting to the running server and executing commands in the shell.
+
+**User Story:**
+As a Suffolk LIT Lab operator, I want to improve the ease of operations in these key areas:
+* Data: recover from a possible data loss by restoring a previous version of the database or rolling back the database
+* App Stability: restart the app in case it crashes or becomes unavailable
+* Monitoring / Alerting: monitor the app and be alerted when there are issues with the API being available
+
+Ideally, this will lower the barrier to entry for others to run their own EfileProxyServer instances.
+
+## Decision
+
+Use the combination of a Platform-as-a-Service (PaaS) with a Database-as-a-Service (DBaaS), specifically Fly.io and Supabase. Both offerings have dashboards with good usability. You can view logs and key metrics (memory usage, CPU, error counts, query performance) through the web interface. You can also perform common operational tasks such as restarting a machine and restoring the database. Some key tasks, such as the daily database backup and security updates to the operating system, are handled automatically by the platform, reducing the manual operational load.
+
+## Alternatives Considered
+
+1. Scripting out common operations and using cron jobs with the existing Lightsail setup
+   * Pros
+     * Builds on existing familiarity with the system
+     * Smaller delta between the new process and the existing process
+   * Cons
+     * Would have to still handle many operational tasks manually
+     * More development work per operational task
+2. Postgres database replication
+   * Pros
+     * Adds resiliency with minimal modification
+   * Cons
+     * Only addresses the Data bullet point
+     * Would still require a decision on the replica database
+3. Using a different PaaS such as Heroku
+   * Pros
+     * Ease-of-use and more features out-of-the-box
+   * Cons
+     * Expensive if the application requires more than 2GB of memory
+     * Less optimized for Docker applications than Fly.io
+4. Using a different Database-as-a-Service (DBaaS) such as Heroku Postgres, AWS Aurora, or Neon
+   * Pros
+      * Similar operational characteristics
+   * Cons
+     * Supabase has the best combination of ease of use, pricing, and track record. All the other alternatives are beaten in one or more of those aspects in comparison  
+   * Notes
+     * This is an easy decision to change our minds on in the future, as any Postgres-compatible database can be swapped in by changing the POSTGRES_* variables in the .env and the data is quite portable with a variety of pg_dump/export/replication options supported by all the DBaaS options.

docs/adr/011-use-papertrail-to-increase-access-to-logs.md

@@ -0,0 +1,33 @@
+# Use Papertrail to Increase Access to Logs
+
+## Context
+
+Logs are viewable within Fly.io. Access to those logs is constrained to authorized users. However, this permission is not fine-grained and authorized users also have the power to perform administrative actions. 
+
+**User Story:**
+As a Suffolk LIT Lab operator, I want to share access to logs with stakeholders who can benefit from being able to self-diagnose any issues using the EFSP API. 
+
+## Decision
+
+Integrate Papertrail through the existing slf4j/logback stack as a log appender option, using TLS-encrypted syslog over TCP. If the Papertrail environment variables are set (see env.example for details), the logs will also be forwarded to Papertrail. Papertrail will serve as a central point of access for viewing the logs. The Papertrail UI makes it easy to filter and search through the logs. Being able to control log access independent of access to more admin operations within Fly.io makes for better security. Separating out the logging concern allows you to ship logs from other environments besides Fly.io as well. For example, the logs from Lightsail instances can be fed into the same dashboard. The current staging and production Fly.io instances will all forward to a single Papertrail dashboard and identified by the system and configurable application name.
+
+## Alternatives Considered
+
+1. Using the Fly.io log shipper
+  * Pros
+    * Closer to the platform
+  * Cons
+    * Requires having extra machines running the log shipping at all times
+    * For one-off runs, the usage is clunkier because output has to be redirected to /dev/console for it to be picked up 
+2. Observability platforms such as Splunk or Datadog
+  * Pros
+    * More complete offerings/advanced features
+  * Cons
+    * Much more expensive
+3. Using a token-based Papertrail appender
+  * Pros
+    * Better identification of the senders
+  * Cons
+    * Not the recommended option by Papertrail for Java apps
+    * Less performant
+    * Likely requires some Java code to implement the token appender