Add script to rotate encryption keys, update README with docs on rota…

…ting keys (#54) * feat: Add script to rotate encryption keys Signed-off-by: Eamonn Mansour <47121388+eamansour@users.noreply.github.com> * docs: Add docs on how to rotate encryption keys Signed-off-by: Eamonn Mansour <47121388+eamansour@users.noreply.github.com> * feat: Add logs to log file and more error handling Signed-off-by: Eamonn Mansour <47121388+eamansour@users.noreply.github.com> * docs: Update encryption keys rotation in the README Signed-off-by: Eamonn Mansour <47121388+eamansour@users.noreply.github.com> * docs: Add recommendation to backup secrets Signed-off-by: Eamonn Mansour <47121388+eamansour@users.noreply.github.com> * fix: Change galasactl-dev to galasactl Signed-off-by: Eamonn Mansour <47121388+eamansour@users.noreply.github.com> * docs: Remove unused flag from rotate-encryption-keys script Signed-off-by: Eamonn Mansour <47121388+eamansour@users.noreply.github.com> * docs: Add manual steps to rotate encryption keys Signed-off-by: Eamonn Mansour <47121388+eamansour@users.noreply.github.com> --------- Signed-off-by: Eamonn Mansour <47121388+eamansour@users.noreply.github.com>
galasa-dev · Nov 4, 2024 · e910832 · e910832
1 parent 3cc6b5d
commit e910832
Show file tree

Hide file tree

Showing 2 changed files with 477 additions and 1 deletion.
diff --git a/README.md b/README.md
@@ -202,9 +202,146 @@ If you want to upgrade the Galasa Ecosystem to use a newer version of Galasa, fo
 
 ```console
 helm repo update
-helm upgrade <release-name> galasa/ecosystem --reuse-values --set galasaVersion=0.33.0 --wait
+helm upgrade <release-name> galasa/ecosystem --reuse-values --set galasaVersion=0.38.0 --wait
 ```
 
+### Rotating Encryption Keys
+
+To maintain the security of your Galasa Ecosystem, you may wish to replace the encryption key being used to encrypt credentials in the Galasa Ecosystem's credentials store with a new encryption key, and re-encrypt all your existing credentials using the new key.
+
+When the Galasa Ecosystem Helm chart is installed, a Kubernetes Secret is created which contains the base64-encoded, 256-bit encryption keys used to encrypt credentials stored in the Galasa Ecosystem's credentials store using AES-256-GCM encryption.
+
+The encryption keys are stored in the following YAML structure:
+```yaml
+encryptionKey: <base64-encoded-encryption-key>
+fallbackDecryptionKeys:
+- <base64-encoded-encryption-key>
+- <base64-encoded-encryption-key>
+```
+
+The `encryptionKey` key in the YAML entry represents the active base64-encoded, 256-bit encryption key used to encrypt credentials stored in the Galasa Ecosystem's credentials store.
+
+The `fallbackDecryptionKeys` list represents a list of base64-encoded encryption keys that are no longer in use, and allows for encryption keys to be rotated without losing previous encryption keys. This allows encrypted credentials to be decrypted with a fallback decryption key and then be encrypted using the newly activated encryption key.
+
+**Before proceeding to rotate encryption keys, it is highly recommended to make a backup of the existing credentials stored in your Galasa Ecosystem by running the following command using the Galasa CLI tool:**
+
+```console
+galasactl secrets get --format yaml > /path/to/backup/file.yaml
+```
+
+**where `/path/to/backup/file.yaml` is either an absolute or relative path of your choice to a file where the backup will be stored.**
+
+#### Prerequisites
+
+The following command-line utilities must be installed:
+
+- kubectl (v1.30.3 or later)
+- galasactl (0.38.0 or later)
+- openssl (3.3.2 or later)
+
+You must have the correct permissions to read and update Kubernetes Secrets in the Kubernetes namespace that you have installed your Galasa Ecosystem within.
+
+You must also have a valid personal access token for your Galasa Ecosystem set on your machine so that you can retrieve and update secrets with `galasactl`.
+
+#### Automated steps
+
+For Linux and macOS, you can download and run the [`rotate-encryption-keys.sh`](./rotate-encryption-keys.sh) script via the command-line to simplify the process of rotating encryption keys and re-encrypting credentials.
+
+The following flags can be supplied when running the script:
+- `--release-name <name>` **Required**. The helm release name provided when installing the Galasa Ecosystem helm chart
+- `--namespace <namespace>` Optional. The Kubernetes namespace where your Galasa Ecosystem is installed
+- `--bootstrap <bootstrap-url>` Optional. The bootstrap URL of the Galasa Ecosystem that is being serviced. Not required if the `GALASA_BOOTSTRAP` environment variable is set and is pointing to the correct bootstrap URL. Overrides the existing `GALASA_BOOTSTRAP` environment variable value if set
+
+For example:
+
+```console
+./rotate-encryption-keys.sh --release-name example --namespace galasa-dev
+```
+
+The `rotate-encryption-keys.sh` script will automatically update the current encryption key with a new one, and then restart your Galasa Ecosystem's API and engine controller pods so that they can pick up the new encryption key. After rotating the encryption keys, the script will re-encrypt the existing secrets in your Galasa Ecosystem using the newly activated encryption key.
+
+Once the encryption keys have been rotated and the existing secrets have been re-encrypted, the script will clear the fallback decryption keys list and restart the API and engine controller pods for a final time to keep the Galasa services in sync with the contents of the encryption keys secret.
+
+#### Manual steps
+
+If the `rotate-encryption-keys.sh` script fails or if you are on Windows and cannot run the script, you can manually rotate encryption keys by performing the following steps.
+
+This guide assumes `kubectl` is configured to manage resources in the namespace that your Galasa Ecosystem is installed within, so the `--namespace` flag is not required when running `kubectl` commands.
+
+This guide also assumes that the `GALASA_BOOTSTRAP` environment variable is set, so the `--bootstrap` flag is not required when running `galasactl` commands.
+
+1. Get all the existing secrets from your Galasa Ecosystem by running:
+   ```
+   galasactl secrets get --format yaml
+   ```
+   Store these secrets in a `.yaml` or `.yml` file so that you can re-encrypt them after rotating the encryption keys
+
+2. Find the name of the Kubernetes Secret containing your Galasa Ecosystem's encryption keys by running:
+   ```
+   kubectl get secrets
+   ```
+   The secret's name should be of the form `{release-name}-encryption-secret`, where `{release-name}` is the Helm release name provided when installing the Galasa Ecosystem Helm chart
+
+3. Get the existing encryption keys data for your Galasa Ecosystem by running:
+   ```
+   kubectl get secret {encryption-secret-name} --output jsonpath='{ .data.encryption-keys\.yaml }' | openssl base64 -d -A
+   ```
+   where `{encryption-secret-name}` is the name of the Kubernetes secret retrieved in step 2. The output should look like the following:
+   ```yaml
+   encryptionKey: <existing-encryption-key>
+   fallbackDecryptionKeys: []
+   ```
+   Place the output into a file
+
+4. Generate a new encryption key by running:
+   ```
+   openssl rand -base64 32
+   ```
+
+5. In the file created at the end of step 3, move the existing `encryptionKey` value into the `fallbackDecryptionKeys` list and place the newly generated encryption key into the `encryptionKey` field. The file should now look like this:
+   ```yaml
+   encryptionKey: <newly-generated-encryption-key>
+   fallbackDecryptionKeys:
+   - <existing-encryption-key>
+   ```
+   where `<newly-generated-encryption-key>` is the new encryption key generated in step 5 and `<existing-encryption-key>` is the old encryption key retrieved in step 3
+
+6. Base64-encode the file contents by running the following command, where `<encryption-keys-file>` is an absolute or relative path to the file created in step 3 (and modified in step 5):
+   ```
+   openssl base64 -in <encryption-keys-file>
+   ```
+   Record the base64-encoded output, making sure there are no spaces or line breaks in the recorded output
+
+7. Update the existing Kubernetes Secret with the rotated keys by running:
+   ```
+   kubectl patch secret {encryption-secret-name} --type='json' -p="[{'op': 'replace', 'path': '/data/encryption-keys.yaml', 'value': '<base64-encoded-encryption-keys>'}]"
+   ```
+   where `{encryption-secret-name}` is the name of the Kubernetes secret retrieved in step 2, and `<base64-encoded-encryption-keys>` is the output recorded from step 6
+
+8. Restart the Galasa Ecosystem's API server deployment by running:
+    ```
+    kubectl rollout restart deployment {release-name}-api
+    kubectl rollout status deployment {release-name}-api
+    ```
+    where `{release-name}` is the name of the Helm release provided when installing the Galasa Ecosystem Helm chart
+
+9. Restart the Galasa Ecosystem's engine controller deployment by running:
+    ```
+    kubectl rollout restart deployment {release-name}-engine-controller
+    kubectl rollout status deployment {release-name}-engine-controller
+    ```
+    where `{release-name}` is the name of the Helm release provided when installing the Galasa Ecosystem Helm chart
+
+10. Once both the API server and engine controller have been restarted successfully, you can re-encrypt your existing secrets using the YAML file you created in step 1, by running:
+    ```
+    galasactl resources apply -f <secrets-yaml-file>
+    ```
+    where `<secrets-yaml-file>` is an absolute or relative path to the YAML file created at the end of step 1
+
+Your Galasa Ecosystem will now use the newly generated encryption key to encrypt and decrypt secrets until the next time it is rotated.
+
+To verify that your secrets can still be read correctly, you can run `galasactl secrets get --format yaml` again and compare the YAML output with the content of YAML file that you applied in step 10. If the output is the same, then the secrets have been re-encrypted successfully.
+
 ### Development
 To install the latest development version of the Galasa Ecosystem chart, clone this repository and update the following values in your [values.yaml](charts/ecosystem/values.yaml) file: