Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create reusable image deploy workflow #3

Open
wants to merge 43 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
c4a2bca
Add deploy image reusable workflow
kathy-t Nov 12, 2024
19772fc
Also build before deploying image
kathy-t Nov 12, 2024
c8cacde
Make dockerContext optional
kathy-t Nov 12, 2024
a223020
Upload image digest to quay repo folder
kathy-t Nov 13, 2024
99a0126
Try specifying token
kathy-t Nov 15, 2024
81681da
Revert "Try specifying token"
kathy-t Nov 15, 2024
328169a
Test inherited permissions
kathy-t Nov 15, 2024
ff4eaa8
Add back permissions
kathy-t Nov 15, 2024
47f3dc0
Move set up up
kathy-t Nov 15, 2024
5e496db
Move permissions out?
kathy-t Nov 15, 2024
03e4140
Modify deploy_tagged so it can figure out the changelist version
kathy-t Nov 18, 2024
61f01be
Simplify set folder name
kathy-t Nov 18, 2024
7be3ef1
Check CHANGELIST_VERSION
kathy-t Nov 18, 2024
7f30b40
Fix export env var
kathy-t Nov 18, 2024
9ad0263
Clean up
kathy-t Nov 18, 2024
ae29afc
Update to latest version
kathy-t Nov 19, 2024
b207552
Add deploy_snapshot
kathy-t Nov 22, 2024
a5299e7
Install git secrets
kathy-t Nov 22, 2024
b6fe7b8
Set maven settings.xml in deploy snapshot
kathy-t Nov 25, 2024
d97397b
Fix id and username
kathy-t Nov 25, 2024
52707c8
Do the same for tagged releases
kathy-t Nov 25, 2024
986880f
Fix deploy command
kathy-t Nov 25, 2024
6c8bbd9
Reusable maven deploy
kathy-t Nov 26, 2024
0787b01
Fix boolean
kathy-t Nov 26, 2024
819ec82
Oops
kathy-t Nov 26, 2024
b68ebe0
Fix again
kathy-t Nov 26, 2024
fcf2d6a
Try this
kathy-t Nov 26, 2024
b646684
Clean up
kathy-t Nov 26, 2024
81c1519
Fix quay repo
kathy-t Nov 26, 2024
c5bb846
Update names
kathy-t Nov 26, 2024
19dcc55
Try temurin
kathy-t Nov 26, 2024
dca27c2
Substitute slashes with dashses
kathy-t Nov 26, 2024
62abc54
Check semantic version tag
kathy-t Nov 26, 2024
27ef0e0
Upload and download artifacts
kathy-t Nov 26, 2024
e4c2997
Use v4
kathy-t Nov 26, 2024
c15245c
Set path
kathy-t Nov 26, 2024
dd138e9
Revert "Set path"
kathy-t Nov 27, 2024
cbb7880
Revert "Use v4"
kathy-t Nov 27, 2024
2970609
Revert "Upload and download artifacts"
kathy-t Nov 27, 2024
3e6de0b
Descriptive names
kathy-t Nov 27, 2024
a2e6a99
Change back to adopt
kathy-t Nov 27, 2024
e1f3131
Fix name
kathy-t Nov 27, 2024
09d95a4
Fix regex
kathy-t Nov 27, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
111 changes: 111 additions & 0 deletions .github/workflows/deploy_artifacts.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# Deploys artifacts using maven and optionally deploys an image to quay
name: Reusable artifacts deploy

on:
workflow_call:
inputs:
createDockerImage:
description: 'Whether to create a Docker image, caller must have Dockerfile in the repository'
required: false
type: boolean
default: false
quayRepository:
description: The quay repository to upload the image to. The repository must belong to the dockstore quay organization.
required: false
type: string
dockerContext:
description: The Docker context containing the Dockerfile of the image to build and push.
required: false
default: .
type: string

env:
IS_DEVELOP_SNAPSHOT: ${{ github.ref_type == 'branch' && github.ref_name == 'develop' }}

jobs:
deploy_maven:
name: Maven deploy ${{ github.ref_type == 'tag' && 'tagged' || 'snapshot' }} release
runs-on: ubuntu-22.04

steps:
- uses: actions/checkout@v4
# Step that does that actual cache save and restore
- uses: actions/cache@v4
with:
path: ~/.m2/repository
key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-

- name: install git secrets
run: |
wget --no-verbose -O git-secrets-1.3.0.tar.gz https://github.com/awslabs/git-secrets/archive/1.3.0.tar.gz
tar -zxf git-secrets-1.3.0.tar.gz
cd git-secrets-1.3.0
sudo make install

- name: Set up JDK
uses: actions/setup-java@v4
with:
java-version: '21.0.2+13.0.LTS'
distribution: 'adopt'
# settings.xml configuration
server-id: ${{ github.ref_type == 'tag' && 'central' || 'snapshots' }}
server-username: DEPLOY_USERNAME
server-password: DEPLOY_TOKEN
Comment on lines +52 to +55
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Defining this allows us to remove all the duplicated settings.xml files across the various repos that upload artifacts to artifactory.


- name: Set changelist version
run: |
set -x
if ${{ github.ref_type == 'tag' }}; then
# Check that the tag follows semantic versioning. Note that the regex is not super strict
if [[ ${{ github.ref_name }} =~ ^[0-9]+\.[0-9]+\.[0-9a-zA-Z-]+(-[0-9a-zA-Z-]+\.[0-9a-zA-Z-]+)*$ ]]; then
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This regex is not as strict as https://semver.org/#is-there-a-suggested-regular-expression-regex-to-check-a-semver-string because that might be overkill/hard to read... thoughts?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As long as it matches the patterns that appear in our refs, should be fine, imho

# Break up the semantic version tag by the '.' delimiter and return the third field onward. Prefix this with the '.'
# Ex: 1.16.0 -> .0 and 1.16.0-alpha.0 -> .0-alpha.0
CHANGELIST_VERSION=.$(echo ${{ github.ref_name }} | cut -d. -f 3-)
else
echo "Invalid semantic version for tag ${{ github.ref_name }}"
exit 1
fi
elif ${{ github.ref_name != 'develop' }}; then
CHANGELIST_VERSION=.0-${{ github.ref_name }}-SNAPSHOT
CHANGELIST_VERSION=${CHANGELIST_VERSION//\//-}
fi

echo "CHANGELIST_VERSION=${CHANGELIST_VERSION}" >> $GITHUB_ENV

- name: Store Maven project version
run: |
set -x
if ${{ env.IS_DEVELOP_SNAPSHOT }}; then
echo "maven_project_version=$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout)" >> $GITHUB_ENV
else
echo "maven_project_version=$(mvn help:evaluate -Dexpression=project.version -Dchangelist=${{ env.CHANGELIST_VERSION }} -q -DforceStdout)" >> $GITHUB_ENV
fi

- name: Read exported variable
run: |
echo "${{ env.maven_project_version }}"

- name: Deploy with mvnw
run: |
git config --global user.email "${{ github.actor }}"
git config --global user.name "${{ github.actor }}"

set -x
if ${{ env.IS_DEVELOP_SNAPSHOT }}; then
./mvnw --batch-mode deploy -ntp -DskipTests
else
./mvnw --batch-mode deploy -ntp -DskipTests -Dchangelist=${{ env.CHANGELIST_VERSION }}
fi
env:
DEPLOY_USERNAME: ${{ github.ref_type == 'tag' && 'dockstore-bot' || 'dockstore-snapshot-bot' }}
DEPLOY_TOKEN: ${{ github.ref_type == 'tag' && secrets.COLLAB_DEPLOY_TOKEN || secrets.SNAPSHOT_DEPLOY_TOKEN }}

deploy_image:
if: ${{ inputs.createDockerImage && inputs.quayRepository != '' }}
uses: dockstore/workflow-actions/.github/workflows/deploy_image.yaml@seab-6771/reusable-image-deploy
with:
quayRepository: ${{ inputs.quayRepository }}
dockerContext: ${{ inputs.dockerContext }}
secrets: inherit
85 changes: 85 additions & 0 deletions .github/workflows/deploy_image.yaml
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alot of these steps are copied from https://github.com/dockstore/dockstore/blob/b604742ab675b6ca47629e7e54a71478c6247191/.github/workflows/deploy_tagged.yml#L1 with a few modifications so that it can be reused by multiple repositories

Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
name: Reusable Image Deploy

on:
workflow_call:
inputs:
quayRepository:
description: The quay repository to upload the image to. The repository must belong to the dockstore quay organization.
required: true
type: string
dockerContext:
description: The Docker context containing the Docker file of the image to build and push.
required: false
default: .
type: string

env:
DOCKER_IMAGE_NAME: quay.io/dockstore/${{ inputs.quayRepository }}

permissions:
id-token: write
contents: read

jobs:
deploy_image:
name: Deploy image to quay.io/dockstore/${{ inputs.quayRepository }}
runs-on: ubuntu-22.04

steps:
- uses: actions/checkout@v4

# Step that does that actual cache save and restore
- uses: actions/cache@v4
with:
path: ~/.m2/repository
key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-

- name: Set up JDK
uses: actions/setup-java@v4
with:
java-version: '21.0.2+13.0.LTS'
distribution: 'adopt'

- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.AWS_ROLE_TO_ASSUME }}
role-session-name: docker_checksum_upload_from_github
aws-region: ${{ secrets.AWS_REGION }}

- name: Login to Quay.io
uses: docker/login-action@v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USER }}
password: ${{ secrets.QUAY_TOKEN }}

- name: Build
run: ./mvnw -B clean install -DskipTests

- name: Set folder name
run: |
S3_FOLDER=${{ github.ref_name }}
echo "S3_FOLDER=${S3_FOLDER//\//_}" >> $GITHUB_ENV

- name: Build and push
id: docker_build
uses: docker/build-push-action@v6
with:
context: ${{ inputs.dockerContext }}
push: true
tags: '${{ env.DOCKER_IMAGE_NAME }}:${{ env.S3_FOLDER }}'

- name: Create checksums
run: |
docker inspect ${{ env.DOCKER_IMAGE_NAME }}:${{ env.S3_FOLDER }} | grep -A 1 RepoDigests
docker inspect ${{ env.DOCKER_IMAGE_NAME }}:${{ env.S3_FOLDER }} | grep -A 1 RepoDigests | grep -oPm1 'sha256:\K\w+' > image-digest.txt

- name: Get short SHA
id: slug
run: echo "::set-output name=sha7::$(echo ${GITHUB_SHA} | cut -c1-7)"

- name: Copy checksum files
run: aws s3 cp image-digest.txt s3://${{ secrets.AWS_BUCKET }}/${{ env.S3_FOLDER }}-${{ steps.slug.outputs.sha7 }}/${{ inputs.quayRepository }}/image-digest.txt
Copy link
Author

@kathy-t kathy-t Nov 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: the image digest is now uploaded to a folder with the quay image name. For example, s3://***/1.16.0-alpha.1-ae8db5d/dockstore-support/image-digest.txt (this folder doesn't exist so don't look for it 🙂)

110 changes: 0 additions & 110 deletions .github/workflows/deploy_tagged.yaml

This file was deleted.