This repository provides Docker images and GitHub Actions for use in CodeQL courses on Learning Lab.
These actions allow you to specify workflows that can check that course participants' queries are correct, by running their queries against a well-known CodeQL database, and checking the results are as expected. Whatever the outcome, the action will post a comment on the commit which was pushed to add the queries.
When a user's results are incorrect, the comment will include details of which results are missing, and which are superfluous, including links to the lines of source code on GitHub when possible:
Screenshot:
Table of Contents
There are two main components to any Learning Lab course for CodeQL that uses the components in this repository:
-
Each course has its own GitHub Action that is designed to be used in workflows that run when a course participant pushes new commits to their repo. The action will check which queries have changed in the push, and run the queries that it recognizes as part of the course (based on the filename).
After running the queries, the action will check the results against a CSV file of expected results. It will then post a comment on the commit, detailing whether each query produced the correct results or not. And if not, it will include details of which results are missing, and which results are unexpected.
These actions are bundled using Docker, and made available using GitHub Packages.
-
This is the course itself. It creates the initial repository the participant will use for their course, posts instructions as GitHub issues, and listens for comments posted by the GitHub action to know when the user has completed the current task correctly, and is ready to advance to the next one.
(for an example of a working action,
see courses/cpp/ctf-segv).
Course actions consist of an action.yml file,
and docker image built from the base image
codeql-learninglab-check.
The base image expects course images built on-top of it
to add the file /home/codeql/config/config.json,
which details the configuration for the course.
The file should look something like this:
{
"databasePath": "<path-to-database-directory>",
"locationPaths": "https://github.com/<owner>/<repo>/blob/<sha>{path}#L{line-start}-L{line-end}",
"expectedResults": {
"step-01.ql": "step-01.csv",
"step-02.ql": "step-02.csv",
"step-03.ql": false,
}
}In addition to the config.json file above,
a course image needs to also add the snapshot directory
that queries should be run against,
and csv files for the expected results.
databasePathshould be a directory in the docker image, relative to theconfig.jsonfile, that contains the extracted CodeQL database that queries will be run against. If you are using the template below, it will usually be the name of the only top-level directory from inside the database zip file.locationPathsis an optional template string that can be used to enable source links in comments, when participants have written queries that output unexpected rows, or are missing results.<owner>,<repo>and<sha>should be replace as appropriate, the placeholders{path},{line-start}and{line-end}are used by the checker, and should be left as-is.expectedResultsis an object that maps expected query filenames to a csv file detailing what the expected results for this query should be. Only the first expression for each row in the query results is checked. If instead of a CSV filename,falseis used, then the checker will assume that the CSV file has simply never been generated, and will print out the resulting output from the query for you to copy into a new file.
To simplify course creation, we recommend structuring your course folder like so:
├── answers <─── Model Answers
│ ├── qlpack.yml
│ ├── step-01.ql <─┬─ Answers with expected paths
│ ├── step-02.ql <─┤ (relative to answers/)
│ └── ... <─┘ as specified in config.json
├── image
│ ├── config
│ │ ├── config.json <─── Main course configuration
│ │ ├── step-01.csv
│ │ ├── step-02.csv
│ │ └── ...
│ └── Dockerfile
└── action.yml
(For your convinience,
we've created a template course that uses this file-structure
in the folder templates/action.
You can simply copy the folder,
and follow the instructions in the template README for what things to replace).
action.yml should look something like this:
name: 'Check queries'
description: 'Check that the queries that have been pushed (as part of the lesson) produce the correct results'
author: 'GitHub <opensource+codeql-learninglab-actions@github.com>'
runs:
using: 'docker'
image: 'docker://docker.pkg.github.com/<owner>/<repo>/<package>'
branding:
icon: 'check-circle'
color: 'purple'and Dockerfile should look something like:
FROM docker.pkg.github.com/github/codeql-learninglab-actions/codeql-learninglab-check:<version>
## Add course config
COPY --chown=codeql:codeql config /home/codeql/config
WORKDIR /home/codeql/config
# Download, unzip and then delete the zip file in one step to reduce image size
RUN wget --quiet <url-for-snapshot-zip> -O database.zip && unzip -qq database.zip && rm -rf database.zipNote that we download, unzip and then delete the zip file of the snapshot in a single step here. This helps us reduce the size of the image, as separate steps would result in intermediate image layers that are built on-top of one another.
You can test the action either locally or on GitHub actions.
Locally:
To test a course locally, from the course directory, run either of these scripts:
scripts/test-course-actual.sh: which will download and use the specific version ofcodeql-learninglab-checkthat is specified inDockerfilescripts/test-course-latest.sh: Which will also build thecodeql-learninglab-checkimage locally, and tag it with the expected base image of the course, allowing you to test how changes to thecodeql-learninglab-checkaffect this specific course, without publishing any new images.
Both scripts take as argument an optional regexp string. If this string is passed, only the queries with names matching the regexp will be run. Otherwise all queries are run.
In GitHub Actions:
If adding a course to this repository,
extend the workflow file .github/workflows/ci.yml
to include your new course.
Any subsequent pushes to any branch should trigger an Action to run
that will succeed only when all the expected queries produce the right results.
If you are creating a course in another repository,
you can copy the scripts/test-course-actual.sh
and scripts/test-course-latest.sh files
into that repository,
and add a similar workflow file to the one mentioned above.
When testing the action (as detailed above),
when a query that is run produces unexpected results,
or it is specified as false in config.yml instead of listing a CSV filename,
the actual results that it produces are printed out in the console.
You can then store this output as the relevant CSV file.
So the workflow for adding a new query and CSV file looks like:
- add the query (
.qlfile) toanswers/. - add the query to the
expectedResultsproperty inconfig.json, with a starting value offalse. - Test the action (whichever method you prefer).
- Copy the CSV output to the appropriate file in
image/config/. - Re-test the action to ensure it marks the query as producing the correct results.
The main thing you need to do here is publish your Docker image somewhere,
and ensure that action.yml referrs to a tag that is downloadable.
We recommend setting up a GitHub Actions Workflow
to automatically publish your docker image
with the version latest to docker.pkg.github.com
whenever you get a new push to master.
This is what we do in
.github/workflows/publish.yml.
Any courses that are added to this repository need to be published in this manner.
If you want to add a course to this repository, ensure that:
- You're creating the course in the
courses/folder, under the appropriate language sub-folder for the project. - You update both
.github/workflows/ci.ymland.github/workflows/publish.ymlto include testing and image publishing for your course.
If you have not created a Learning Lab course before, it is recommended to take the course on creating a course!
There are core repositories that need to be created as part of any learning-lab course:
- The course repository: All the course configuration, instructions etc...
- The template repository: The initial contents that populate the repository created on behalf of the course participant. (All courses are taken with respect to it's own repository)
We've created two template directories that you can use as a starting point for your own CodeQL Learning Lab Course:
Simply copy the contents of these templates into their own repositories, and follow the template instructions to get started.
(Remember that you need to create 2 separate repositories for your Learning Lab course, they can't be directories in an existing repo).
Feel free to add your own courses to this list! See CONTRIBUTING.md.
We welcome contributions,
both for new courses,
and improvements to existing courses ot the
codeql-learninglab-check docker image.
The code in this repository is licensed under MIT (see LICENSE.md). However as it makes use of the CodeQL CLI, you must also abide by the GitHub CodeQL Terms and Conditions, whenever your usage involves the CodeQL CLI.
In particular, you are not permitted to use these docker images or actions to create CodeQL databases using the CLI in CI/CD, as per the terms & conditions:
the Software cannot be used [...] To generate CodeQL databases for or during automated analysis, continuous integration or continuous delivery, whether as part of normal software engineering processes or otherwise.