Skip to content

Latest commit

 

History

History
117 lines (97 loc) · 5.14 KB

README.rst

File metadata and controls

117 lines (97 loc) · 5.14 KB
Raw

wdl-packager

Package a WDL and its imports into a zip file. This idea has since been ported to miniwdl and is now available as the miniwdl zip command. Miniwdl zip packages reproducibly by default (no --reproducible flag needed) and does not require git for the reproducible packaging to work correctly. As such miniwdl is the preferred option for packaging WDL files.

The zip file can be used as a valid imports zip for cromwell. It can also be used to archive your workflow.

Wdl-packager can create zip packages that are binary reproducible when used with the --reproducible flag. It can also include additional files such as licenses, READMEs and example configuration files. This makes it ideal for distributing WDL workflows as packages.

Wdl-packager currently only supports file based imports.

Usage

usage: wdl-packager [-h] [-o OUTPUT] [-a ADDITIONAL_FILES]
                    [--use-git-version-name] [--use-git-commit-timestamp]
                    [--reproducible] [--version]
                    WDL_FILE

positional arguments:
  WDL_FILE              The WDL file that will be packaged.

optional arguments:
  -h, --help            show this help message and exit
  -o OUTPUT, --output OUTPUT
                        The output zip file. By default uses the name of the
                        input. This overrides the git name option.
  -a ADDITIONAL_FILES, --additional-file ADDITIONAL_FILES
                        Additional files to be included in the zip. Additional
                        files will be added according to their relative
                        position to the WDL file. If that is not possible they
                        will be added to the base of the zip. Multiple '-a'
                        flags can be used.
  --use-git-version-name
                        Use git describe to determine the name of the zip.
  --use-git-commit-timestamp
                        Use the git commit timestamp to timestamp all the
                        files in the zip.
  --reproducible        shorthand for --use-git-version-name and --use-git-
                        commit-timestamp
  --version             show program's version number and exit

Reproducibility

The internal process to create a reproducible package is as follows:

  • It checks each file with git log --n1 --pretty=%at to get the unix timestamp of the latest commit that affected that file.
  • Each file is then copied to a temporary directory where the last modified time is changed to the unix timestamp found in the first step.
  • The list of files is then sorted by their destination path in the zip. The sorting ensures that the files will always be added in the same order.
  • The timezone of the process is changed to UTC, as the timezone affects the timestamp of the files in the zip archive.
  • The files are added to the zip package in sorted order.

The name for a reproducible package consists of

  • The WDL file basename without .wdl.
  • A version description by git describe --always.
  • A .zip extension.

Known issues

  • Old versions of Cromwell contain a bug that causes a crash when opening zip files with nested directories. This was fixed in cromwell version 49.
  • http:// imports are currently not supported. Wdl-packager could be an ideal tool to fetch WDLs from the web and package their imports in such a way that they can be used locally. Unfortunately this requires rewriting the import paths inside the WDL files before they are packaged. This is non-trivial to implement and it may have unintentioned side effects.
  • When packaging reproducibly the timezone of the python process needs to be changed to 'UTC'. Otherwise the last modified times in the zip will differ for each timezone. This will affect all code run in the same process. This does not matter for wdl-packager itself, but it does matter for programs that use wdl_packager as a library. Not using use_git_timestamps circumvents this problem.