This repo contains tasks, workflows, image definitions, and datatypes used to standardize the orchestration of common bioinformatics tasks using Flyte.
-
Spin up the local development sandbox
-
pip install union -
git clone https://github.com/unionai-oss/unionbio.git && cd unionbio -
union --config ~/.flyte/config-sandbox.yaml run --remote src/unionbio/workflows/simple_variant_calling.py calling_wfNavigate to the local URL that's produced and watch your workflow run!
✨Adding custom dependencies alongside Flytekit
ImageSpecs contained in the images module build a standard set of OCI-compliant container images for use throughout the different workflows. They can be built with entrypoints present in pyproject.toml.
✨Leverage dataclasses to keep things organized
Using dataclasses to define your samples provides a clean and extensible data structure to keep your workflows tidy. Instead of writing to directories and keeping track of things manually on the commandline, these dataclasses will capture relevant metadata about your samples and let you know where to find them in object storage.
✨Run arbitrary shell commands
FastQC is a very common tool written in Java for gathering QC metrics about raw reads. It doesn't have any python bindings, but luckily Flyte lets us run arbitrary ShellTasks with a clean way of passing in inputs and receiving outputs. Just define a script for what you need to do and ShellTask will handle the rest.
✨Decide wether to continue workflow execution based on QC metrics via conditionals
FastQC generates a summary file with a simple PASS / WARN / FAIL call across a number of different metrics. We can use conditionals in our workflow to check for any FAIL lines in the summary and automatically halt execution. This can surface an early failure without wasting valuable compute or anyone's time doing manual review.
✨Specify resources and parallelize via map task
FastP is another common pre-processing tool for filtering out bad reads, trimming, and adapter removal. It can be a bit more memory hungry than Flyte's defaults are set to; luckily we can use Resources to bump that up and allow it to run efficiently. Additionally, we can make use of a map task in our workflow to parallelize the processing of fastp across all our samples.
✨Pause processing while waiting for human input
As a final check before moving onto the alignment, we can define an explicit approval right in the workflow. Aggregating reports of all processing done up to this point, and visualizing it via Decks (more on that later), a researcher is able to quickly get a high level view of the work done so far and approve the analysis for further processing.
✨Leverage caching to save time on successive runs
Index generation can be a very compute intensive step. Luckily, we can take advantage of Flyte's native caching when building that index for bowtie and hisat. We've also defined a cache_version in the config that relies on a hash of the reference location in the object store. This means that changing the reference will invalidate the cache and trigger a rebuild, while allowing you to go back to your old reference with impunity.
✨Compare aligners across an arbitrary number of inputs via dynamic workflows
When prototyping a new pipeline, it's usually a good idea to evaluate a few different tools to see how they perform with respect to runtime and resource requirements. This is easy with a dynamic workflow, which allows us to pass in an arbitrary number of inputs to be used with whatever tasks we want. In the main workflow you'll pass a list of filtered samples to each tool and be able to capture run statistics in the Alignment dataclass as well as visualize their runtimes in the Flyte console.
✨Visualize performance via Decks
We use MultiQC, an excellent multi-modal visualization tool for reporting. After gathering all relative metrics from a workflow, we're able to render that report via Decks, giving us rich run statistics without ever leaving the Flyte console!