What Is UW Anyway?

Since we started developing Unified Workflow Tools, our team has received many questions along these lines. Perhaps now is a good time to start dispelling some myths and preconceptions.

First and foremost, Unified Workflow (UW) is a versatile Python package that provides command-line tools and a Python API to streamline the configuration and running of computational codes common to workflow tasks within the UFS. UW defines a consistent YAML-based control language that configures tools like one for comparing unordered namelist files with mixed-case keys, for rendering templates, and even one for creating the Rocoto workflow manager’s XML workflow definition file.

We also have a set of tools that we call drivers that are tied to a particular component (e.g. models, pre-processors, post-processors) and deal with preparing run directories, staging data files, defining a component’s resource requirements, and executing a program, either directly or by submitting a job to a batch system. Each of these drivers has its own structured UW YAML language defining execution parameters and configuration settings specific to its respective program, thus defining a YAML interface to configure and run a component. All the UW YAML definitions are validated against corresponding schemas to help users understand where they may have introduced errors as early as possible. These drivers are compatible with all UFS applications and are highly configurable to meet the needs of any app. We currently have drivers that not only support current UFS Application components, but planned components, including JEDI and MPAS. Using UW tools, running FV3 on an interactive node from the command line can be as simple as: $ uw fv3 run --config-file config.yaml --cycle 2024-06-06T12

The UW library of drivers has grown significantly in the past few months, and we are now at a point to define how these tools come together to create a workflow framework. We define a workflow framework as the set of scripts that configure an experiment, often referred to as a configuration layer or configuration subsystem, and the scripts necessary to run the tasks defined by a workflow. Here, workflow is specifically defined as the way a particular set of components is meant to run together to create products used for guidance. A workflow, in this context, might be labeled as RRFS or HAFS to indicate that it is the particular definition of how an operational system runs, or it might be more generally defined as a “cold start” workflow where there is room to modify this in countless ways to run a forecast of interest without data assimilation.

Historically, each application has created bespoke tools to drive its workflows, leading to duplication of effort and incompatibility across UFS applications. Our vision is to unify these disparate configuration layers into a single, cohesive framework that interfaces with existing workflows and workflow managers. We aim to alleviate common pain points, provide the tools necessary to create a concise and efficient runtime environment, flatten the learning curve to get what you need out of a UFS app, and empower scientists to realize their experiments exactly as they envision them.