Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DM-47529: Migrate Prompt Processing PipelinesConfig to YAML #222

Merged
merged 7 commits into from
Nov 25, 2024
Merged

DM-47529: Migrate Prompt Processing PipelinesConfig to YAML #222

merged 7 commits into from
Nov 25, 2024

Commits on Nov 25, 2024

  1. Fix incorrect return type for PipelinesConfig.get_pipeline_files.

    @kfindeisen
    kfindeisen committed Nov 25, 2024
    Configuration menu
    Copy the full SHA
    ecd3e54 View commit details
    Browse the repository at this point in the history

  2. Remove "not a singleton" disclaimer from PipelinesConfig. 

    A typical run has two simultaneous instances (one for preprocessing,
one for the main pipeline), so this disclaimer is redundant.
    @kfindeisen
    kfindeisen committed Nov 25, 2024
    Configuration menu
    Copy the full SHA
    1f93558 View commit details
    Browse the repository at this point in the history

  3. Switch pipelines config format from Python-like to YAML. 

    Delegating parsing to YAML makes it much easier to add more fields to
the pipelines spec, and even this basic implementation catches some
escaping errors in the original tests.
    @kfindeisen
    kfindeisen committed Nov 25, 2024
    Configuration menu
    Copy the full SHA
    0bbe11d View commit details
    Browse the repository at this point in the history

  4. Factor YAML parsing out of PipelinesConfig. 

    Making the activator responsible for YAML input leads to better
seperation of concerns. It also makes unit tests slightly
better-behaved, because they can be expressed in pure Python. However,
this change makes some test cases obsolete, because they specifically
test text formatting.
    @kfindeisen
    kfindeisen committed Nov 25, 2024
    Configuration menu
    Copy the full SHA
    93f072a View commit details
    Browse the repository at this point in the history

  5. Allow ordered evaluation of pipeline configs. 

    Previously, pipeline configs behaved as a mapping from survey to
pipeline list; this design didn't support more complex specs (wildcards,
spatial constraints, etc.). Taking the first block that "matches" a
visit allows for more free-form specification of when pipelines should
be run.
    @kfindeisen
    kfindeisen committed Nov 25, 2024
    Configuration menu
    Copy the full SHA
    7e8698b View commit details
    Browse the repository at this point in the history

  6. Move pipelines checks from PipelinesConfig to _Spec. 

    This change allows for cleaner separation of concerns, as all input
handling is now done by _Spec.
    @kfindeisen
    kfindeisen committed Nov 25, 2024
    Configuration menu
    Copy the full SHA
    e527b2c View commit details
    Browse the repository at this point in the history

  7. Make survey optional in PipelinesConfig. 

    No longer requiring a survey makes it possible to naturally add in other
optional constraints, such as position.
    @kfindeisen
    kfindeisen committed Nov 25, 2024
    Configuration menu
    Copy the full SHA
    46a0f27 View commit details
    Browse the repository at this point in the history