params function registry

[daavoo]

Summary

It would be nice to expose part of the params functionality to some sort of function registry (i.e. using catalogue) in order to allow users to inject custom logic for handling params.

There have been several discussions and feature requests for extending or modifying params: #4780 #4883 #5477 #5777 #5868 #5971 #6107 #6129 #6511 #6512 discord discussion about hydra

Motivation

I think that allowing users to register custom functions could partially alleviate these requests and dvc could focus on maintaining a minimal built-in functionality.

For example, having the registry would allow:

• dvclive to provide some opinionated dvc params functionality focused on specific ML Frameworks as part of its integrations without requiring changes to dvc core.

• dvc params to integrate with other configuration libraries like configparser, dynaconf, hydra, yacs, etc.

• Add strict validation to dvc params (i.e. using pydantic

And probably by just exposing the options to the users some more ideas would come.

Implementation

I see a few potential places to inject a registry:

• load_param_overrides
• MODIFIERS
• LOADERS

It would require to be more explicit (on docs and/or code) about the expected inputs/outputs for each registry place.

[dberenbaum]

Great synopsis of the issues, @daavoo! Before we get to a function registry, I think it would be helpful to recap some of the problems in those issues, and some implementations to solve those particular issues. Abstracting any solutions into a function registry could be revisited as a next step if they aren't flexible enough to meet all user's needs.

Problems

Too many parameters to specify

There are too many parameters to explicitly hard code each one. If they are in params.yaml, I don't have to explicitly list the parameters individually, but if I want to use some other params path (either because yaml is inconvenient for me or because my project/framework is already configured to read params from some other path), I have to list every parameter individually.

#5477 (comment): "Problem here is that I can modify the learning rate, momentum, architecture details...."

#6107: "The scripts could have a lot of args (this train.py has about 20 of them, all handled by Python argparse module)."

Repetition

Parameters have to be repeated even in simple scenarios (they need to be defined in the params file, dvc.yaml, and in the code). For ML frameworks that have their own parameter configuration formats/logic, they may also need to be repeated in command-line arguments or in separate files.

#5477 (comment): "I will end up repeating most of my config file in the params.yaml."

#6107: "I’d need to duplicate all arguments in two places: in params.yaml and in dvc.yaml, thus having three slightly different copies of the arguments."

Complex objects/expressions are not supported

Type constructors and other complex expressions that may be used in ML frameworks for parameter configuration cannot be parsed today as parameters.

#5477 (comment): "At the end it would be super useful to just use my python config files as params directly. At this moment I cannot afford to remove type constructors of all my config files."

#6512: processed_data_path = raw_data_dir / processed_data_filename

Modifying existing workflows

I can mitigate some of these issues if I adapt my workflow to what DVC expects. I can move my parameters to params.yaml as a simple example. However, if I want to use an existing ML framework that expects configuration a certain way, it may be difficult and time-consuming to adapt my workflow.

#5477 (comment): "At this moment I cannot afford to remove type constructors of all my config files."

#6107: "And if I’m not sure I want to migrate and just want to give DVC a try I would want to keep all my current scripts unchanged to remove DVC later if I won’t be planning on using it."

--set-param has unexpected parsing behavior

I consider this a separate issue from the others since it is confined to the --set-param flag, so I won't discuss it below.

It's not always obvious how strings passed to --set-param will be parsed or dumped back to params files.

#5777: "When trying to run an experiment --set-param does not update class variable TrainConfig.TEMP=1e-8 to TrainConfig=0.0001 declared and defined in params.py"

#5868: "Parameters to be stored in a JSON file that are ISO dates (yyyy-mm-dd) are automatically treated as date values by exp run, it seems -- which then errors when writing them to JSON."

#5971: "The problem is that the yaml library only supports yaml 1.1 when DVC uses yaml1.2. This issue seems to happen when exponents are generated."

#6129: When I try to run experiment and set some parameter to be empty or "", I get an error.

Discussion

DVC params began as a way to more granularly track dependencies, but now params are at least as much about tracking experiment info. params in DVC are really a special case for when the same config file dependency is reused across multiple stages. See https://dvc.org/doc/command-reference/params#examples-python-parameters-file:

This allows you to define stage dependencies more granularly: changes to other parts of the params file will not affect the stage. Parameter dependencies also prevent situations where several stages share a regular dependency (e.g. a config file), and any change in it invalidates all of them (see dvc status), causing unnecessary re-executions upon dvc repro.

Often, the problem is the opposite, where parameters for a single stage are spread across multiple files. In this case, listing each config file as a dependency is easy, but it's hard to track all the parameters in those files and integrate with the dvc concept of params.

What's needed is a way to log generic parameters without the special meaning of "granular dependencies" that DVC expects. I can think of a couple solutions:

Parameter logging API method

A proposal for log_param() has already been raised in iterative/dvclive#100. A workaround today could be a stage before training that dumps all the info from the config files to params.yaml.

I have a few questions about how to implement a parameter logging api:

• Should it be a Python API or is there some way to make this language agnostic (besides the workaround mentioned above)?
• Can it support dumping in bulk? Looking at examples like https://github.com/mmeendez8/mmdetection_dvc/blob/main/configs/coco_sample.py, logging everything in the file seems like it will still take a lot of custom user code.
• Should the logged parameters be saved to a file? Would this be considered a dependency, an output, neither? I think it's fine to treat these as an output but show them as parameters when using commands to view parameters like dvc exp show.

Parsing more complex files

Another option is to be able to read more complex Python files (and maybe other file types if needed). This might included reading type constructors or other complex expressions. I think this approach seems more problematic both because it's less generic (dvc would need to handle all kinds of complex parsing) and it's hard to imagine how to do it in a safe way that doesn't risk executing arbitrary code.

If we take this approach, dvc might need to support specifying custom params files without specifying each individual parameter to track in that file, since it seem cumbersome to have to repeat every parameter from a lengthy config file. This is actually likely a separate issue that I think might need to be addressed regardless.

I'd suggest we start moving forward with the parameter logging API since it seems like a pretty general solution that addresses all of the problems above. What do you think?

[dberenbaum]

There are many related discussions/issues discussing ideas similar to a function registry for dependencies, which seem to expand this idea from params to any dependency:

• Plugins for semantic changes tracking in dependencies #5920
• Support callback dependencies #2378
• Support function specific dependencies #3439

Also, a lot of the parameter issues noted here were considered when first implementing params: #3393.

[dberenbaum]

Another related discussion: https://discuss.dvc.org/t/dvc-and-hydra-integration/868/2

[dberenbaum]

Also somewhat related: #6805

[daavoo]

Another about hydra:
#7044

[daavoo]

And about https://github.com/explosion/thinc : #7122

Also, yesterday in Office Hours there was a question about yacs (one of the libraries listed in the initial discussion comment)

[dberenbaum]

Yet another config framework integration request 😅 : #7224

[dberenbaum]

Another hydra integration question: https://stackoverflow.com/questions/70711015/is-there-an-alternative-to-dvc-pipelines-to-create-a-dag-which-is-also-aware-of

[courentin]

Looking at this discussion, it seems that it would be a good way to address this feature request: #8072 about integrating HyperPyYAML.

Is it still something dvc wants to address?

[dberenbaum]

Hi @courentin!

Handling the basic problems of parameter tracking is a priority, but adding a function registry or similar is not something we are pursuing right now. We haven't closed it off as an option at some point, but for many people this would be a heavy and complex solution, and it raises a lot of implementation questions.

One avenue we are pursuing is Hydra integration since that is a popular, ML-focused configuration library, and it gives users some straightforward ways to solve a lot of configuration needs even though it's more opinionated than supporting a generic function registry.

If HyperPyYAML is valid YAML with a Python library to parse custom tags and do interpolation, I think it should be largely compatible already except for the interpolation, which for now we want to support in a single implementation like the Hydra integration.