Time logging

[dberenbaum]

It may be useful to log the time of each stage in the pipeline. Some use cases for this information:

• Monitor stages that are bottlenecks to optimize execution time.
• Compare times between experiments as a metric for choosing between them.
• Compare times between runs on different machines to decide optimal machine setup.

This requires:

• Logging time for each stage of each run.
• Way to compare these times similar to how metrics are compared.

Nice to have features:

• Total time for each run.
• Way to compare machine specs between runs.
• Logging actual resource usage (like max/avg compute/memory during stage run) for each run and comparing.

[karajan1001]

Now we have time info before every log

2021-12-30 10:33:39,495 DEBUG: Adding '/Users/gao/Code/test/exp-example/.dvc/config.local' to gitignore file.
2021-12-30 10:33:39,497 DEBUG: Adding '/Users/gao/Code/test/exp-example/.dvc/tmp' to gitignore file.
2021-12-30 10:33:39,497 DEBUG: Adding '/Users/gao/Code/test/exp-example/.dvc/cache' to gitignore file.

So we can calculate the interval time between every two logs, is it enough for this?

[daavoo]

I was thinking more in the lines of fully-featured MonitorTask (https://github.com/iterative/dvc/blob/main/dvc/stage/monitor.py) that could be also used to track resource usage

[ambroserb3]

+1 This feature would be great for monitoring dvc in production. Knowing how long pipelines take to run at a granular level is incredibly helpful for debugging problems especially those related to timeouts. I.e is there a data dependency that wasn't pulled down in time?