0.3.1 Release #158

HCookie · 2024-11-22T16:55:33Z

Release `0.3.1`

A number of significant changes have been contributed. So a release is due.

Merged

Waiting on

150 n320 callback number of pixels used by datashader #152

Nice to have

Update sanity checks for training data consistency #120

Releases needed

graphs and datasets must be release prior to this.

📚 Documentation preview 📚: https://anemoi-training--158.org.readthedocs.build/en/158/

* Refactor Callbacks - Split into seperate files - Use list in config to add callbacks - Provide legacy config enabled approach - Fix ruff issues * Update changelog * Fix TypeError * Move to hydra.instantiate * Add __all__ * Add to base config * Fix nested list * Fix nested get issue * Fix type checking * feat: edge plot in callbacks * feat: set default extra callbacks * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: typing & refactoring * fix: remove list comprehension * Refactor according to PR - Prefill config with callbacks - Warn on deprecations for old config - Expand config enabled - Add back SWA - Fix logging callback - Add flag to disable checkpointing - Add testing * Update deprecation warning * Refactor: Remove backwards compatability, - Split plots - Rename, lr to optimiser - Refactor plotting callbacks to be more init config * Fix tests * PR Fixes - Remove enabled from plotting callbacks - Connect sample_idx in config * Update Changelog * Refactor rollout (#87) Refactor rollout logic * Remove batch frequency from LongRolloutPlots * Remove TP reference * Remove missing config reference * Authentication support for mlflow sync (#51) * feat: authentication support for mlflow sync * chore: formatting * chore: changelog * chore: changelog add link * fix: sync authentication flag * refactor: move `health_check` to submodule top level * feat: add health check * chore: update error msg * refactor: mlflow utils * New mlflow authentication API (#78) * fix: mlflow auth use web seed token * feat: make target env var an optional argument * chore: docstrings * fix: tests * chore: add comment * chore: changelog * chore: docstring * Update changelog * rebase * Update deprecation warning * Refactor: Remove backwards compatability, - Split plots - Rename, lr to optimiser - Refactor plotting callbacks to be more init config * add scatter plot * adding async * fix * tests * fix failing tests * rm change to ds valid * precommit hooks * fix linting * rebase * Update deprecation warning * Refactor: Remove backwards compatability, - Split plots - Rename, lr to optimiser - Refactor plotting callbacks to be more init config * add scatter plot * adding async * fix * tests * fix failing tests * rm change to ds valid * precommit hooks * fix linting * revert unnecessary config changes * change config files * Swapped histogram and spectrum * Update copyright notice * Fix issues with split of PlotAdditionalMetrics * Fix CHANGELOG * Fix documentation for callbacks * Add all callback submodules to docs * Apply suggestions from code review Co-authored-by: Sara Hahner <44293258+sahahner@users.noreply.github.com> * Fix init args issue in RolloutPlots * Add rollout_eval config * Add training mode to rollout step * Force LongRolloutPlots to plot in serial * Add warning to LongRolloutPlots when async * Fix asserrt calculation * Apply post_processors before plotting in LongRolloutPlots * Fix reference to batch * Fix debug config * brinding plot for mean wave direction and fixing type hinting * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add changelog entry * fixes for async plots to work * fix pre-commit styling * improved loop closing and readability * fixing for pre-commit hooks * remove commented block * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * address sugestion for args and kwargs and missing type hints * update flag to datashader rather than scatter * update configs * update docs * update comment for readability * update branch * update branch and test --------- Co-authored-by: Harrison Cook <harrison.cook@ecmwf.int> Co-authored-by: Mario Santa Cruz <mariosanta_cruz@hotmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Gert Mertes <13658335+gmertes@users.noreply.github.com> Co-authored-by: Sara Hahner <44293258+sahahner@users.noreply.github.com> Co-authored-by: anaprietonem <ana.prietonemesio@ecmwf.int> Co-authored-by: Ana Prieto Nemesio <91897203+anaprietonem@users.noreply.github.com>

[Changelog] Update to 0.3.0

* feat: increase mlflow http retry and timeout * chore: changelog * feat: reduce values * feat: update values for 1 hour of downtime * refactor: set env vars before mlflow import * chore: formatting * Update CHANGELOG.md

@lzampier

Rollout video of variable dynamics --- Review: @lzampier @HCookie @mc4117 @mchantry

* no longer checking tracking uri when running mlflow offline old behaviour when running offline with the default tracking uri (???) would cause a crash. My understanding is there is no reason to check tracking uri when running offline because this is provided later when the run is synced to mlflow. One use case for this would be for external vendors benchmarking anemoi. They could use the mlflow logging features to analyse GPU utilization etc. without requiring a tracking uri. * Update src/anemoi/training/diagnostics/logger.py Co-authored-by: Harrison Cook <Harrison.cook@ecmwf.int> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Harrison Cook <Harrison.cook@ecmwf.int> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* added red gpu monitoring, increased green gpu monitoring and refactored monitors into their own files * applied feedback * added changelog entry * fixed Changelog entry

* feat: mask NaNs in training loss function --------- Co-authored-by: Jakob Schloer <jakob.schloer@gmail.com> Co-authored-by: Harrison Cook <harrison.cook@ecmwf.int>

* Update plots.py To plot the power spectra, we need to create a regular grid (n_pix_lat x n_pix_lon) and interpolate the data on it. The way n_pix_lat and n_pix_lon were previously defined is not robust and might lead to errors. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update plots.py * Update CHANGELOG.md --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* feat: improve dataloader memory - Add reader groups to support sharded reading of batches - Add dataloader.read_group_size in config to control read behaviour - Add GraphForecaster.allgather_batch() to reconstruct full batch from shards - Refactor callbacks to call allgather on batches as needed * docs: update docstring with instructions on reader group usage * refactor: rank computations via SLURM_PROCID - Pass model/reader group information from DDPGroupStrategy instead --------- Co-authored-by: gabrieloks <116646686+gabrieloks@users.noreply.github.com> Co-authored-by: Harrison Cook <Harrison.cook@ecmwf.int> Co-authored-by: sahahner <hahner.sa@gmail.com>

* Change training end date/validation start date assertion to a warning to allow flexibility. Co-authored-by: Harrison Cook <Harrison.cook@ecmwf.int> --------- Co-authored-by: Harrison Cook <Harrison.cook@ecmwf.int>

* warmup config for reproducibility of aifs v0.3 * add entry to changelog * update docs

* Disable scalar indices if no variable scalar is used in val_metrics

* enable plothistogram and plotspectrum for only one variable

HCookie and others added 15 commits November 14, 2024 18:12

[changelog update] Update to 0.3.0

dd9883f

Merge branch 'develop' into docs/changelog-update-0.3.0

3c6f2d0

[Changelog] Update to 0.3.0

d0f2035

[Changelog] Update to 0.3.0

Increase MLflow HTTP retry (#111)

23fd045

* feat: increase mlflow http retry and timeout * chore: changelog * feat: reduce values * feat: update values for 1 hour of downtime * refactor: set env vars before mlflow import * chore: formatting * Update CHANGELOG.md

Rollout video of variable dynamics (#65)

7f4023d

Rollout video of variable dynamics --- Review: @lzampier @HCookie @mc4117 @mchantry

added red gpu & increased green gpu monitoring (#147)

553f247

* added red gpu monitoring, increased green gpu monitoring and refactored monitors into their own files * applied feedback * added changelog entry * fixed Changelog entry

Feature/mask NaNs in training loss function (#72)

25abf5e

* feat: mask NaNs in training loss function --------- Co-authored-by: Jakob Schloer <jakob.schloer@gmail.com> Co-authored-by: Harrison Cook <harrison.cook@ecmwf.int>

Validation and Training dataset dates assertion (#154)

e3fe023

* Change training end date/validation start date assertion to a warning to allow flexibility. Co-authored-by: Harrison Cook <Harrison.cook@ecmwf.int> --------- Co-authored-by: Harrison Cook <Harrison.cook@ecmwf.int>

warmup config for reproducibility of aifs v0.3 (#155)

0608f21

* warmup config for reproducibility of aifs v0.3 * add entry to changelog * update docs

hotfix: Expand scalar to prevent index out of bound error (#160)

1abb65e

* Disable scalar indices if no variable scalar is used in val_metrics

Callback PlotHistogram breaks if only one variable is specified. (#165)

fa43078

* enable plothistogram and plotspectrum for only one variable

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

0.3.1 Release #158

0.3.1 Release #158

HCookie commented Nov 22, 2024 •

edited

Loading

0.3.1 Release #158

Are you sure you want to change the base?

0.3.1 Release #158

Conversation

HCookie commented Nov 22, 2024 • edited Loading

Release 0.3.1

Merged

Waiting on

Nice to have

Releases needed

HCookie commented Nov 22, 2024 •

edited

Loading

Release `0.3.1`