Add age of air diagnostic functionality

[jwarner8]

Adding capability to run the age of air diagnostic on data, linked to issue #269

[github-actions]

[jwarner8]

Tests to do include;

• Test calc distance with some known values.
• Test shape is correctly rejecting if not all equal, also if time coord not hourly.
• Check if time decreasing will fail correctly.
• Check pressure level not found fails correctly.
• check all variables present (need to add this in function directly).

[jwarner8]

.yaml is going to look something like 1. read cubes, 2. filter cubes (u,v,w,p), might need some additional filtering to only get pressure level data, otherwise it might start returning 10m winds and we don't want/use this. Better to do this filtering in dedicated operators than start finding ways around this within the age of air operator itself. Run the age of air diagnostic, and then produce contourf plot; tbd (and data)

[jfrost-mo]

Some vague review comments. I've not properly read through and understood the code yet though. I'll do that a bit later.

[jwarner8]

Some vague review comments. I've not properly read through and understood the code yet though. I'll do that a bit later.

thanks! There is a lot more to add yet, this is mostly initial commits for copy/pasting code into CSET. So best to hold of any review, as code needs a lot of tidying up/documentation, plus all the testing and cylc/yaml additions

[jfrost-mo]

Got a bit eager, but its great to see it coming along. I'll mark this PR as a draft for now.

[jwarner8]

Meta

• Create new template variable, with description. Q: what is sort-key.
• If age of air turned on, create a new variety of options; plev; list, list of pressure levels to run code on. Includevertical; bool, cyclic; bool.
• These variables (plev, includevertical, cyclic) need passing to yaml. Q: how to we get menu's to appear if diagnostic turned on.

YAML

• Will inherit plev, includevertical, cyclic from those selected in the GUI (and thus rose-suite.conf). For the filter cubes, if includevertical is false, we don't need vertical velocity fields. Q: how to make the YAML not look for this STASH if option selected? Can we put logic in YAML? Or would it be better to have two YAML (vertical and no vertical)?
• To plot, we can use standard contour plot and timeseries plot functionality already in CSET.

Cycling

• Will run AOA task on each forecast intialisation, and loop through each plev specified in rose-suite.conf. For example for 2 forecasts, with only ageofair diagnostic turned on, and plev= [250,500,850], run-time would look like;

20240417T0000Z
task: ageofair_250
task: ageofair_500
task: ageofair_850
20240417T1200Z
task: ageofair_250
task: ageofair_500
task: ageofair_850

So all pressure levels run in parallel.

[jwarner8]

Only thing left to do now is build the cylc workflow, tests etc. all done.

[jwarner8]

Now ready for first stage technical review (quite a big changelog)
A TODO is how to visualise the result, 2D plots for each leadtime, but also want a timeseries (collapse latitude,longitude)

[jfrost-mo]

Thanks James, this seems like a really useful operator.

The largest thing is going to be updating the recipe to accommodate the changes we have made in CSET over the past couple months. It would probably be worth rebasing your changes on the latest version of the main branch to make that easier. (git rebase main, then fix a bunch of conflicts...)

I've noted a bunch of specific points, but the general design and approach are sound.

[jfrost-mo]

Note that there are other suggestions that GitHub has hidden under "hidden conversations". You might be better off looking through the Files tab for them.

[jwarner8]

I think the design of this operator needs some consideration with how it will work with the current default of CSET of cycling through valid-time. The operator expects to be run once, for a given initialisation, and passed all the data for that forecast. I don't really know how compatible this is with the current architecture, because each calculation at each forecast time depends on the preceding data.

[jfrost-mo]

For now this can be worked around by running the operator within the collate step. We are currently discussing a longer term change to cycling on data time, which would alleviate this.

[jwarner8]

Update 04/07 - dependency on ability for regrid to handle cubelists (which will make the yaml much nicer as we need to repeat regridding for multiple variables). Links #720 #721

[jwarner8]

I have added an additional call to the plot line series in the recipe, as it is good to also visualise this as a domain average. This currently appends the domain average timeseries plot to the back of the other spatial plots. I'd rather not copy paste the recipe to be re-run for domain average (a lot of extra code, plus duplication of actual run-time to derive the diagnostic). I tried to redefine the title half way through the yaml but it didn't like it. At this stage happy for domain average timeseries to be appended to end of spatial plots, but wondered if there was a better solution to computationally heavy diagnostics that can be visualised in various ways

[jwarner8]

I am also getting very low coverage (15%) despite tests covering most combinations, am I missing something? The coverage report highlights most lines in the function but no hover-over/info to tell me why the line isn't executed during the tests. Otherwise, just need to rebase and then ready for final technical/science review

[jwarner8]

I need to add some modifications to the script to handle ensemble data (essentially checking to see if a realization dimension exists, and if so, then iterate over it and return same cube with the additional realization dimension to make it compatible). Will also need to add a new test for this (and create some sample data).