Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature 380 phys tend plot actual #382

Merged
merged 37 commits into from
Jun 6, 2024
Merged

Feature 380 phys tend plot actual #382

merged 37 commits into from
Jun 6, 2024

Conversation

ahijevyc
Copy link
Contributor

@ahijevyc ahijevyc commented Aug 7, 2023 •
edited
Loading

Pull Request Testing

  • Describe testing already performed for these changes:
    plot all 4 state variables (tmp, spfh, ugrd, vgrd) in all 3 scripts (planview, cross section, and vertical profile) with 1-hour time window and 2-hour time window, starting at or after initialization time.

  • planview with and without vmin vmax, with all 3 types of vertical interpolation

  • Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:
    Not sure how to import physics_tend.py. Is it import physics_tend? or from . import physics_tend? pylint recommends from ., but this doesn't work when I test it. it has to be simpler import physics_tend

  • upload new fv3_history.nc and grid_spec.nc

  • Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [Yes or No]

  • Do these changes include sufficient testing updates? [Yes or No]

  • Will this PR result in changes to the test suite? [Yes or No]

    If yes, describe the new output and/or changes to the existing output:

  • Please complete this pull request review by [Fill in date].

Pull Request Checklist

See the METplus Workflow for details.

  • Add any new Python packages to the METplus Components Python Requirements table.
  • Review the source issue metadata (required labels, projects, and milestone).
  • Complete the PR definition above.
  • Ensure the PR title matches the feature or bugfix branch name.
  • Define the PR metadata, as permissions allow.
    Select: Reviewer(s)
    Select: Organization level software support Project or Repository level development cycle Project
    Select: Milestone as the version that will include these changes
  • After submitting the PR, select the ⚙️ icon in the Development section of the right hand sidebar. Search for the issue that this PR will close and select it, if it is not already selected.
  • After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
  • Close the linked issue and delete your feature or bugfix branch from GitHub.

@ahijevyc
Add panel for actual tendency
4212a01 
Only apply add_time0 function if time0 if needed.

Round times to nearest millisecond. This fixes times like 2019-06-15 18:03:0.000003,
which is supposed to be 2019-06-15 18:03 but has imprecise floating-point value.

Allow metpy.quantify() to protect units
This eliminates the need to restore units after applying .mean to DataArray.

Round times to nearest millisecond. Model output times are sometimes
imprecise due to floating point error.

Let argparse assume arguments are strings unless specified otherwise.

Allow time window to be a float, not just an int.

Refer to sum of all tendencies as "all_tendencies" instead of "total".

"dstate_variable" ->  "actual_tendency"

Don't treat nophys tendency differently. UFS output changed.
Removed bug where final units are assumed to be temperature tendency "K/hour".
Keep the tendencies in their native units of "[something] per second" instead
of converting to "[something] per hour".

Define time slice starting with time-after-time0 and ending with validtime.
We use the time *after* time0 because the time range of the tendency field is the period
immediately prior to the tendency timestamp.
That way, slice(time_after_time0, validtime) has a time range of [time0,validtime].
Compute the time-weighted mean tendencies for time index slice.

Be clear about "all" tendencies. It is the sum of all tendencies,
physics and non-physics. It does not include the
physics tendency variable, which may be part of the history file and
included in the tendency_vars list. That would be double-counting.

Subtract physics tendency variable if it was in tendency_vars.

Command line changes:
 Read file-type command line args as strings, not argparse.FileType('r').
 For simpler code.

 --norobust flag (to use data extremes for colormap range, not 99th
 and 2nd percentiles.)

logging info about renaming variables.
 Keep characters after final underscore. The first part is redundant.
 For example dtend_u_pbl -> pbl

actual_change -> actual_tendency

Test for shp is not None, not shp.
(Accept shp=something evaluated as False, like 0)

Print entire arg Namespace as fineprint, not just select variables.
Use plt.figtext instead of plt.annotate. Makes for cleaner code.

Reduce xarray time selection tolerance from 1 sec to 1 millisecond.

pep8 changes

reorder load modules

module and function documentation

shorten lines

lazy logging messages (no f-string)

specify encoding on open
@ahijevyc
tendency variable names changed
f2fb67d
@ahijevyc
Apply same upgrades to cross section vertical script.
77e5415 
Fix names of tendency variables in config file. The
state variables are not the same as the tendency variable
prefixes. For example spfh vs. qv.
@ahijevyc
applied same enhancements to vert_profile
61b3f9e
@ahijevyc
applied autopep8 formatting
fa90967
@ahijevyc
Widen table columns
77509a9 
new variable names

add table for derived tendencies

note about time window starting at initialization time and dealing
with it by appending state variables at time0.
@ahijevyc
Update fv3_physics.rst
d4c1f77 
equation for residual in description
@ahijevyc
subtract before cleaning times
ca65c63 
new date for new history file in Users Guide

new figures for Users Guide

fill -> tendencytype

simplify construction of da2plot in planview_fv3.py

don't allow user to plot state variable at validtime or time0.
it makes code so much uglier for such a simple request.

No explicit debug option for physics_tend.pts_in_shp

Restore statevariable in default output filename in planview_fv3.py.

100 dpi default instead of 150
@ahijevyc
Merge branch 'feature_380_phys_tend_plot_actual' of https://github.co…
b125fe7 
…m/ahijevyc/METplotpy into feature_380_phys_tend_plot_actual
@ahijevyc
Update fv3_physics.rst
c95a932 
fix link to cross section png
@ahijevyc
fix path to cross section png
7c63bb6
@ahijevyc
Merge branch 'feature_380_phys_tend_plot_actual' of https://github.co…
d8edfc9 
…m/ahijevyc/METplotpy into feature_380_phys_tend_plot_actual
@ahijevyc
pep8 formatting
6e2de69
@ahijevyc
use open_mfdataset for dask
a63b770
@ahijevyc
tendencies that are available depend on user's diag_table
a3ec403
@ahijevyc
wordsmith tendency timestamp description
9df587e
@ahijevyc
wordsmith tendency timestamp description
bc1ef28
@ahijevyc
mention time0_varname in config file in Guide
84d9a5c
@ahijevyc
Remove add_time0 function.
ed0445d 
Remove time_varname from yaml config file.
Assert time0 exists, and if not, help the user by listing the
nearest time available.

Make longitude the xaxis of cross section instead of 'index'.

Dequantify *after* cross section to preserve units.
Specify lat lon order of endpoints in cross_section_vert.py.

Remove dindex argument. xarray does a good job of choosing the
major axis interval.

Note vertical interpolation method in planview_fv3.py and reveal
actual pressure level in title instead of requested level.
@ahijevyc
update cross section png
917cdb2
@ahijevyc
fix .rst format warnings/errors
a94f8ec
@ahijevyc ahijevyc added type: enhancement Improve something that it is currently doing priority: medium Medium Priority requestor: NOAA/GSL NOAA Global Systems Laboratory METplotpy: Plots labels Aug 7, 2023
@ahijevyc ahijevyc added this to the Consider for Next Release milestone Aug 7, 2023
@ahijevyc ahijevyc requested a review from bikegeek August 7, 2023 22:09
@ahijevyc ahijevyc self-assigned this Aug 7, 2023
@ahijevyc ahijevyc linked an issue Aug 7, 2023 that may be closed by this pull request
Update fv3_physics_tend for new FV3 output format #380
Closed
33 tasks
@bikegeek
Copy link
Collaborator

bikegeek commented Aug 7, 2023

  1. Under the "Recommended Testing", there is a list item to upload new fv3_history.nc and grid_spec.nc, but in the documentation, the link is pointing to the data that was put on 'mohawk' on 8/2022. Are these new files different from the ones used in last year's development?

  2. You state that pylint recommends importing like this: from . rather than import physics_tend, but when using other linters, such as sonarlint (recommended for AF STIG) or ruff, this recommendation is not observed.

@ahijevyc
open ifile with dask. supply name to stacked dataarray to avaoid obsc…
4ae4719 
…ure default name like "stack-23412h4234kj"
@ahijevyc
don't drop=True when masking shapefile
b03bf90 
avoids error about Fancy indexing not implemented yet.

remove total area information. you'd have to mask it too, which
makes the code ugly.
@ahijevyc
Redo figures with colorbar labels that have field name.
d8eaff0 
And correct order of nophys and phys.

Dequantify before cross_section to avoid NotImplementedError: Don't yet support
 nd fancy indexing from cross_section(). something to do with dask and units.
@bikegeek
Copy link
Collaborator

I am following the instructions in the updated User's Guide, and using the updated fv3_history.nc (584M) and grid_spec.nc (481K), when attempting the following:

python planview_fv3.py $CONFIG $WORKING_DIR/fv3_history.nc $WORKING_DIR/grid_spec.nc tmp pbl
-p 500 -t 1 -v 20190615T20 --nofineprint? -p 500 -t 1 -v 20190615T20 --nofineprint

Error message:
Traceback (most recent call last):
File "/Volumes/d1/minnawin/feature_380_physics_tendency/METplotpy/metplotpy/contributed/fv3_physics_tend/planview_fv3.py", line 340, in
main()
File "/Volumes/d1/minnawin/feature_380_physics_tendency/METplotpy/metplotpy/contributed/fv3_physics_tend/planview_fv3.py", line 282, in main
"projection": cartopy.crs.LambertConformal(
File "/Volumes/d1/minnawin/miniconda3/envs/metplus_analysis_env/lib/python3.10/site-packages/cartopy/crs.py", line 1738, in init
n_parallels = len(standard_parallels)
TypeError: object of type 'NoneType' has no len()

I noticed there were some additional commits, so I made sure I had the latest changes in the branch I cloned and used for testing. Can you reproduce these errors?

@ahijevyc
Copy link
Contributor Author

ahijevyc commented Aug 22, 2023 via email

@ahijevyc
remove standard_parallels=None
b63c28f 
cartopy changed the default params of the LambertConformal class from
`standard_parallels = None` to `standard_parallels = (33, 45)` between v0.20.2 and v0.22.

I fixed the problem by removing `standard_parallels=None`.
@ahijevyc
note about central_lat in planview
e2d5693
@ahijevyc
Copy link
Contributor Author

ahijevyc commented Aug 22, 2023 via email

@bikegeek
Copy link
Collaborator

I re-ran that same command:
python planview_fv3.py $CONFIG $WORKING_DIR/fv3_history.nc $WORKING_DIR/grid_spec.nc tmp pbl -p 500 -t 1 -v 20190615T20 --nofineprint

and now I see this error:
Traceback (most recent call last):
File "/Volumes/d1/minnawin/feature_380_physics_tendency/METplotpy/metplotpy/contributed/fv3_physics_tend/planview_fv3.py", line 339, in
main()
File "/Volumes/d1/minnawin/feature_380_physics_tendency/METplotpy/metplotpy/contributed/fv3_physics_tend/planview_fv3.py", line 288, in main
pcm = da2plot.plot.pcolormesh(x="lont", y="latt", col=col, col_wrap=ncols, robust=robust,
File "/Volumes/d1/minnawin/miniconda3/envs/metplus_analysis_env/lib/python3.10/site-packages/xarray/plot/accessor.py", line 802, in pcolormesh
return dataarray_plot.pcolormesh(self._da, args, **kwargs)
File "/Volumes/d1/minnawin/miniconda3/envs/metplus_analysis_env/lib/python3.10/site-packages/xarray/plot/dataarray_plot.py", line 1495, in newplotfunc
return _easy_facetgrid(darray, kind="dataarray", **allargs)
File "/Volumes/d1/minnawin/miniconda3/envs/metplus_analysis_env/lib/python3.10/site-packages/xarray/plot/facetgrid.py", line 1059, in easy_facetgrid
return g.map_dataarray(plotfunc, x, y, **kwargs)
File "/Volumes/d1/minnawin/miniconda3/envs/metplus_analysis_env/lib/python3.10/site-packages/xarray/plot/facetgrid.py", line 341, in map_dataarray
func, self.data.to_numpy(), **kwargs
File "/Volumes/d1/minnawin/miniconda3/envs/metplus_analysis_env/lib/python3.10/site-packages/xarray/core/dataarray.py", line 749, in to_numpy
return self.variable.to_numpy()
File "/Volumes/d1/minnawin/miniconda3/envs/metplus_analysis_env/lib/python3.10/site-packages/xarray/core/variable.py", line 1298, in to_numpy
data, * = chunkmanager.compute(data)
File "/Volumes/d1/minnawin/miniconda3/envs/metplus_analysis_env/lib/python3.10/site-packages/xarray/core/daskmanager.py", line 70, in compute
return compute(data, **kwargs)
File "/Volumes/d1/minnawin/miniconda3/envs/metplus_analysis_env/lib/python3.10/site-packages/dask/base.py", line 599, in compute
results = schedule(dsk, keys, **kwargs)
File "/Volumes/d1/minnawin/miniconda3/envs/metplus_analysis_env/lib/python3.10/site-packages/dask/threaded.py", line 89, in get
results = get_async(
File "/Volumes/d1/minnawin/miniconda3/envs/metplus_analysis_env/lib/python3.10/site-packages/dask/local.py", line 511, in get_async
raise_exception(exc, tb)
File "/Volumes/d1/minnawin/miniconda3/envs/metplus_analysis_env/lib/python3.10/site-packages/dask/local.py", line 319, in reraise
raise exc
File "/Volumes/d1/minnawin/miniconda3/envs/metplus_analysis_env/lib/python3.10/site-packages/dask/local.py", line 224, in execute_task
result = _execute_task(task, data)
File "/Volumes/d1/minnawin/miniconda3/envs/metplus_analysis_env/lib/python3.10/site-packages/dask/core.py", line 119, in _execute_task
return func((_execute_task(a, cache) for a in args))
File "/Volumes/d1/minnawin/miniconda3/envs/metplus_analysis_env/lib/python3.10/site-packages/dask/core.py", line 119, in
return func((_execute_task(a, cache) for a in args))
File "/Volumes/d1/minnawin/miniconda3/envs/metplus_analysis_env/lib/python3.10/site-packages/dask/core.py", line 113, in _execute_task
return [_execute_task(a, cache) for a in arg]
File "/Volumes/d1/minnawin/miniconda3/envs/metplus_analysis_env/lib/python3.10/site-packages/dask/core.py", line 113, in
return [_execute_task(a, cache) for a in arg]
File "/Volumes/d1/minnawin/miniconda3/envs/metplus_analysis_env/lib/python3.10/site-packages/dask/core.py", line 119, in _execute_task
return func((_execute_task(a, cache) for a in args))
File "/Volumes/d1/minnawin/miniconda3/envs/metplus_analysis_env/lib/python3.10/site-packages/dask/optimization.py", line 990, in call
return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
File "/Volumes/d1/minnawin/miniconda3/envs/metplus_analysis_env/lib/python3.10/site-packages/dask/core.py", line 149, in get
result = _execute_task(task, cache)
File "/Volumes/d1/minnawin/miniconda3/envs/metplus_analysis_env/lib/python3.10/site-packages/dask/core.py", line 119, in _execute_task
return func((_execute_task(a, cache) for a in args))
File "/Volumes/d1/minnawin/miniconda3/envs/metplus_analysis_env/lib/python3.10/site-packages/dask/utils.py", line 73, in apply
return func(*args, **kwargs)
File "<array_function internals>", line 200, in sum
File "/Volumes/d1/minnawin/miniconda3/envs/metplus_analysis_env/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 2324, in sum
return _wrapreduction(a, np.add, 'sum', axis, dtype, out, keepdims=keepdims,
File "/Volumes/d1/minnawin/miniconda3/envs/metplus_analysis_env/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 86, in _wrapreduction
return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
TypeError: The dtype and signature arguments to ufuncs only select the general DType and not details such as the byte order or time unit. You can avoid this error by using the scalar types np.float64 or the dtype string notation.

Do you have the version numbers of the Python packages you are using? Here is a list of the Python packages and versions:
current.yaml.txt

@ahijevyc
Copy link
Contributor Author

ahijevyc commented Aug 22, 2023 via email

@ahijevyc
Copy link
Contributor Author

ahijevyc commented Aug 22, 2023 via email

@ahijevyc
revert to open_dataset from open_mfdataset
97a2c8e 
to avoid TypeError: The dtype and signature arguments to ufuncs only select the general DType and not details such as the byte order or time unit. You can avoid this error by using the scalar types np.float64 or the dtype string notation.

xarray 2022.3.0 was okay with how I handled xarray DataArrays with units (pint quantities), but xarray 2023.5.0 is not okay.
@ahijevyc
Copy link
Contributor Author

ahijevyc commented Aug 24, 2023 via email

@bikegeek bikegeek marked this pull request as draft March 25, 2024 16:16
@bikegeek
Copy link
Collaborator

bikegeek commented Jun 5, 2024

The planview and vert_profile_fv3 scripts work

but it looks like there is something wrong with the cross_section_vert.py:
python cross_section_vert.py $CONFIG $WORKING_DIR/fv3_history.nc $WORKING_DIR/grid_spec.nc tmp -t 2 -v 20190615T20 -s 32 -115 -e 34 -82 --nofineprint

produces this assertion error (line 144):
assert time0 in fv3ds.time, (f"time0 {time0} not in history file. Closest is "
AssertionError: time0 2019-06-15 18:00:00 not in history file. Closest is 2019-06-15T18:03:00.000000000

When no valid time is specified, the same error is generated. This is puzzling, since the fv3_history.nc file has
time:units = "hours since 2019-06-15 18:00:00" ;

bikegeek
bikegeek requested changes Jun 5, 2024
View reviewed changes
Copy link
Collaborator

@bikegeek bikegeek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are assertion errors for the cross_section_vert.py (see comment)

@ahijevyc
Copy link
Contributor Author

ahijevyc commented Jun 6, 2024

I will look at cross_section_vert.py today

@ahijevyc
Copy link
Contributor Author

ahijevyc commented Jun 6, 2024

The cross_section_vert.py test command should be
python cross_section_vert.py $CONFIG $WORKING_DIR/fv3_history.nc $WORKING_DIR/grid_spec.nc ugrd -t 1 -v "2019-06-15 20" -s 28 -120 -e 26 -75 --nofineprint
Note the -t 1 time window instead of -t 2.
-t 1.95 is another workable time window since the first time in the history file is 18:03 UTC, not 18:00 UTC.

I fixed a few matplotlib.pyplot depreciation errors in this script and two other scripts:

  • axes to axs
  • legendHandles to legend_handles
  • plt.cm.get_cmap('cool', 2) to colormaps['cool']

working on pushing the changes...

@ahijevyc
Copy link
Contributor Author

ahijevyc commented Jun 6, 2024

I pushed my changes to feature_380_phys_tend_plot_actual branch

@bikegeek
Copy link
Collaborator

bikegeek commented Jun 6, 2024 via email

@ahijevyc
Copy link
Contributor Author

ahijevyc commented Jun 6, 2024 via email

@bikegeek bikegeek marked this pull request as ready for review June 6, 2024 21:39
bikegeek
bikegeek approved these changes Jun 6, 2024
View reviewed changes
Copy link
Collaborator

@bikegeek bikegeek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated tests with new data pass (all tests are being skipped due to large dataset).

@bikegeek bikegeek merged commit a904351 into develop Jun 6, 2024
6 of 7 checks passed
@ahijevyc ahijevyc deleted the feature_380_phys_tend_plot_actual branch October 17, 2024 20:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bikegeek bikegeek bikegeek approved these changes

Assignees

@ahijevyc ahijevyc

Labels
METplotpy: Plots priority: medium Medium Priority requestor: NOAA/GSL NOAA Global Systems Laboratory type: enhancement Improve something that it is currently doing
Projects
METplus-Analysis-6.0.0 Development
Status: 🏁 Done
Milestone
METplotpy-3.0.0
Development

Successfully merging this pull request may close these issues.

Update fv3_physics_tend for new FV3 output format
2 participants
@ahijevyc @bikegeek