Changes made in the code sprint into develop (#14)

* Add some hofx test data (#6) Co-authored-by: danholdaway <danholdaway@users.noreply.github.com> * Add in declarative plotting (#5) * Create figure.py * Add in the rest of declarative plotting * Fix missing EOL * pycodestyle * more pycodestyle * Remove mstats * Add missing save_figure method Co-authored-by: Dan Holdaway <27729500+danholdaway@users.noreply.github.com> * Changes to ObsCorrelationScatter to use new plotting methods (#7) * Create figure.py * Add in the rest of declarative plotting * Fix missing EOL * pycodestyle * more pycodestyle * Update obs_correlation_scatter.py * Remove mstats * Add missing save_figure method * Finish thought * Changes to plot * Small fixes * make dots blue * pycodestyle * Remove old plot routine * Move diagnostics into new, separate directory (#8) * Start to move file * add init file; modify base.py to import from eva.diagnostics * pycodestyle Co-authored-by: Dan Holdaway <27729500+danholdaway@users.noreply.github.com> * apt get CI update (#10) * Feature/new api (#9) * move to single argument api * update logger passing * norms * review comments * update requirements * req typo and review comment * req typo pasted * try removing cartopy version * cartopy 0.19 * cartopy 0.19 * fix cartopy version Co-authored-by: danholdaway <danholdaway@users.noreply.github.com> * Fix colorbar issue in `figure.py` (#11) * address colorbar issue * added note about using single colorbar * pycodestyle * Remove deprecated base class file (#13) Co-authored-by: danholdaway <danholdaway@users.noreply.github.com> * Feature/add unittests (#12) * feat: introduce unit test for the ObsCorrelationScatter method body: This unit test relies on a new method for loading yaml files which allows the user to create a substution for an environment variable identified by the yaml tag !ENVVAR . If the tag !ENVVAR exists before a yaml key value, and the environment variable specified between the {} brackets is set, the yaml load routine will substitude whatever is set in the specified environment variable. For instance, if one set an environment variable 'EVA_TESTS_DIR' to the following: export EVA_TESTS_DIR=/absolute/path/to/eva/install/location the the yaml load routine will change the following yaml key yaml_key: !ENVVAR ${EVA_TESTS_DIR}/data/datafile.nc4 to yaml_key: /absolute/path/to/eva/install/location/data/datafile.nc4 * refactor: remove unnecessary logging and associated lines * fix: update test yaml config to adopt the new format body: due to a change made to the expected format of the config yaml format, I had to fix a few yaml keys (i.e. the yaml key 'diagnostics' needed to be changed to 'applications'. I also fixed a yaml subkey 'diagnostic name' to 'application name'. * fix: accidentally moved several print statements, moving them back * fix: making changes to comply with pycodestyle * refactor: move and modify yaml loading routine body: This commit is to revert the yaml loading routine to what it was prior to the unit test addition, however it is moved to it's own method so that the unit test can substitute it's own version of 'load_yaml_file'. The logger is added to the unit test as well as the new function'load_yaml_file'. * fix: fixing bad indentation within a helper function Co-authored-by: danholdaway <danholdaway@users.noreply.github.com> Co-authored-by: Cory Martin <cory.r.martin@noaa.gov> Co-authored-by: kevindougherty-noaa <69815622+kevindougherty-noaa@users.noreply.github.com> Co-authored-by: Steve Lawrence <57503604+slawrence87544@users.noreply.github.com>
JCSDA-internal · Feb 23, 2022 · 10774b1 · 10774b1
1 parent 3c08671
commit 10774b1
Show file tree

Hide file tree

Showing 22 changed files with 1,951 additions and 252 deletions.
diff --git a/.gitattributes b/.gitattributes
@@ -0,0 +1,4 @@
+*.nc filter=lfs diff=lfs merge=lfs -text
+*.nc4 filter=lfs diff=lfs merge=lfs -text
+*.res filter=lfs diff=lfs merge=lfs -text
+*.odb filter=lfs diff=lfs merge=lfs -text
diff --git a/.github/workflows/codestyle.yml b/.github/workflows/codestyle.yml
@@ -9,6 +9,11 @@ jobs:
     name: Check Python Coding Norms
     runs-on: ubuntu-latest
     steps:
+    - name: Install ubuntu dependencies
+      run: |
+        sudo apt-get install libproj-dev proj-data proj-bin
+        sudo apt-get install libgeos-dev musl-dev libc-dev
+        sudo ln -s /usr/lib/x86_64-linux-musl/libc.so /lib/libc.musl-x86_64.so.1
     - uses: actions/checkout@v2
     - uses: actions/setup-python@v2
       with:

diff --git a/README.md b/README.md
@@ -1,3 +1,4 @@
+
 # Evaluation and Verification of the Analysis (EVA)
 
 ### Continuous integration:
@@ -6,6 +7,14 @@
 | --------- | --------|
 | Python coding norms | ![Status](https://github.com/danholdaway/eva/actions/workflows/codestyle.yml/badge.svg) |
 
+### Licence:
+
+(C) Copyright 2021-2022 United States Government as represented by the Administrator of the National
+Aeronautics and Space Administration. All Rights Reserved.
+
+[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
+
+
 ### Installation
 
 The eva package can be installed using pip:
@@ -14,30 +23,28 @@ The eva package can be installed using pip:
 
 ### Usage
 
-eva uses a strict dictionary only API. This ensures flexible use for different applications. The most straightforward use is achieved by choosing the Class containing the diagnostic and passing a yaml configuration file:
-
-	eva ObsCorrelationScatter obs_correlation.yaml
-
-Alternatively you can invoke eva passing the class name within the yaml configuration:
+eva uses a strict dictionary/configuration only API. This ensures flexible use for different applications. The most straightforward use of eva is achieved by passing it a YAML configuration file on the command line:
 
 	eva obs_correlation.yaml
 
-Where the yaml must contain a list of the diagnostics to be used:
+Where the YAML must contain a list of the diagnostics to be used in the following format:
 
 ```
-applications:
-  - application name: ObsCorrelationScatter
+diagnostics:
+  - diagnostic name: ObsCorrelationScatter
     ...
-  - application name: ObsMapScatter
+  - diagnostic name: ObsMapScatter
     ...
 ```
 
-eva can also be invoked from another Python module that passes a dictionary, rather than a Yaml file that is later translated into a dictionary. This is achieved as follows:
+eva can also be invoked from another Python module that passes it a dictionary, rather than a YAML file that is later translated into a dictionary. This is achieved as follows:
 
 ```
-from eva.base import create_and_run
+from eva.eva_base import eva
 
-create_and_run("ObsCorrelationScatter", eva_dict)
+eva(eva_dict)
 ```
 
-Note that this also allows for use of eva within Jupyter notebooks.
+The dictionary must take the same hierarchy as shown above in the YAML file, i.e. with a list of diagnostics to be run. Note that the calling routine can still pass a string with a path to the YAML file if so desired.
+
+Note that eva can also be invoked within Jupyter notebooks using the above API.
diff --git a/requirements.txt b/requirements.txt
@@ -2,3 +2,5 @@ pyyaml>=5.4
 pycodestyle>=2.8.0
 netCDF4>=1.5.7
 matplotlib>=3.4.3
+cartopy==0.19.0.post1
+scikit-learn>=1.0.2
diff --git a/setup.py b/setup.py
@@ -36,10 +36,18 @@
         'pycodestyle>=2.8.0',
         'netCDF4>=1.5.7',
         'matplotlib>=3.4.3',
+        'cartopy==0.19.0.post1',
+        'scikit-learn>=1.0.2',
     ],
+    package_data={
+        '': [
+               'tests/config/*',
+               'tests/data/*',
+             ],
+    },
     entry_points={
         'console_scripts': [
-            'eva = eva.base:main',
+            'eva = eva.eva_base:main',
         ],
     },
     )

diff --git a/src/eva/base.py b/src/eva/base.py
diff --git a/src/eva/diagnostics/__init__.py b/src/eva/diagnostics/__init__.py
@@ -0,0 +1,9 @@
+# (C) Copyright 2021-2022 United States Government as represented by the Administrator of the
+# National Aeronautics and Space Administration. All Rights Reserved.
+#
+# This software is licensed under the terms of the Apache Licence Version 2.0
+# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
+
+import os
+
+repo_directory = os.path.dirname(__file__)
diff --git a/src/eva/obs_correlation_scatter.py → ...va/diagnostics/obs_correlation_scatter.py b/src/eva/obs_correlation_scatter.py → ...va/diagnostics/obs_correlation_scatter.py
@@ -1,3 +1,5 @@
+# (C) Copyright 2021-2022 NOAA/NWS/EMC
+#
 # (C) Copyright 2021-2022 United States Government as represented by the Administrator of the
 # National Aeronautics and Space Administration. All Rights Reserved.
 #
@@ -8,10 +10,11 @@
 # --------------------------------------------------------------------------------------------------
 
 
-from eva.base import Base
+from eva.eva_base import EvaBase
 from eva.utilities import ioda_definitions
 from eva.utilities import ioda_netcdf_api
-from eva.plot_tools.scatter_correlation import scatter_correlation_plot
+from eva.plot_tools.figure import CreatePlot, CreateFigure
+from eva.plot_tools.plots import Scatter, LinePlot
 
 import netCDF4
 import numpy as np
@@ -23,7 +26,7 @@
 
 # TODO: needs to be ioda-erized and r2d2-erized
 
-class ObsCorrelationScatter(Base):
+class ObsCorrelationScatter(EvaBase):
 
     def execute(self):
 
@@ -179,9 +182,30 @@ def execute(self):
                             plot_title = platform_long_name + ' | ' + variable_name_no_
 
                             # Create the plot
-                            scatter_correlation_plot(data_ref, data_exp, ref_metric_long_name,
-                                                     exp_metric_long_name, plot_title, output_file,
-                                                     marker_size=marker_size)
+                            # set up the scatter layer
+                            scatter = Scatter(data_ref, data_exp)
+                            scatter.markersize = marker_size
+                            scatter.color = 'blue'
+                            data_min = min(min(data_ref), min(data_exp))
+                            data_max = max(max(data_ref), max(data_exp))
+                            data_diff = data_max - data_min
+                            plotmin = data_min - (0.1 * data_diff)
+                            plotmax = data_max + (0.1 * data_diff)
+                            # add a 1:1 line layer
+                            line = LinePlot([plotmin, plotmax], [plotmin, plotmax])
+                            line.color = 'black'
+                            # set up the plot
+                            plot = CreatePlot(plot_layers=[line, scatter])
+                            plot.add_title(plot_title)
+                            plot.set_xlim([plotmin, plotmax])
+                            plot.set_ylim([plotmin, plotmax])
+                            plot.add_xlabel(ref_metric_long_name)
+                            plot.add_ylabel(exp_metric_long_name)
+                            # create the figure
+                            fig = CreateFigure(figsize=(8, 8))
+                            fig.plot_list = [plot]
+                            fig.create_figure()
+                            fig.save_figure(output_file)
 
             # Close files
             fh_exp.close()