-
Notifications
You must be signed in to change notification settings - Fork 15
Running FV3‐JEDI or MPAS‐JEDI ctest and setting up an experiment
- NOAA RDHPCS Hera
- NOAA RDHPCS Orion
- FV3
- MPAS
Follow the instruction at: https://github.com/NOAA-EMC/RDASApp/wiki/Clone-and-Build-RDASapp
The compiling step can load the modules needed for running the ctest. If running the ctest with a new terminal, extra steps are needed to load the modules:
module use modulefiles/
module load RDAS/hera.intel.lua # platform-dependent, see below
Three environmental files are available: hera.intel.lua hercules.intel.lua orion.intel.lua
export SLURM_ACCOUNT=$account # for tcsh use: setenv SLURM_ACCOUNT $account
cd RDASApp/build
ctest -VV -R rrfs_fv3jedi_hyb_2022052619
ctest -VV -R rrfs_fv3jedi_letkf_2022052619
ctest -VV -R rrfs_mpasjedi_2022052619_Ens3Dvar
ctest -VV -R rrfs_mpasjedi_2022052619_letkf
Where $account is the slurm resource account (e.g., fv3-cam, da-cpu, etc.). We note that more experiments will be added in the future.
When you build using the -r option, it will copy the staged RRFS JEDI data to the following paths:
RDASApp/bundle/rrfs-test-data/rrfs-data_fv3jedi_2022052619 # when DYCORE=FV3
RDASApp/bundle/rrfs-test-data/rrfs-data_mpasjedi_2022052619 # when DYCORE=MPAS
There is a convenient script to set up your experiment from the above data located at RDASApp/rrfs-test/scripts/setup_experiment.sh. Just make sure to edit the USER INPUT block of the script to set up the experiment to point to your installation of RDASApp as well as to the location of your desired experiment directory, the desired dycore for which you are setting up an experiment for, the platform you are running on, and finally your slurm account information.
cd RDASApp/rrfs-test/scripts
bash ./setup_experiment.sh # edit USER INPUT block before running.
In your newly created experiment directory, you will have a run script run_${dycore}jedi_${platform}.sh which was copied and edited in your directory by the setup_experiment.sh script. This script runs JEDI using the yaml file defined in setup_experiment.sh, but new tests in the same directory can easily be run by adding a new Yaml file as input to the run script:
sbatch run_${dycore}jedi_${platform}.sh $inputYaml
tail -f jedi.log
Where $platform is one of the supported options orion or hera, $dycore is one of fv3 or mpas, and $inputYaml is a yaml like the one provided in the test case testinput/rrfs_fv3jedi_hyb_2022052619.yaml or testinput/msonet_singleob_airTemperature_${dycore}jedi.yaml.
Simple scripts were created for demonstration purposes to visualize the single observation output (results from sbatch run_${dycore}jedi_${platform}.sh testinput/msonet_singleob_airTemperature_${dycore}jedi.yaml). For example, to plot single observation experiment results for FV3-JEDI:
cd rrfs-data_fv3jedi_2022052619
module use RDASApp/modulefiles
module load EVA/$platform
python fv3jedi_increment_singleob.py
An example plotting script is also included to plot increments for the full domain (i.e., not for single observation tests). Please note that the User Input section of the fv3jedi_increment_fulldom.py script needs to be edited whether you are plotting LETKF or variational-based results (either hybrid of EnVar). For example, to plot LETKF results:
cd rrfs-data_fv3jedi_2022052619
module use RDASApp/modulefiles
module load EVA/$platform
python fv3jedi_increment_fulldom.py
New ctests that use the same staged input data can be easily added through a two step process. First, a new yaml file needs to be created a put in RDASApp/rrfs-test/testinput. Second, you also need to modify RDASApp/rrfs-test/CMakeLists.txt under either the FV3_DYCORE or MPAS_DYCORE section, depending on which dynamical core this test is for. The new test name should be added under the rrfs_${dycore}jedi_tests list (the name will be the same as the new yaml file without the extension). Then, the corresponding executable also needs to be added under the rrfs_${dycore}jedi_exes list. Once these steps are done, you can rebuild RDASApp and run your new ctest.