Skip to content

Running FV3‐JEDI or MPAS‐JEDI ctest and setting up an experiment

Jump to bottom
Samuel Degelia edited this page May 2, 2024 · 17 revisions

Running FV3-JEDI or MPAS-JEDI ctest and setting up an experiment

Currently supported platforms

  • NOAA RDHPCS Hera
  • NOAA RDHPCS Orion

Currently supported dycores

  • FV3
  • MPAS

Cloning and build RDASApp

Follow the instruction at: https://github.com/NOAA-EMC/RDASApp/wiki/Clone-and-Build-RDASapp

Loading modules for ctest

The compiling step can load the modules needed for running the ctest. If running the ctest with a new terminal, extra steps are needed to load the modules:

module use modulefiles/
module load RDAS/hera.intel.lua` # platform-dependent, see below

Three environmental files are available: hera.intel.lua hercules.intel.lua orion.intel.lua

Running ctest

export SLURM_ACCOUNT=$account # for tcsh use: setenv SLURM_ACCOUNT $account
cd RDASApp/build
ctest -VV -R rrfs_fv3jedi_hyb_2022052619
ctest -VV -R rrfs_mpasjedi_2022052619_Ens3Dvar

Where $account is the slurm resource account (e.g., fv3-cam, da-cpu, etc.).

Getting test case data

When you build using the -r option, it will copy the staged RRFS JEDI data to the following paths. More experiments will be added in the future.

RDASApp/build/rrfs-test/rundir-rrfs_fv3jedi_hyb_2022052619 # when DYCORE=FV3
RDASApp/build/rrfs-test/rundir-rrfs_mpasjedi_2022052619_Ens3Dvar # when DYCORE=MPAS

There is a convenient script to set up your experiment from the ctest data located in the following path. Just make sure to edit the USER INPUT block of the script to set up the experiment to point to your installation of RDASApp as well as to the location of your desired experiment directory, the desired dycore for which you are setting up an experiment for, the platform you are running on, and finally your slurm account information.

cd RDASApp/rrfs-test/scripts
bash ./setup_experiment.sh # edit USER INPUT block before running.

Running your own experiments with the ctest case

In your newly created experiment directory, you will have a run script run_${dycore}jedi_${platform}.sh which was copied and edited in your directory by the setup_experiment.sh script. Just provide this script with a yaml configuration and that is all you need to run a JEDI analysis.

sbatch run_${dycore}jedi_${platform}.sh $inputYaml
tail -f jedi.log

Where $platform is one of the supported options orion or hera, $dycore is one of fv3 or mpas, and $inputYaml is a yaml like the one provided in the test case testinput/rrfs_fv3jedi_hyb_2022052619.yaml or testinput/msonet_singleob_airTemperature_${dycore}jedi.yaml.

Visualizing results

Simple scripts were created for demonstration purposes to visualize the single ob output (results from sbatch run_${dycore}jedi_${platform}.sh testinput/msonet_singleob_airTemperature_${dycore}jedi.yaml). For example, to plot results for FV3-JEDI:

cd rundir-rrfs_fv3jedi_hyb_2022052619
module use RDASApp/modulefiles
module load EVA/$platform
python fv3jedi-increment.py

Please note that sometimes you may have to run module load EVA/$platform twice. It should show (eva) to denote that the EVA conda environment has been activated. You can use this as a guide for visualizing your own results.

Clone this wiki locally