-
Notifications
You must be signed in to change notification settings - Fork 9
GLBt0.72
The GLBt0.72 grid is a global tripolar Arakawa C-grid of 0.72º horizontal resolution with refinement at the equator (500 cells on the zonal direction and 382 in the meridional direction). The bottom topography is derived from the 2-minute NAVO/Naval Research Laboratory DBDB2 global dataset.
In the GLBt0.72 directory, 10 subdirectories are provided:
GLBt0.72/archive contains scripts to manipulate/convert outputs of HYCOM.
GLBt0.72/datasets contains a script to download the domain-dependent atmospheric forcing, initial conditions, topography and grid. (N.B.: you can create most of them from the non domain-dependent files, i.e. ~/HYCOM-examples/datasets, and the scripts provided in the GLBt0.72 directory).
GLBt0.72/expt_01.0 is an example of running directory.
GLBt0.72/expt_01.1 is twin of expt_01.0 using on-the-fly calculation of the atmospheric forcing and src_2.3.00.
GLBt0.72/expt_01.2 is a twin of expt_01.1 for src_2.3.01. Note that old wndflg=5 (expt_01.1) was absolute winds, which is now wndflg=-5 (expt_01.2).
GLBt0.72/force contains example scripts to create domain-dependent CORE2 atmospheric files (N.B.: requires forcing products downloaded from ~/HYCOM-examples/datasets/get_datasets.csh).
GLBt0.72/meanstd contains scripts to calculate the mean, standard deviation and square of HYCOM archives.
GLBt0.72/postproc contains scripts called at the end of simulation to create mean files, converts to Netcdf and other examples.
GLBt0.72/relax contains scripts to create initial conditions, surface relaxation e-folding time file, thermobaricity file and variable thickness diffusion file.
GLBt0.72/topo contains scripts to create the grid and the multi-processor partitioning.
All the input files for the GLBt0.72 configuration can be downloaded from the HYCOM.org website through the GLBt0.72/datasets/get_datasets.csh script. To do so, edit the location of your scratch S and datasets DS directory.
All the atmospheric files (except the seawifs (kpar) and rivers files), the relax files and the grid file can be re-created with the scripts located in GLBt0.72/force, GLBt0.72/relax and GLBt0.72/topo respectively (See below for details).
The HYCOM sources can be cloned directly from GitHub in GLBt0.72/src_2.3.01-17tsig2_relo_mpi:
git clone --recursive https://github.com/HYCOM/HYCOM-src.git ~/HYCOM-examples/GLBt0.72/src_2.3.01-17tsig2_relo_mpi
See the HYCOM-src wiki page for compilation details.
To run your first experiment, we advise the user to leave GLBt0.72/expt_01.2 untouched and configure new_expt.csh in a new expt directory (i.e. expt_02.0) to run locally as a twin of this one.
~/HYCOM-examples/GLBt0.72> mkdir expt_02.0
~/HYCOM-examples/GLBt0.72> cp expt_01.2/new_expt.csh expt_02.0
~/HYCOM-examples/GLBt0.72> cd expt_02.0
Here are a few tips about the new_expt.csh script:
-
The new_expt.csh script creates a new expt from an existing one.
-
The two expt can be for different regions (set RO and RN to different regions).
-
The two expt can use different directory paths (e.g. DO and DN could be HYCOM-examples/${RO}/${EO} and hycom/${RN}/${EN}).
-
The scratch directory prefix must be separate from the permanent path, but differs from machine to machine (e.g. /p/work1/${user} or ${PN}/../../scratch).
-
The datasets directory (chosen in the GLBt072/datasets/get_datasets.csh) can be on scratch and need not include "datasets" (e.g. ~/${DN}/.. or ~/${DN}/../datasets or ${SCRN}/${DN}/../datasets).
-
Often the new case will use the same relax files as the old, if so uncomment the relax link.
-
Set the OS (Operating System) that is suitable for the type of machine you are using. User might need to go through the scripts to add options specific for their OS, such as the batch queues and running commands. Note that we only provide a submission script suitable for PBS queues (010pbs.csh), user on machine with a different scheduler needs to create their own submission script.
-
Our pre-2.3 scripts did not have a separate datasets directory (DS = ~/${DN}/..), and typically ran under ~/hycom (DN = hycom/${RN}/${EN}).
Once you finish editing the new_expt.csh, run the script and everything should be set up and ready to run!
Note that each expt directory contains a PRE and POST directory. The PRE directory has scripts to prepare the atmospheric forcing on-the-fly. This option is turn on with MAKE_FORCE at 1 in the new_expt.csh script. The POST directory has scripts to create tar files. When ARCHIVE is set to 1 in new_expt.csh, those tar files can be transferred automatically on an archive server of your choosing (edit the expt_XX.X/POST/*_rcp.csh and posproc/*_rcp.csh) at the end of the run.
Here are the different steps to create most of the input files necessary to run the GLBt0.72:
-
Create the grid:
Go to GLBt0.72/topo, edit the DS path in regional.grid.csh and run the script. -
Create the forcing file (the CORE2_NYF product is required for the scripts to work, see the HYCOM-examples wiki page for details):
Go to GLBt0.72/force, edit the non domain-dependent datasets directory DSR and the domain-dependent datasets directory DS for each script and run them. -
Create the relax file (the PHC3_HYCOM, WOA13_HYCOM or WOA18_HYCOM product is required for the scripts to work, see the HYCOM-examples wiki page for details):
Go to GLBt0.72/relax, edit the EXPR.src to set the path names, experiments number of the files to be created and the number of layers to use.
To create the horizontally variable sigma-target file (iso_sigma.[ab]): if needed, edit the vertical distribution of sigma target, and run the script.
To create the initial conditions: edit z_woa13.csh (to use WOA13_HYCOM), z_woa18.csh (to use WOA18_HYCOM) or z_levitus.csh (to use PHC3_HYCOM) and run it to create a horizontally interpolated files on their native vertical coordinates. Edit the blkdat.input if a new vertical distribution is wanted, edit the relax.csh and run the script.
To create sefold.csh, tbaric.csh and thkdf4.csh: if needed, edit the different scripts and run them.
Create the multi-processor partitioning:
Go to GLBt0.72/topo/partit/15
**Note that 15 corresponds to the topography number of the bathymetry file (i.e. depth_GLBt0.72_15.[ab]) and needs to be in the directory structure for the scripts to work. **
a) Edit and run depth_2d_Ssq.csh to generate candidate partitions
b) Edit and run size_S.csh to list the partitions in MPI task order (size_S.lis)
c) Edit and run ppm1.csh to softlink to the chosen partitions and make image maps
d) Edit and run resize8.csh to set the partition's 1st dimension for better vector performance
The resulting partitions are *s8:
-rw-r-----. 1 wallcraf 0375G018 518 Mar 23 19:02 depth_GLBt0.72_15.016s8
-rw-r-----. 1 wallcraf 0375G018 690 Mar 23 19:02 depth_GLBt0.72_15.024s8
-rw-r-----. 1 wallcraf 0375G018 776 Mar 23 19:02 depth_GLBt0.72_15.028s8
-rw-r-----. 1 wallcraf 0375G018 1094 Mar 23 19:02 depth_GLBt0.72_15.046s8
-rw-r-----. 1 wallcraf 0375G018 1462 Mar 23 19:02 depth_GLBt0.72_15.064s8
-rw-r-----. 1 wallcraf 0375G018 2040 Mar 23 19:02 depth_GLBt0.72_15.089s8
These were chosen for 16, 24, 32, or 48 cores per node, but will also work well with 2/4/8/12 cores per node. Most discard tiles that are entirely over land, and so don't necessarily fill the node. For example 046s8 is a 10x5 partition with 4 discarded tiles, so two cores will be unused on a 48-core node. We typically allocate unused cores to the 1st MPI node, but how to do this varies between MPI libraries.
> head -n 2 depth_GLBt0.72_15.046s8
npes npe mpe idm jdm ibig jbig nreg minsea maxsea avesea
46 10 5 500 382 52 77 3 0 3850 2536
For other domains:
- Use partit instead of partit_arctic for non-tripole grids
- Only tripole grids require npe to be an even divisor of idm
- For larger regions the ppm images from ppmX can be scaled down