Build JEDI environment on Gaea/C6.

[lliu1220]

Seeking support on building JEDI environment on C6. JEDI environment is currently only configured on C5 according to:
https://spack-stack.readthedocs.io/en/1.5.1/PreConfiguredSites.html

Were able to load most similar spack-stack modules of C5 on C6:
module load PrgEnv-intel/8.3.3
module load intel-classic/2023.1.0
module load cray-mpich/8.1.25
module load python/3.9.12

module use /lustre/f2/dev/wpo/role.epic/contrib/spack-stack/c5/modulefiles
module load ecflow/5.8.4
module load mysql/8.0.31

module use /lustre/f2/dev/wpo/role.epic/contrib/spack-stack/c5/spack-stack-1.5.1/envs/unified-env/install/modulefiles/Core
module load stack-intel/2023.1.0
module load stack-cray-mpich/8.1.25
module load stack-python/3.10.8

But the actual JEDI environment cannot be loaded:
module load jedi-fv3-env
module load ewok-env
module load soca-env
module load sp

for instance, when I tried to load JEDI FV3 ENvironemnt after preconfiguration, I got the following error:
home2/Ling.Liu> module load jedi-fv3-env
Lmod has detected the following error:
The following module(s) are unknown:
"jedi-fv3-env"

[travissluka]

spack stack 1.8 is supported on C6 , see https://spack-stack.readthedocs.io/en/1.8.0/PreConfiguredSites.html#noaa-rdhpcs-gaea-c6

[gspetro-NOAA]

Hi @lliu1220,

I believe @RatkoVasic-NOAA is looking into this for you in more detail. We also received your email, but I think it's best to continue the conversation here so that others can benefit from it.

Best,
Gillian Petro

[gspetro-NOAA]

I am posting @ulmononian 's email reply here so that others can benefit:

Hi Ling,

Currently, only spack-stack/1.6.0 and 1.8.0 are installed on Gaea C6. spack-stack versions installed on C5 will not work correctly on C6.

Gaea C5's OS and PrgEnv were just updated, so we are currently working to re-build the stacks on C5. Then, we will install spack-stack/1.5.1 on Gaea C6. We can notify you when this installation is ready for use. In the meantime, if you would like to try spack-stack/1.6.0 on C6 (this is the version currently used in the UFS-WM develop branch), it can be loaded as follows:

module load PrgEnv-intel/8.5.0
module load intel-classic/2023.2.0
module load cray-mpich/8.1.29
module load python/3.11

module use /ncrc/proj/epic/spack-stack/c6/spack-stack-1.6.0/envs/unified-env/install/modulefiles/Core
module load stack-intel/2023.2.0
module load stack-cray-mpich/8.1.29

To load the jedi, ewok, and soca environments, please also do:

module load jedi-fv3-env
module load ewok-env
module load soca-env

I hope this helps in the interim. As I mentioned, we will let you know when 1.5.1 is available on C6. Please let us know if you have additional questions.

Thank you,
Cameron

[lliu1220]

Thank you @gspetro-NOAA @travissluka @ulmononian .
I have tested on C6 on my end, so most of the modules loaded well except there is no: python/3.11
So I had substituted with python/3.10.13.
Then the ecbuild ran into a python error:
If you are familiar with this error and how to avoid it please let me know.

--------------------------------------------------------------------------------------
-- Configuring done
CMake Error at ioda/src/engines/ioda/CMakeLists.txt:316 (target_link_libraries):
  Target "ioda_engines" links to:
    Python3::Python
  but the target was not found.  Possible reasons include:

    * There is a typo in the target name.
    * A find_package call is missing for an IMPORTED target.
    * An ALIAS target is missing.
CMake Error at ioda/src/engines/ioda/python/CMakeLists.txt:28 (target_link_libraries):
  Target "_ioda_python" links to:
    Python3::Module
  but the target was not found.  Possible reasons include:
    * There is a typo in the target name.
    * A find_package call is missing for an IMPORTED target.
    * An ALIAS target is missing.
-- Generating done
CMake Generate step failed.  Build files cannot be regenerated correctly.

[ulmononian]

@lliu1220 please try module load stack-python before you load the jedi/ewok/soca environments, and then try to build again and let us know if anything changes.

[lliu1220]

So , I turned off IODA-ENGINE and the compile worked (just loading stack-python wouldn't do).
however, when I try to run a simple testcase, this is the error I got:

roms-jedi/test> srun -M c6 -A drsa-hurr3 -t 00:05:00 -p batch -N 1 -n 1 test_romsjedi_geometry testinput/geometry.yaml 
srun: job 207216790 queued and waiting for resources
srun: job 207216790 has been allocated resources
OOPS Starting 2024-10-23 14:11:45 (UTC-0400)
Configuration input file is: testinput/geometry.yaml
Full configuration is:YAMLConfiguration[path=testinput/geometry.yaml, root={geometry => {project_dir => Data/wc13 , roms_stdinp => roms_wc13_20040103.in , fields metadata => Data/fields_metadata.yaml , ng => 1 , write to gmsh => false , gmsh filename => wc13.msh}}]
OOPS_STATS ObjectCountHelper started.
OOPS_STATS Run start                                - Runtime:      0.08 sec,  Memory: total:    98.50 Mb, per task: min =    98.50 Mb, max =    98.50 Mb
Run: Starting oops::Test running test::Geometry<ROMSJEDI>
Running 2 tests:
Running case 0: interface/Geometry/testConstructor ...
--------------------------------------------------------------------------------
 Model Input Parameters:  ROMS/TOMS version 4.2  
                          Wednesday - October 23, 2024 -  2:11:45 PM
--------------------------------------------------------------------------------
forrtl: severe (153): allocatable array or pointer is not allocated
Image              PC                Routine            Line        Source             
libroms.so         00007F5A7243398A  get_metadata_mod_         976  get_metadata.f90
libroms.so         00007F5A72134230  mod_ncparam_mp_in        1175  mod_ncparam.f90
libroms.so         00007F5A7251DF9E  read_phypar_              240  read_phypar.f90
libroms.so         00007F5A72456729  inp_par_mod_mp_in         101  inp_par.f90
libroms.so         00007F5A720E39AF  roms_kernel_mod_m         139  roms_kernel.f90
libromsjedi.so     00007F5A8BC91A37  roms_geom_mod_mp_     Unknown  Unknown
libromsjedi.so     00007F5A8BC7D94D  roms_geom_init_f9     Unknown  Unknown
libromsjedi.so     00007F5A8BC644E5  _ZN8romsjedi8Geom     Unknown  Unknown
test_romsjedi_geo  000000000040ECD8  Unknown               Unknown  Unknown
test_romsjedi_geo  000000000040E87C  Unknown               Unknown  Unknown
test_romsjedi_geo  0000000000413598  Unknown               Unknown  Unknown
test_romsjedi_geo  0000000000412FC7  Unknown               Unknown  Unknown
test_romsjedi_geo  000000000040AAB4  Unknown               Unknown  Unknown
liboops.so         00007F5A8421D608  _ZN4oops3Run7exec     Unknown  Unknown
test_romsjedi_geo  000000000040A889  Unknown               Unknown  Unknown
libc-2.31.so       00007F5A71EDE24D  __libc_start_main     Unknown  Unknown
test_romsjedi_geo  000000000040A74A  Unknown               Unknown  Unknown
[CRAYBLAS_WARNING] **Application linked against multiple cray-libsci libraries**
srun: error: c6n0200: task 0: Exited with exit code 153
srun: Terminating StepId=207216790.0

[ulmononian]

@lliu1220 not sure if this will work w/ this specific problem, but please try doing the following before compiling/running (in the ufs-wm, it is done in the modulefiles, which are loaded before compile and runtime):

unload("darshan-runtime")
unload("cray-libsci")

[gspetro-NOAA]

@lliu1220 Just wanted to check in and see if @ulmononian 's suggestions resolved the problem for you or if you're still seeing issues?

[lliu1220]

@gspetro-NOAA
@ulmononian 's suggestions are helpful, I was able to load the environment and build ROMS-JEDI following these suggestions.
The new error message above is a srun error, which has not yet been resolved.
I have reached out to C6 helpdesk and awaiting for a response.

[ulmononian]

@lliu1220 did you have any luck trying

unload("darshan-runtime")
unload("cray-libsci")

before running the model? you may have to recompile with the cray-libsci unloaded as well, and then unload before running.

[lliu1220]

@ulmononian I did try what you suggested on the cray libsci, it didn't matter; the other module was never loaded.
C6 updated python to :

python/3.10.13
stack-python/3.10.13

then the compile issue is outstanding again:

-- The following OPTIONAL packages have not been found:

 * gsibec (required version >= 1.2.1)
 * bufr_query
 * Python3
 * crtm (required version >= 2.4.0)
 * rttov (required version >= 12.1.0)
 * ropp-ufo
 * geos-aero
 * oasim

-- Configuring done
CMake Error at ioda/src/engines/ioda/CMakeLists.txt:316 (target_link_libraries):
  Target "ioda_engines" links to:

    Python3::Python

  but the target was not found.  Possible reasons include:

    * There is a typo in the target name.
    * A find_package call is missing for an IMPORTED target.
    * An ALIAS target is missing.



CMake Error at ioda/src/engines/ioda/python/CMakeLists.txt:28 (target_link_libraries):
  Target "_ioda_python" links to:

    Python3::Module

  but the target was not found.  Possible reasons include:

    * There is a typo in the target name.
    * A find_package call is missing for an IMPORTED target.
    * An ALIAS target is missing.



-- Generating done
CMake Generate step failed.  Build files cannot be regenerated correctly.

[ulmononian]

@lliu1220 can you please reiterate what the current problem is? it sounded like it was compiling, but failing to run, but now you again cannot compile? if you share the build/run steps, i can ask around to see if we can find a solution.

[lliu1220]

@ulmononian @RatkoVasic-NOAA, adding @IGRivin -
I just tried to compile and tested again.
NOW I was able to compile with python binding turned OFF, so I don't run into IODA engine error.- But this could cause problem later, so far the error was unrelated.
Then when I test it. I ran into this different SRUN error. Maybe my srun line wasn't correct?

srun -M c6 -A drsa-hurr3 -t 00:05:00 -p patch -N 1 -n 1 test_romsjedi_geometry testinput/geometry.yaml

I tried two partitions:

C6 and batch;
es and eslogin

If any of you ever had this srun error please write something here. thank you

roms-jedi/test> srun -M c6 -A drsa-hurr3 -t 00:05:00 -p patch -N 1 -n 1 test_romsjedi_geometry testinput/geometry.yaml
srun: job 67852101 queued and waiting for resources
srun: job 67852101 has been allocated resources
srun: error: Task launch for StepId=67852101.0 failed on node gaea62: Header lengths are longer than data received
srun: error: Application launch failed: Header lengths are longer than data received
srun: Job step aborted
roms-jedi/test> 

Here is my entire module load for your testing, notice loading stack python will automatically update python to the latest:

module load PrgEnv-intel/8.5.0
module load intel-classic/2023.2.0
module load cray-mpich/8.1.29

module use /ncrc/proj/epic/spack-stack/c6/spack-stack-1.6.0/envs/unified-env/install/modulefiles/Core
module load stack-intel/2023.2.0
module load stack-cray-mpich/8.1.29
module load stack-python

module load jedi-fv3-env
module load ewok-env
module load soca-env

ecbuild -DMPIEXEC_EXECUTABLE=/usr/bin/srun -DMPIEXEC_NUMPROC_FLAG="-n" -DUSE_ECCODES=ON -DBUILD_IODA_CONVERTERS=OFF -DBUILD_PYTHON_BINDINGS=OFF  -DENABLE_LORENZ95_MODEL=OFF -DENABLE_QG_MODEL=OFF -DBUNDLE_SKIP_ROPP-UFO=OFF  -DCMAKE_BUILD_TYPE=Release ../Bundle_wc13

[lliu1220]

@gspetro-NOAA @ulmononian @travissluka @IGRivin
So we worked with C6 helpdesk and found out that ROMS won't work with intel/2023.2.0

Please see following discussion for proof:
https://www.myroms.org/forum/viewtopic.php?p=24931&hilit=forrtl%3A+severe+153+%3A#p24931

I saw you mentioned spack-stack 1.5.1 will become available on C6, may I ask if it will be with a different compiler, and when it will become available?

Thank you.

[RatkoVasic-NOAA]

@lliu1220 on C6 spack-stack/1.5.1 is available only with intel/2023.2.0
/ncrc/proj/epic/spack-stack/c6/spack-stack-1.5.1/envs/unified-env/install/modulefiles/Core

I see one other classic intel compiler available: intel/2022.1.0
We can try to install spack-stack/1.5.1 with 2022.1.0 on Gaea-C6. I'll let you know if that one installed OK (1.5.1 is soon going away, and cannot spend too much time on that).

[lliu1220]

@RatkoVasic-NOAA
Ok, Thank you.
Our colleague has tested intel/2024.0 would work with ROMS, we aren't sure about 2022.1.0, but would be good to have an option to test.

module load PrgEnv-intel/8.5.0 cray-hdf5 cray-netcdf
module swap intel/2024.0

So looking forward to it, thanks.

[RatkoVasic-NOAA]

OK, I'll try with intel/2024.0

[RatkoVasic-NOAA]

Installation on Gaea C6, spack-stack-1.5.1 using Intel compiler 2024.0 didn't work:
/ncrc/proj/epic/spack-stack/c6/spack-stack-1.5.1/envs/ue-intel-2024.0/

[lliu1220]

Thank you for your response @RatkoVasic-NOAA .
Is it possible to try spack-stack/1.5.1 with 2022.1.0 also?
Many thanks.
P.S. If you have written instruction handy and able to share, I could try install 1.5.1 /2022.1.0 on my own as well, so to save your time.