Dummy PR for CI Test on Gaea C5

[AnilKumar-NOAA]

Description

• CI Test for Gaea C5 system

Type of change

• Bug fix (fixes something broken)
• New feature (adds functionality)
• Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

• Is this a breaking change (a change in existing functionality)? YES/NO
• Does this change require a documentation update? YES/NO
• Does this change require an update to any of the following submodules? YES/NO (If YES, please add a link to any PRs that are pending.)
  • EMC verif-global
  • GDAS
  • GFS-utils
  • GSI
  • GSI-monitor
  • GSI-utils
  • UFS-utils
  • UFS-weather-model
  • wxflow

How has this been tested?

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have documented my code, including function, input, and output descriptions
• My changes generate no new warnings
• New and existing tests pass with my changes
• This change is covered by an existing CI test or a new one has been added
• I have made corresponding changes to the system documentation if necessary

[TerrenceMcGuinness-NOAA]

The Jinja2 python module is not availed in the shell environment that the Gaea Node is configured to:

+ /gpfs/f5/epic/proj-shared/global/CI/2964/gfs/ci/scripts/utils/get_host_case_list.py gaea
Traceback (most recent call last):
  File "/gpfs/f5/epic/proj-shared/global/CI/2964/gfs/ci/scripts/utils/get_host_case_list.py", line 6, in <module>
    from wxflow import parse_j2yaml
  File "/gpfs/f5/epic/proj-shared/global/CI/2964/gfs/ci/scripts/utils/wxflow/__init__.py", line [1](https://jenkins.epic.oarcloud.noaa.gov/job/global-workflow/job/EMC-Global-Pipeline/view/change-requests/job/PR-2964/1/pipeline-console/?start-byte=0&selected-node=113#log-1)3, in <module>
    from .jinja import Jinja
  File "/gpfs/f5/epic/proj-shared/global/CI/[2](https://jenkins.epic.oarcloud.noaa.gov/job/global-workflow/job/EMC-Global-Pipeline/view/change-requests/job/PR-2964/1/pipeline-console/?start-byte=0&selected-node=113#log-2)964/gfs/ci/scripts/utils/wxflow/jinja.py", line 7, in <module>
    import jinja2
ModuleNotFoundError: No module named 'jinja2'
script returned exit code 1

It can be found here.

[AnilKumar-NOAA]

Thanks Terry! Will check with Kris and will sort out jinja2 configuration issue.

[kbooker79]

Thanks Terry! Will check with Kris and will sort out jinja2 configuration issue.

@TerrenceMcGuinness-NOAA, is it expected that the py-jinja2 module be loaded into the Jenkins runner? This seems like something that the shell environment here should load at runtime.

[TerrenceMcGuinness-NOAA]

@kbooker79 It is required that the shell that the Jenkins Node is connected to be configured with all the modules specified in ${HOMEgfs}/modulefiles/module_gwsetup.${MACHINE_ID}.lua for Global-Workflow. This is done on the EMC Nodes (here) in the shell that launches the Java Agent to Jenkins. Which would be similar to Method 2 outlined (here)

[kbooker79]

@kbooker79 It is required that the shell that the Jenkins Node is connected to be configured with all the modules specified in ${HOMEgfs}/modulefiles/module_gwsetup.${MACHINE_ID}.lua for Global-Workflow. This is done on the EMC Nodes (here) in the shell that launches the Java Agent to Jenkins. Which would be similar to Method 2 outlined (here)

@TerrenceMcGuinness-NOAA, can we try using method 1, and just have Jenkins set the environment variable MODULE_GWSETUP_PATH=$HOMEgfs/modulefiles, then load the module path and module for the node module_gwsetup.gaea at runtime? My only issue with method 2 is that it sets certain module persistently in a shell environment that is shared by other teams. I hope this makes sense.

[AnilKumar-NOAA]

@TerrenceMcGuinness-NOAA Please re-run the PR again. Thanks

[kbooker79]

@kbooker79 It is required that the shell that the Jenkins Node is connected to be configured with all the modules specified in ${HOMEgfs}/modulefiles/module_gwsetup.${MACHINE_ID}.lua for Global-Workflow. This is done on the EMC Nodes (here) in the shell that launches the Java Agent to Jenkins. Which would be similar to Method 2 outlined (here)

@TerrenceMcGuinness-NOAA @AnilKumar-NOAA, I've reconfigured Gaea to use Method2 instead, if $HOMEgfs (and subsequently MODULE_GWSETUP_PATH should be) it is set properly this should work now. Please re-run this workflow on Gaea and let me know if you have questions.

[TerrenceMcGuinness-NOAA]

It still doesn't appear the shell running on Gaea is getting the gwsetup module loaded:

+ /gpfs/f5/epic/proj-shared/global/CI/2964/gfs/ci/scripts/utils/get_host_case_list.py gaea
Traceback (most recent call last):
  File "/gpfs/f5/epic/proj-shared/global/CI/2964/gfs/ci/scripts/utils/get_host_case_list.py", line 6, in <module>
    from wxflow import parse_j2yaml
  File "/gpfs/f5/epic/proj-shared/global/CI/2964/gfs/ci/scripts/utils/wxflow/__init__.py", line [1](https://jenkins.epic.oarcloud.noaa.gov/job/global-workflow/job/EMC-Global-Pipeline/view/change-requests/job/PR-2964/1/pipeline-console/?start-byte=0&selected-node=113#log-1)3, in <module>
    from .jinja import Jinja
  File "/gpfs/f5/epic/proj-shared/global/CI/[2](https://jenkins.epic.oarcloud.noaa.gov/job/global-workflow/job/EMC-Global-Pipeline/view/change-requests/job/PR-2964/1/pipeline-console/?start-byte=0&selected-node=113#log-2)964/gfs/ci/scripts/utils/wxflow/jinja.py", line 7, in <module>
    import jinja2
ModuleNotFoundError: No module named 'jinja2'
script returned exit code 1

as such:

Terry.McGuinness@gaea54:~/NOAA/global-workflow/modulefiles> module purge
Terry.McGuinness@gaea54:~/NOAA/global-workflow/modulefiles> module use $PWD
Terry.McGuinness@gaea54:~/NOAA/global-workflow/modulefiles> module load module_gwsetup.gaea 
Terry.McGuinness@gaea54:~/NOAA/global-workflow/modulefiles> python -c "import jinja2"

[DavidBurrows-NCO]

@TerrenceMcGuinness-NOAA It looks like you are using module purge on Gaea which may lead to issues (from the Gaea docs): "Do not use the command module purge on Gaea. This will remove all modules currently loaded by default in your environment and may lead to errors, and can make your session unusable. If you do run the command to purge your module environment, log out and back in to Gaea to restore the default environment with the default modules loaded."

[WalterKolczynski-NOAA]

@TerrenceMcGuinness-NOAA It looks like you are using module purge on Gaea which may lead to issues (from the Gaea docs): "Do not use the command module purge on Gaea. This will remove all modules currently loaded by default in your environment and may lead to errors, and can make your session unusable. If you do run the command to purge your module environment, log out and back in to Gaea to restore the default environment with the default modules loaded."

module-setup.sh is using module reset instead of module purge, which is our preferred method anyway.

[AnilKumar-NOAA]

For CI test - We have updated module setup for Gaea. Some issue with old module path and need to be updated to fix all environments for gaea system

[WalterKolczynski-NOAA]

Where does this one stand? Does it need the Gaea CI Ready label reapplied? Should it be closed?

[AnilKumar-NOAA]

we are still working on it. It's just on hold because gaea c5 has some OS upgrades with spack stack changes. We submitted all the PR's related to all these changes. Soon these PR's will be merged and we will start working on CI test after that.

[WalterKolczynski-NOAA]

Marked this as draft since it is just a dummy PR.

[AnilKumar-NOAA]

Closing this PR. I'll create another dummy PR once we are ready for CI test on Gaea system after all OS upgrade PR's are merged. Thanks