Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug fix for MERRA2 coupled Thompson microphysics and UPP update #1915

Merged
merged 19 commits into from
Oct 18, 2023

Conversation

AnningCheng-NOAA
Copy link
Contributor

@AnningCheng-NOAA AnningCheng-NOAA commented Sep 21, 2023

PR Author Checklist:

Description

  • A bug fixed for MERRA2 coupled Thompson microphysics. The cloud water number concentration was not passed back when calling the microphysics, resulting model crash for some hurricane cases.
  • Inline post UPP update: grib2 datasets GFSPRS.GrbF*, NATLEV*, PRSLEV* to support GFSV17 HR3

Linked Issues and Pull Requests

Associated UFSWM Issue to close

related to issue 1914: #1914

Subcomponent Pull Requests

Blocking Dependencies

Subcomponents involved:

  • AQM
  • CDEPS
  • CICE
  • CMEPS
  • CMakeModules
  • FV3
  • GOCART
  • HYCOM
  • MOM6
  • NOAHMP
  • WW3
  • stochastic_physics
  • none

Anticipated Changes

Input data

  • No changes are expected to input data.
  • Changes are expected to input data:
    • New input data.
    • Updated input data.

Regression Tests:

  • No changes are expected to any regression test.
  • Changes are expected to the following tests:

baseline cases for mraerosol=T: merra2_thompson and atmaero_control_p8_rad_micro

Tests effected by changes in this PR:
003 cpld_control_gfsv17_iau_intel failed in check_result
014 cpld_bmark_p8_intel failed in check_result
023 control_flake_intel failed in check_result
025 control_CubedSphereGrid_parallel_intel failed in check_result
026 control_latlon_intel failed in check_result
027 control_wrtGauss_netcdf_parallel_intel failed in check_result
029 control_c192_intel failed in check_result
030 control_c384_intel failed in check_result
031 control_c384gdas_intel failed in check_result
032 control_stochy_intel failed in check_result
034 control_lndp_intel failed in check_result
035 control_iovr4_intel failed in check_result
036 control_iovr5_intel failed in check_result
037 control_p8_intel failed in check_result
039 control_qr_p8_intel failed in check_result
041 control_decomp_p8_intel failed in check_result
042 control_2threads_p8_intel failed in check_result
043 control_p8_lndp_intel failed in check_result
044 control_p8_rrtmgp_intel failed in check_result
045 control_p8_mynn_intel failed in check_result
046 merra2_thompson_intel failed in check_result
047 regional_control_intel failed in check_result
049 regional_control_qr_intel failed in check_result
051 regional_decomp_intel failed in check_result
052 regional_2threads_intel failed in check_result
055 regional_2dwrtdecomp_intel failed in check_result
056 regional_wofs_intel failed in check_result
057 rap_control_intel failed in check_result
058 regional_spp_sppt_shum_skeb_intel failed in check_result
059 rap_decomp_intel failed in check_result
060 rap_2threads_intel failed in check_result
062 rap_sfcdiff_intel failed in check_result
063 rap_sfcdiff_decomp_intel failed in check_result
065 hrrr_control_intel failed in check_result
066 hrrr_control_qr_intel failed in check_result
067 hrrr_control_decomp_intel failed in check_result
068 hrrr_control_2threads_intel failed in check_result
071 rrfs_v1beta_intel failed in check_result
076 control_ras_intel failed in check_result
078 control_p8_faster_intel failed in check_result
079 regional_control_faster_intel failed in check_result
106 regional_spp_sppt_shum_skeb_dyn32_phy32_intel failed in check_result
107 rap_control_dyn32_phy32_intel failed in check_result
108 hrrr_control_dyn32_phy32_intel failed in check_result
109 hrrr_control_qr_dyn32_phy32_intel failed in check_result
110 rap_2threads_dyn32_phy32_intel failed in check_result
111 hrrr_control_2threads_dyn32_phy32_intel failed in check_result
112 hrrr_control_decomp_dyn32_phy32_intel failed in check_result
119 rap_control_dyn64_phy32_intel failed in check_result
127 hafs_regional_atm_intel failed in check_result
137 hafs_global_multiple_4nests_atm_intel failed in check_result
138 hafs_global_multiple_4nests_atm_qr_intel failed in check_result
139 hafs_regional_specified_moving_1nest_atm_intel failed in check_result
168 control_atmwav_intel failed in check_result
169 atmaero_control_p8_intel failed in check_result
170 atmaero_control_p8_rad_intel failed in check_result
171 atmaero_control_p8_rad_micro_intel failed in check_result
176 control_stochy_gnu failed in check_result
177 control_ras_gnu failed in check_result
178 control_p8_gnu failed in check_result
179 control_flake_gnu failed in check_result
180 rap_control_gnu failed in check_result
181 rap_decomp_gnu failed in check_result
182 rap_2threads_gnu failed in check_result
184 rap_sfcdiff_gnu failed in check_result
185 rap_sfcdiff_decomp_gnu failed in check_result
187 hrrr_control_gnu failed in check_result
188 hrrr_control_qr_gnu failed in check_result
189 hrrr_control_2threads_gnu failed in check_result
190 hrrr_control_decomp_gnu failed in check_result
193 rrfs_v1beta_gnu failed in check_result
209 rap_control_dyn32_phy32_gnu failed in check_result
210 hrrr_control_dyn32_phy32_gnu failed in check_result
211 hrrr_control_qr_dyn32_phy32_gnu failed in check_result
212 rap_2threads_dyn32_phy32_gnu failed in check_result
213 hrrr_control_2threads_dyn32_phy32_gnu failed in check_result
214 hrrr_control_decomp_dyn32_phy32_gnu failed in check_result
221 rap_control_dyn64_phy32_gnu failed in check_result

Libraries

  • Not Needed
  • Needed
    • Create separate issue in JCSDA/spack-stack asking for update to library. Include library name, library version.
    • Add issue link from JCSDA/spack-stack following this item
Code Managers Log
  • This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR.
  • Move new/updated input data on RDHPCS Hera and propagate input data changes to all supported systems.
    • N/A

Testing Log:

  • RDHPCS
    • Hera
    • Orion
    • Hercules
    • Jet
    • Gaea
    • Cheyenne
  • WCOSS2
    • Dogwood/Cactus
    • Acorn
  • CI
    • Completed
  • opnReqTest
    • N/A
    • Log attached to comment

@AnningCheng-NOAA
Copy link
Contributor Author

regression test went well at /scratch1/NCEPDEV/global/Anning.Cheng/ufs-weather-model/tests

@DeniseWorthen
Copy link
Collaborator

DeniseWorthen commented Sep 22, 2023

@AnningCheng-NOAA It looks like your changes are actually in ccpp/physics

diff --git a/physics/module_mp_thompson.F90 b/physics/module_mp_thompson.F90
index ca913c6e..271db11d 100644
--- a/physics/module_mp_thompson.F90
+++ b/physics/module_mp_thompson.F90
@@ -1509,6 +1509,14 @@ MODULE module_mp_thompson
             enddo
          endif

+         if (merra2_aerosol_aware) then
+            do k = kts, kte
+               nc(i,k,j) = nc1d(k)
+               nwfa(i,k,j) = nwfa1d(k)
+               nifa(i,k,j) = nifa1d(k)
+            enddo
+         endif
+
          do k = kts, kte
             qv(i,k,j) = qv1d(k)
             qc(i,k,j) = qc1d(k)

But there are no committed changes in your FV3 branch. Your PR to UFS needs point to your FV3 and ccpp branches.

@AnningCheng-NOAA
Copy link
Contributor Author

AnningCheng-NOAA commented Sep 22, 2023 via email

@DeniseWorthen
Copy link
Collaborator

Hi Anning. Thanks for adding the PR information. But the FV3 hash in this PR (e4ead8a) does not appear to contain your ccpp branch. It points to ccpp hash bbc5bf8. I think it needs to point to your feature branch in ccpp 5377c7c

@AnningCheng-NOAA
Copy link
Contributor Author

AnningCheng-NOAA commented Sep 22, 2023 via email

@DeniseWorthen
Copy link
Collaborator

DeniseWorthen commented Sep 22, 2023

Start by getting your FV3 branch correct. I think you've missed one step.

git clone https://github.com/AnningCheng-NOAA/fv3atm.git
cd fv3atm
git checkout origin/mrfd
git submodule update --init --recursive

cd ccpp/phyics (git remote -v correctly shows you are in your fork)
git checkout remotes/origin/mrfd
cd ../../ (should be back at top-level of fv3atm)

now, git status will show you've made an update to ccpp/physics

~/fv3atm % git status
HEAD detached at origin/mrfd
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   ccpp/physics (new commits)

The next step is the step I think you missed. You need to add and commit that new hash:
git add ccpp/physics
git commit
git push

Once you do that, you should see your FV3 branch mrfd contains ccpp/physics with the correct hash. You can check by doing

git submodule status --recursive

which will show you

 52bf918c194b7d906776447c6324bc75558133db atmos_cubed_sphere (201912_public_release-370-g52bf918)
 1b6352fb24f053b738bde72eed0ddf0b60ec7c0f ccpp/framework (ccpp_transition_to_vlab_master_20190705-577-g1b6352f)
 5377c7c0ab39f275804749a50f31f0e03f7abab4 ccpp/physics (master-tag-before-replacing-with-ipd-setup-step-fast-4703-g5377c7c0)
 0dc54f5ecaeb1e1e342efd1e02d0bcd41737bde2 ccpp/physics/physics/rte-rrtmgp (v1.5-5-g0dc54f5)
 520cc233f7919dbbe5dc7fc0246354ae95caa2dc upp (upp_v10.2.0-130-g520cc23)
-41dd43d7c552b0981b894dcc0f9db507a120f7e2 upp/sorc/libIFI.fd
-d0ad3241659fd2b30ea24fc9daecebde5c51806f upp/sorc/ncep_post.fd/post_gtg.fd

@DeniseWorthen
Copy link
Collaborator

Once you've got your FV3 branch correct, you do the same process with your UFS branch. You need to both checkout your FV3 branch (which should now contain your ccpp/phyics) and add your FV3 to your ufs feature branch.

@DusanJovic-NOAA DusanJovic-NOAA changed the title Mrfd Bug fix for MERRA2 coupled Thompson microphysics Oct 6, 2023
@zach1221
Copy link
Collaborator

Hi, @AnningCheng-NOAA . I think we may begin testing against your PR soon. Could you please sync up your ufs-wm and fv3atm branches to resolve conflicts?

@AnningCheng-NOAA
Copy link
Contributor Author

AnningCheng-NOAA commented Oct 12, 2023 via email

@jkbk2004
Copy link
Collaborator

@AnningCheng-NOAA looks like CICE, CMEPS, and stochastic physics not synced yet.

@AnningCheng-NOAA
Copy link
Contributor Author

AnningCheng-NOAA commented Oct 12, 2023 via email

@AnningCheng-NOAA
Copy link
Contributor Author

I have just run
git check out and git remote update in develop/master in CICE, CDEP, and stochatics
cd ..
git push origin mrfd

@jkbk2004
Copy link
Collaborator

@AnningCheng-NOAA you can take a look at "synching latest HEAD section" in https://github.com/ufs-community/ufs-weather-model/wiki/Making-code-changes-in-the-UFS-weather-model-and-its-subcomponents. But sounds like you just need to update hashes for CICE/CMEPS/Stochastic-physics submodules. Let me see if I can push to update hashes directly.

@jkbk2004
Copy link
Collaborator

@AnningCheng-NOAA Can you make to sync up ccpp physics with latest HEAD of https://github.com/ufs-community/ccpp-physics/tree/ufs/dev? That's where your code changes are originated. So you need to make sure code changes are reflected correctly in ufs-community/ccpp-physics#109.

@jkbk2004
Copy link
Collaborator

@AnningCheng-NOAA @grantfirl branches all looks synced up ok now. @WenMeng-NOAA baseline is expected to change with upp updates. We will go ahead to update baseline.

@jkbk2004 jkbk2004 changed the title Bug fix for MERRA2 coupled Thompson microphysics Bug fix for MERRA2 coupled Thompson microphysics and UPP update Oct 13, 2023
@jkbk2004
Copy link
Collaborator

@zach1221 zach1221 added the Baseline Updates Current baselines will be updated. label Oct 13, 2023
@DeniseWorthen
Copy link
Collaborator

DeniseWorthen commented Oct 16, 2023

@BrianCurtis-NOAA In your original testing of this PR on Acorn when you got the failures, the failures occurred at different lines in two different tries (not in debug mode). Is that right? That hints at somehow triggering an uninitialized variable or something like that. But, I don't have an explanation for a) why on Acorn and b) why this PR.

@DeniseWorthen
Copy link
Collaborator

@zach1221 Might also be worth testing GNU + debug on Hercules.

@BrianCurtis-NOAA
Copy link
Collaborator

I had the same error trying in debug mode with the PR branch.

The test seems to fail at different places (in non-debug mode) leaning towards potentially an uninitialized variable. This seems to be coming from this PR's changes, as I can run the test in develop branch OK.

@zach1221
Copy link
Collaborator

@zach1221 Might also be worth testing GNU + debug on Hercules.

Ok, I'll give it a try.

@BrianCurtis-NOAA
Copy link
Collaborator

I'm going to skip this test on Acorn. We'll have to look into why Acorn specifically.

@zach1221
Copy link
Collaborator

We can continue to work the issue #1944 , but I think we start the merging process here.

@BrianCurtis-NOAA
Copy link
Collaborator

Acorn failing tests:

brian.curtis@alogin01:/lfs/h1/emc/nems/noscrub/brian.curtis/git/AnningCheng-NOAA/ufs-weather-model/tests/logs
/log_acorn> tail -50 run_019_control_CubedSphereGrid_parallel_intel.log 
+ atparse
+ local __set_x
+ '[' -o xtrace ']'
+ __set_x='set -x'
+ set +x
+ export OMP_ENV=
+ OMP_ENV=
+ [[ pbs = \n\o\n\e ]]
+ [[ false = \f\a\l\s\e ]]
+ submit_and_wait job_card
+ [[ -z job_card ]]
+ '[' -o xtrace ']'
+ set_x='set -x'
+ set +x
Job id 2441741
TEST 019 control_CubedSphereGrid_parallel_intel is waiting to enter the queue
TEST 019 control_CubedSphereGrid_parallel_intel is submitted 
1 min. TEST 019 control_CubedSphereGrid_parallel_intel is waiting in a queue,  status: Q jobid 2441741
2 min. TEST 019 control_CubedSphereGrid_parallel_intel is waiting in a queue,  status: Q jobid 2441741
3 min. TEST 019 control_CubedSphereGrid_parallel_intel is waiting in a queue,  status: Q jobid 2441741
4 min. TEST 019 control_CubedSphereGrid_parallel_intel is waiting in a queue,  status: Q jobid 2441741
5 min. TEST 019 control_CubedSphereGrid_parallel_intel is running,  status: R jobid 2441741
6 min. TEST 019 control_CubedSphereGrid_parallel_intel is running,  status: R jobid 2441741
7 min. TEST 019 control_CubedSphereGrid_parallel_intel is running,  status: R jobid 2441741
qstat: 2441741.abqs01 Job has finished, use -x or -H to obtain historical job information
qstat: 2441741.abqs01 Job has finished, use -x or -H to obtain historical job information
8 min. TEST 019 control_CubedSphereGrid_parallel_intel is finished,  status: - jobid 2441741
+ [[ false = false ]]
+ check_results
+ '[' -o xtrace ']'
+ set_x='set -x'
+ set +x

baseline dir = /lfs/h1/emc/nems/noscrub/emc.nems/RT/NEMSfv3gfs/develop-20231013/control_CubedSphereGrid_parallel_intel
working dir  = /lfs/h2/emc/ptmp/brian.curtis/FV3_RT/rt_60387/control_CubedSphereGrid_parallel_intel
Checking test 019 control_CubedSphereGrid_parallel_intel results ....
 Comparing sfcf000.nc .........OK
 Comparing sfcf024.nc ............ALT CHECK......NOT OK
 Comparing atmf000.nc .........OK
 Comparing atmf024.nc ............ALT CHECK......NOT OK
 Comparing cubed_sphere_grid_sfcf000.nc .........OK
 Comparing cubed_sphere_grid_sfcf024.nc ............ALT CHECK......NOT OK
 Comparing cubed_sphere_grid_atmf000.nc .........OK
 Comparing cubed_sphere_grid_atmf024.nc ............ALT CHECK......NOT OK
 Comparing GFSFLX.GrbF00 .........OK
 Comparing GFSFLX.GrbF24 .........NOT OK
 Comparing GFSPRS.GrbF00 .........OK
 Comparing GFSPRS.GrbF24 .........NOT OK
Test 019 control_CubedSphereGrid_parallel_intel FAIL Tries: 2

brian.curtis@alogin01:/lfs/h1/emc/nems/noscrub/brian.curtis/git/AnningCheng-NOAA/ufs-weather-model/tests/logs
/log_acorn> tail -50 run_021_control_wrtGauss_netcdf_parallel_intel.log 
+ ((  NODES * TPN < TASKS  ))
+ NODES=2
+ export NODES
+ TASKS=256
+ export TASKS
+ [[ pbs = \p\b\s ]]
+ [[ -e /lfs/h1/emc/nems/noscrub/brian.curtis/git/AnningCheng-NOAA/ufs-weather-model/tests/fv3_conf/fv3_qsub.IN_acorn ]]
+ atparse
+ local __set_x
+ '[' -o xtrace ']'
+ __set_x='set -x'
+ set +x
+ export OMP_ENV=
+ OMP_ENV=
+ [[ pbs = \n\o\n\e ]]
+ [[ false = \f\a\l\s\e ]]
+ submit_and_wait job_card
+ [[ -z job_card ]]
+ '[' -o xtrace ']'
+ set_x='set -x'
+ set +x
Job id 2441760
TEST 021 control_wrtGauss_netcdf_parallel_intel is waiting to enter the queue
TEST 021 control_wrtGauss_netcdf_parallel_intel is submitted 
1 min. TEST 021 control_wrtGauss_netcdf_parallel_intel is waiting in a queue,  status: Q jobid 2441760
2 min. TEST 021 control_wrtGauss_netcdf_parallel_intel is waiting in a queue,  status: Q jobid 2441760
3 min. TEST 021 control_wrtGauss_netcdf_parallel_intel is running,  status: R jobid 2441760
4 min. TEST 021 control_wrtGauss_netcdf_parallel_intel is running,  status: R jobid 2441760
qstat: 2441760.abqs01 Job has finished, use -x or -H to obtain historical job information
qstat: 2441760.abqs01 Job has finished, use -x or -H to obtain historical job information
5 min. TEST 021 control_wrtGauss_netcdf_parallel_intel is finished,  status: - jobid 2441760
+ [[ false = false ]]
+ check_results
+ '[' -o xtrace ']'
+ set_x='set -x'
+ set +x

baseline dir = /lfs/h1/emc/nems/noscrub/emc.nems/RT/NEMSfv3gfs/develop-20231013/control_wrtGauss_netcdf_parallel_intel
working dir  = /lfs/h2/emc/ptmp/brian.curtis/FV3_RT/rt_60387/control_wrtGauss_netcdf_parallel_intel
Checking test 021 control_wrtGauss_netcdf_parallel_intel results ....
 Comparing sfcf000.nc .........OK
 Comparing sfcf024.nc ............ALT CHECK......NOT OK
 Comparing atmf000.nc .........OK
 Comparing atmf024.nc ............ALT CHECK......NOT OK
 Comparing GFSFLX.GrbF00 .........OK
 Comparing GFSFLX.GrbF24 .........NOT OK
 Comparing GFSPRS.GrbF00 .........OK
 Comparing GFSPRS.GrbF24 .........NOT OK
Test 021 control_wrtGauss_netcdf_parallel_intel FAIL Tries: 2

brian.curtis@alogin01:/lfs/h1/emc/nems/noscrub/brian.curtis/git/AnningCheng-NOAA/ufs-weather-model/tests/logs
/log_acorn> tail -50 run_023_control_c192_intel.log 
+ __set_x='set -x'
+ set +x
+ export OMP_ENV=
+ OMP_ENV=
+ [[ pbs = \n\o\n\e ]]
+ [[ false = \f\a\l\s\e ]]
+ submit_and_wait job_card
+ [[ -z job_card ]]
+ '[' -o xtrace ']'
+ set_x='set -x'
+ set +x
Job id 2441784
TEST 023 control_c192_intel is waiting to enter the queue
TEST 023 control_c192_intel is submitted 
1 min. TEST 023 control_c192_intel is waiting in a queue,  status: Q jobid 2441784
2 min. TEST 023 control_c192_intel is waiting in a queue,  status: Q jobid 2441784
3 min. TEST 023 control_c192_intel is waiting in a queue,  status: Q jobid 2441784
4 min. TEST 023 control_c192_intel is waiting in a queue,  status: Q jobid 2441784
5 min. TEST 023 control_c192_intel is waiting in a queue,  status: Q jobid 2441784
6 min. TEST 023 control_c192_intel is running,  status: R jobid 2441784
7 min. TEST 023 control_c192_intel is running,  status: R jobid 2441784
8 min. TEST 023 control_c192_intel is running,  status: R jobid 2441784
9 min. TEST 023 control_c192_intel is running,  status: R jobid 2441784
10 min. TEST 023 control_c192_intel is running,  status: R jobid 2441784
11 min. TEST 023 control_c192_intel is running,  status: R jobid 2441784
12 min. TEST 023 control_c192_intel is running,  status: R jobid 2441784
13 min. TEST 023 control_c192_intel is running,  status: R jobid 2441784
14 min. TEST 023 control_c192_intel is running,  status: R jobid 2441784
qstat: 2441784.abqs01 Job has finished, use -x or -H to obtain historical job information
qstat: 2441784.abqs01 Job has finished, use -x or -H to obtain historical job information
15 min. TEST 023 control_c192_intel is finished,  status: - jobid 2441784
+ [[ false = false ]]
+ check_results
+ '[' -o xtrace ']'
+ set_x='set -x'
+ set +x

baseline dir = /lfs/h1/emc/nems/noscrub/emc.nems/RT/NEMSfv3gfs/develop-20231013/control_c192_intel
working dir  = /lfs/h2/emc/ptmp/brian.curtis/FV3_RT/rt_60387/control_c192_intel
Checking test 023 control_c192_intel results ....
 Comparing sfcf000.nc .........OK
 Comparing sfcf024.nc ............ALT CHECK......NOT OK
 Comparing atmf000.nc .........OK
 Comparing atmf024.nc ............ALT CHECK......NOT OK
 Comparing GFSFLX.GrbF00 .........OK
 Comparing GFSFLX.GrbF24 .........NOT OK
 Comparing GFSPRS.GrbF00 .........OK
 Comparing GFSPRS.GrbF24 .........NOT OK
Test 023 control_c192_intel FAIL Tries: 2

brian.curtis@alogin01:/lfs/h1/emc/nems/noscrub/brian.curtis/git/AnningCheng-NOAA/ufs-weather-model/tests/logs
/log_acorn> tail -50 run_024_control_c384_intel.log 
+ [[ -e /lfs/h1/emc/nems/noscrub/brian.curtis/git/AnningCheng-NOAA/ufs-weather-model/tests/fv3_conf/fv3_qsub.IN_acorn ]]
+ atparse
+ local __set_x
+ '[' -o xtrace ']'
+ __set_x='set -x'
+ set +x
+ export OMP_ENV=
+ OMP_ENV=
+ [[ pbs = \n\o\n\e ]]
+ [[ false = \f\a\l\s\e ]]
+ submit_and_wait job_card
+ [[ -z job_card ]]
+ '[' -o xtrace ']'
+ set_x='set -x'
+ set +x
Job id 2441852
TEST 024 control_c384_intel is waiting to enter the queue
TEST 024 control_c384_intel is submitted 
1 min. TEST 024 control_c384_intel is running,  status: R jobid 2441852
2 min. TEST 024 control_c384_intel is running,  status: R jobid 2441852
3 min. TEST 024 control_c384_intel is running,  status: R jobid 2441852
4 min. TEST 024 control_c384_intel is running,  status: R jobid 2441852
5 min. TEST 024 control_c384_intel is running,  status: R jobid 2441852
6 min. TEST 024 control_c384_intel is running,  status: R jobid 2441852
7 min. TEST 024 control_c384_intel is running,  status: R jobid 2441852
8 min. TEST 024 control_c384_intel is running,  status: R jobid 2441852
9 min. TEST 024 control_c384_intel is running,  status: R jobid 2441852
10 min. TEST 024 control_c384_intel is running,  status: R jobid 2441852
qstat: 2441852.abqs01 Job has finished, use -x or -H to obtain historical job information
qstat: 2441852.abqs01 Job has finished, use -x or -H to obtain historical job information
11 min. TEST 024 control_c384_intel is finished,  status: - jobid 2441852
+ [[ false = false ]]
+ check_results
+ '[' -o xtrace ']'
+ set_x='set -x'
+ set +x

baseline dir = /lfs/h1/emc/nems/noscrub/emc.nems/RT/NEMSfv3gfs/develop-20231013/control_c384_intel
working dir  = /lfs/h2/emc/ptmp/brian.curtis/FV3_RT/rt_60387/control_c384_intel
Checking test 024 control_c384_intel results ....
 Comparing sfcf000.nc .........OK
 Comparing sfcf012.nc ............ALT CHECK......NOT OK
 Comparing atmf000.nc .........OK
 Comparing atmf012.nc ............ALT CHECK......NOT OK
 Comparing GFSFLX.GrbF00 .........OK
 Comparing GFSFLX.GrbF12 .........NOT OK
 Comparing GFSPRS.GrbF00 .........OK
 Comparing GFSPRS.GrbF12 .........NOT OK
Test 024 control_c384_intel FAIL Tries: 2

brian.curtis@alogin01:/lfs/h1/emc/nems/noscrub/brian.curtis/git/AnningCheng-NOAA/ufs-weather-model/tests/logs
/log_acorn> tail -50 run_025_control_c384gdas_intel.log 
+ check_results
+ '[' -o xtrace ']'
+ set_x='set -x'
+ set +x

baseline dir = /lfs/h1/emc/nems/noscrub/emc.nems/RT/NEMSfv3gfs/develop-20231013/control_c384gdas_intel
working dir  = /lfs/h2/emc/ptmp/brian.curtis/FV3_RT/rt_60387/control_c384gdas_intel
Checking test 025 control_c384gdas_intel results ....
 Comparing sfcf000.nc .........OK
 Comparing sfcf006.nc ............ALT CHECK......NOT OK
 Comparing atmf000.nc .........OK
 Comparing atmf006.nc ............ALT CHECK......NOT OK
 Comparing GFSFLX.GrbF00 .........OK
 Comparing GFSFLX.GrbF06 .........NOT OK
 Comparing GFSPRS.GrbF00 .........OK
 Comparing GFSPRS.GrbF06 .........NOT OK
 Comparing RESTART/20210322.060000.coupler.res .........OK
 Comparing RESTART/20210322.060000.fv_core.res.nc .........OK
 Comparing RESTART/20210322.060000.fv_core.res.tile1.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.060000.fv_core.res.tile2.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.060000.fv_core.res.tile3.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.060000.fv_core.res.tile4.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.060000.fv_core.res.tile5.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.060000.fv_core.res.tile6.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.060000.fv_srf_wnd.res.tile1.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.060000.fv_srf_wnd.res.tile2.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.060000.fv_srf_wnd.res.tile3.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.060000.fv_srf_wnd.res.tile4.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.060000.fv_srf_wnd.res.tile5.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.060000.fv_srf_wnd.res.tile6.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.060000.fv_tracer.res.tile1.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.060000.fv_tracer.res.tile2.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.060000.fv_tracer.res.tile3.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.060000.fv_tracer.res.tile4.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.060000.fv_tracer.res.tile5.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.060000.fv_tracer.res.tile6.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.060000.phy_data.tile1.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.060000.phy_data.tile2.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.060000.phy_data.tile3.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.060000.phy_data.tile4.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.060000.phy_data.tile5.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.060000.phy_data.tile6.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.060000.sfc_data.tile1.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.060000.sfc_data.tile2.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.060000.sfc_data.tile3.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.060000.sfc_data.tile4.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.060000.sfc_data.tile5.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.060000.sfc_data.tile6.nc ............ALT CHECK......NOT OK
Test 025 control_c384gdas_intel FAIL Tries: 2

@zach1221
Copy link
Collaborator

@BrianCurtis-NOAA should these cases be turned off for Acorn?

@AnningCheng-NOAA
Copy link
Contributor Author

@BrianCurtis-NOAA code modified in PR109 is not executed in those tests either. The two experiments executing the code are atmaero_control_p8_rad_micro and merra2_thompson.

@BrianCurtis-NOAA
Copy link
Collaborator

Yes, go ahead and skip Acorn.

@zach1221
Copy link
Collaborator

Yes, go ahead and skip Acorn.

Ok, will do. The fv3atm sub-pr is ready for review. Fyi @jkbk2004 @BrianCurtis-NOAA

@zach1221
Copy link
Collaborator

@AnningCheng-NOAA FV3atm sub-pr has been merged. Please go ahead and revert the changed url for the .gitmodule as well as update the submodule pointer. FV3atm hash: eadb52f6953502d8f5fc6ee3d07b257571013345

@AnningCheng-NOAA
Copy link
Contributor Author

AnningCheng-NOAA commented Oct 17, 2023 via email

@zach1221 zach1221 merged commit fb788ba into ufs-community:develop Oct 18, 2023
@DeniseWorthen
Copy link
Collaborator

@jkbk2004 Please see if you can reproduce this issue NOAA-EMC/UPP#804

@DeniseWorthen
Copy link
Collaborator

DeniseWorthen commented Oct 18, 2023

I am testing a PR branch which should have no impact on baselines, and all the Post files (ie, GFSPRS.GrbF00) are not reproducing. I will run some tests from develop, but I think there is an issue w/ the UPP update from yesterday.

Partial list:

rt_003_cpld_control_gfsv17_iau_intel.log:8: Comparing GFSPRS.GrbF12 .........NOT OK
rt_014_cpld_bmark_p8_intel.log:8: Comparing GFSPRS.GrbF06 .........NOT OK
rt_027_control_flake_intel.log:11: Comparing GFSPRS.GrbF00 .........NOT OK
rt_027_control_flake_intel.log:12: Comparing GFSPRS.GrbF24 .........NOT OK
rt_029_control_CubedSphereGrid_parallel_intel.log:15: Comparing GFSPRS.GrbF00 .........NOT OK
rt_029_control_CubedSphereGrid_parallel_intel.log:16: Comparing GFSPRS.GrbF24 .........NOT OK

@DeniseWorthen
Copy link
Collaborator

I think I've gotten everything aligned now (finally) and these tests are now reproducing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Baseline Updates Current baselines will be updated. jenkins-ci Jenkins CI: ORT build/test on docker container Ready for Commit Queue The PR is ready for the Commit Queue. All checkboxes in PR template have been checked.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants