C192mx025 model instability

[benjamin-cash]

I am attempting a coupled run from the HR4-based SFS_baseline @NeilBarton-NOAA has created, and the model is crashing with the errors described here NeilBarton-NOAA/global-workflow#12 (comment). Reducing DELTIM from 600 to 300 and even 150 did not eliminate the problem. From this issue #1042 I wonder if the problem is the gravity wave drag settings. For C192, the workflow has them set as:

export cdmbgwd_gsl="10.0,3.5,1.0,1.0" # settings for GSL drag suite

If someone could suggest different settings to try for these parameters or has other insights into why it might be crashing I would appreciate any help. Thanks!

[jkbk2004]

@benjamin-cash There was a report that HR4 run crashes on WCOSS2. @NickSzapiro-NOAA @jiandewang @RuiyuSun any comment or update?

[benjamin-cash]

Hi @jkbk2004 - thanks for the update. My C96mx100 runs ran more or less ok (a couple test runs failed but that could have been due to a cluster issue). @NeilBarton-NOAA said some of his runs crashed at C192mx025, but that it failed in the sea ice model. If I can find the right ICs I'll try a run at C384mx025.

[RuiyuSun]

I am attempting a coupled run from the HR4-based SFS_baseline @NeilBarton-NOAA has created, and the model is crashing with the errors described here NeilBarton-NOAA/global-workflow#12 (comment). Reducing DELTIM from 600 to 300 and even 150 did not eliminate the problem. From this issue #1042 I wonder if the problem is the gravity wave drag settings. For C192, the workflow has them set as:

export cdmbgwd_gsl="10.0,3.5,1.0,1.0" # settings for GSL drag suite

If someone could suggest different settings to try for these parameters or has other insights into why it might be crashing I would appreciate any help. Thanks!

@JongilHan66 Do you have any suggestion?

[JongilHan66]

@RuiyuSun As I understand, HR4 GWD doesn't use the first 2 parameters of cdmbgwd_gsl (i.e., 10.0, 3.5).

[lisa-bengtsson]

@JongilHan66 are the cdmbgwd_gsl differences you refer to for C1152 resolution only, or for any configuration using HR4 version of gravity wave drag?

[JongilHan66]

@lisa-bengtsson As far as I know, the HR4 GWD does not use the first 2 parameters of cdmbgwd for any resolutions

[benjamin-cash]

@JongilHan66 @lisa-bengtsson - The config.ufs generated by the global-workflow includes the following settings for C1152 in my experiment directory:

"C1152")
        export DELTIM=150
        export layout_x=8
        export layout_y=16
        export layout_x_gfs=8
        export layout_y_gfs=16
        export nthreads_fv3=4
        export nthreads_fv3_gfs=4
        export nthreads_ufs=4
        export nthreads_ufs_gfs=4
        export cdmbgwd="4.0,0.10,1.0,1.0"  # mountain blocking, ogwd, cgwd, cgwd src scaling
        export cdmbgwd_gsl="1.67,8.8,1.0,1.0"   # settings for GSL drag suite

I looked at the HR4 tag of the workflow in the repo and these are the same settings, does this need to be changed? Is HR4 simply ignoring those settings, or is it pulling in incorrect values?

[jiandewang]

I have finished more than 95% of all IC cases for HR4 C1152 and only had 2 cases that had numerical instability. These two cases finished with reduced time step for fv3.

[benjamin-cash]

Hi @jiandewang - Did those runs crash in the same place with this same error message, or did the instability occur elsewhere?

[jiandewang]

no, they just had instability where temperature became unreasonable values

[benjamin-cash]

Thanks! Given that my crashes persisted even after I drastically reduced the timestep it seems like that probably wasn't the issue here then.

[benjamin-cash]

Hi @jiandewang - were you seeing errors like this?

 (zap_snow_temperature)zap_snow_temperature: temperature out of bounds!
  (zap_snow_temperature)k:           1
  (zap_snow_temperature)zTsn:  4.563558220938299E-003
  (zap_snow_temperature)Tmin:  -100.000000000000
  (zap_snow_temperature)Tmax:  2.271875959455578E-004
  (zap_snow_temperature)zqsn:  -110216828.418308
MOM Date   1994/05/01 00:00:00      0: En 3.002696E-01, MaxCFL  0.11050, Mass 1.381842761701E+21, Salt  34.72636202182, Temp   3.57610744009
    Total Energy: 44367E3EFBE50F8A  4.1492535488017190E+20
    Total Mass:   1.3818427617013004E+21, Change:   0.0000000000000000E+00 Error:  0.00000E+00 ( 0.0E+00)
    Total Salt:   4.7986372000074285E+19, Change:   0.0000000000000000E+00 Error:  0.00000E+00 ( 0.0E+00)
    Total Heat:   1.9726939779178277E+25, Change:   0.0000000000000000E+00 Error:  0.00000E+00 ( 0.0E+00)

FATAL from PE    52: NaN in input field of mpp_reproducing_sum(_2d), this indicates numerical instability

[benjamin-cash]

@pjpegion - I recall you mentioning seeing unreasonable temperature errors with the previous land ICs at C192, did they look like this?

[benjamin-cash]

@NeilBarton-NOAA @pjpegion - I swapped in the new land ICs Phil provided but the model still blew up with this snow temperature out of bounds error.

[lisa-bengtsson]

@benjamin-cash do you have stochastic physics turned on? Some runs from Neil crashed using SKEBS with the hydrostatic dycore.

[benjamin-cash]

According to the log DO_SKEB=NO

[jiandewang]

@benjamin-cash my run error is different from yours, see below
Range violation on: 2020/ 8/22 6:47:30
nid001267.cactus.wcoss2.ncep.noaa.gov 0: NOTE from PE 0: ==> Error from range_check: data out of bound
nid001651.cactus.wcoss2.ncep.noaa.gov 8744: Warn_K= 1 (i,j)= 118 822 (lon,lat)=286.725 -49.822 TA = 146.34183

[benjamin-cash]

Update! An issue with the sfc input files was uncovered, and with the updated files the model runs past this initial error (thanks @pjpegion !). However, it is now crashing with the following segfault:

whalo =    3, ehalo =    3, shalo =    3, nhalo =    3
  X-AXIS =  240 240 240 240
  Y-AXIS =  160 160 160 160 160 160
 in radiation_clouds_prop=           8 F           4 F F           2           1
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source
libc.so.6          00002B08B57DC520  Unknown               Unknown  Unknown
ufs_model.x        0000000002828690  gfs_rrtmg_pre_mp_         768  GFS_rrtmg_pre.F90
ufs_model.x        000000000258CAB5  gfs_v17_coupled_p         300  ccpp_FV3_GFS_v17_coupled_p8_ugwpv1_radiation_cap.F90
ufs_model.x        00000000023FA123  ccpp_static_api_m         555  ccpp_static_api.F90
libiomp5.so        00002B08B3167D93  __kmp_invoke_micr     Unknown  Unknown

My guess is that this is also related to the interpolated initial conditions in some way, but it is just a guess. Has anyone (@lisa-bengtsson @jiandewang @jkbk2004 @RuiyuSun @NeilBarton-NOAA) seen this issue before?

[RuiyuSun]

Ben, thanks for the update. I haven't run into this issue.

[benjamin-cash]

@RuiyuSun have you made any runs at C192?

[RuiyuSun]

No. All my tests are at C1152.

[jiandewang]

@benjamin-cash all HR1-4 runs are in C768 minimum

[benjamin-cash]

This turned out to be related to an issue in another thread, and was a combination of an error in the sfc*nc IC files, and two stacksize settings that were not applied to the job script. I'll go ahead and close this as answered now.

ulimit -s unlimited
export OMP_STACKSIZE=512M