Memory profiling of coupled model

[DeniseWorthen]

The cpld_control_wave test case from the develop branch of ufs-weather-model was run for 10days under the following conditions:

1. Add the code to the WW3 cap to add ESMF memory profiling (see branch)
2. Reduce the wave PEs from 200 (!) to 20
3. Turn on ProfileMemory=true for CICE, MOM6 and WW3

I created a small script to pull the VmPeak value at the end of the Model_Advance phase for each PET for ICE, OCN and WW3. The Memory use for ICE is shown below. The % increase over 240 hours for CICE ranges between 0.13 to 0.95

The Memory use for W3 is shown below. The % increase over 240 hours for W3 ranges between 0.27 to 1.9

The Memory use for OCN is shown below. The % increase over 240 hours for MOM6 ranges between 0.0 and 2.6. If the final memory use value is excluded however, the maximum % increase is reduced to 1.3. This removes the final "uptick" of memory use on PE 150. (Is this is what GFDL was referring to when they said there was a memory leak associated with the Finalize?). See issue #764

[jiandewang]

I repeated Denise's run and re-plotted the memory usage in a slight different way, we can see there is ~1% increase of memory usage in this 10day run. Note here PE150 is not included in the plot. Bob explained that the sudden jump in Denise's previous plot in PE150 is due to writing of restart file in the model ending stage

e.

[jiandewang]

this figure is based on the same 10 day run but only save last timestep of ocean results, the memory usage became flat

.

[DeniseWorthen]

@jiandewang Very interesting. Could you repeat this test (once w/ 6hr output, once w/ output only at the end) for the c384mx025 case ?

[jiandewang]

@DeniseWorthen this is from C384 OCN025 run with 6hr output frequency, note total OCN PE is 120 but here I only plotted every other 4 PEs

y

[jiandewang]

this is the same above but save last step of snapshot

t

[MatthewMasarik-NOAA]

For WW3, the profiling output points to memory increase occurring in the call to wmwave, so internal to the model vs. in the cap. Given this we've begun to track down the leaks by running valgrind over our WW3 regtest, ww3_ufs1.1, and starting to work through the output.