Memory "management" issue with intel

[guillaumevernieres]

The soca variational application takes an insane amount of memory on Hecules and Gaea (~8TB for the simple 3DVAR), both use intel 2021.9.0. The same application on Hera requires ~0.8TB of memory, the intel compiler version on Hera is 2021.5.0.

I have no idea if the compiler is the issue.

[guillaumevernieres]

I'm labeling this as soca, but I wonder if it's an issues for the fv3-jedi application as well. @RussTreadon-NOAA , @CoryMartin-NOAA or others, have you tried running the variational application on Hercules lately?

[travissluka]

fyi @fmahebert

[guillaumevernieres]

path to logs:

/work2/noaa/da/gvernier/runs/profiling-3dvar

[fmahebert]

Yes, we've seen the intel compiler produce executables that take up way more memory for a little while now. We haven't been able to pinpoint the cause yet. Thanks for opening an issue to track this and to share your measurements.

Related issue (presumably):

• https://github.com/JCSDA-internal/fv3-jedi/issues/1256

[guillaumevernieres]

Running with map profiling:
https://github.com/JCSDA-internal/fv3-jedi/issues/1256#issuecomment-2471504465