SOCA2CICE conversion segfault issue (caused by shuffle)

[Dooruk]

UPDATE

Per a tip by @guillaumevernieres, it is the shuffle option related. Don't shuffle, folks.

Description

Not quite sure if this should be defined as a bug or some compiler issue regarding Discover. Essentially soca2cice conversion is acting unpredictable for our 0.25-deg setup. @guillaumevernieres told me they haven't had any issues and Skylab uses high res setup with ocean+sea-ice (not sure if that's tested on Discover?). I can run the 5 degree SOCA tests no problem.

To Reproduce:

What compilers/machines are you seeing this with?

Spack-stack 1.7, NCCS Discover SLES12

The following command, using a single node, has a success rate of about %60, runs successfully most of the time, segfault error other times. I just kept executing the same code over and over again:

mpirun -np 24 $jedibuild/soca_convertstate.x soca2cice.yaml

The following, when I use multiple nodes, fails all the time (I just repeated the same command 8-10 times):

mpirun -np 720 $jedibuild/soca_convertstate.x soca2cice.yaml