The status about the chunk sizes specification in the model

[TingLei-NOAA]

Hi, UFS developers, currently, what is considered in specifying the chunk size for UFS netcdf output? In the restart files, we recently found, the chunk size is "too small" for large model domains, as in the regional parallel runs at EMC,
netcdf fv3_dynvars {
dimensions:
xaxis_1 = 3646 ;
xaxis_2 = 3647 ;
yaxis_1 = 2527 ;
yaxis_2 = 2526 ;
zaxis_1 = 65 ;
Time = UNLIMITED ; // (1 currently)

........
float T(Time, zaxis_1, yaxis_2, xaxis_1) ;
T:_Storage = "chunked" ;
T:_ChunkSizes = 1, 7, 281, 406 ;
T:_Endianness = "little" ;
.............
Such small values of the chunk sizes make the recently implemented GSI parallel netcdf IO (for netcdf 4 based on hdf5) almost totally fail to show any speeding up.
However, when, for example, the chunk size were converted to values equal to their dimension sizes, the total wall clock time could be reduced 10 to 30 % ( from Eric Rogers) .
So, That will be great if , first, we can know what the current status is for the chunk size specification for UFS, and finally, if we can have an " optimal" chose for the
chunk sizes in terms of the speeding performances of UFS model performance and the GSI (and, maybe, other postprocessing tools).
Thanks

[jswhit2]

@TingLei-NOAA the netcdf library will choose chunksizes if you don't specify the chunk size parameters (ichunk2d, ichunk3d etc) in model_configure. Not sure what your question is - are you asking how best to choose the ichunk,jchunk parameters in model_configure? There is some discussion of this here: https://www.unidata.ucar.edu/blogs/developer/en/entry/chunking_data_choosing_shapes

Note that the chunk size that produces the fastest write may not be optimized for a given read pattern, so there will always be compromises involved.

[jswhit2]

To be clear, the default values for the ichunk,jchunk, kchunk parameters (0) will cause the domain decompisition for the IO tasks to be used as chunk sizes. If the ichunk,jchunk,kchunk parameters are set to -1, then the netcdf-c lib will chose the chunksizes.

[DusanJovic-NOAA]

@TingLei-NOAA You are asking about chunking in restart files, which are written by FMS. The above explanation of ichunk,jchunk,kchunk options applies only if write grid component is used (history files).

[TingLei-NOAA]

@jswhit2 @DusanJovic-NOAA Thank you so much for your information/clarifications. Yes, now, my target is the restart files.
So, seems for the chunksize in the restart files, UFS doesn't have any control on top of FMS, right?
Then, I will ask my question at FMS site.
Really appreciate your quick help!

[TingLei-NOAA]

@DusanJovic-NOAA Thanks. I will keep updated here if I get any information from FMS experts.

[junwang-noaa]

I think the chunksize (ncchksz) is in namelist fms2_io_nml, the default is 64k

[TingLei-NOAA]

@junwang-noaa Thanks a lot. So, you mean , the NF90_DEF_VAR_CHUNKING is not used in fms to specify the chunk shapes as well as sizes in the fms? They would be automatically specified by the netcdf lib according to the chunksize value given in nf90_open?

[junwang-noaa]

No, I mean fms2 is using namelist variable ncchksz to control the chunksize, if you do not specify this namelist variable, the default value 64k will be used in the restart file writing

[TingLei-NOAA]

@junwang-noaa Thanks for the further clarification. I am expecting chunking shapes could be specified in addition to the sizes.
From my searching in fms, seems FMS doesn't control the shapes, while they are important for our purpose. I will see if I can get any clarification from FMS developers. Thanks.

[jswhit2]

From the FMS Changelog

FMS2_IO: An fms2_io_nml namelist has been created. It includes the variable ncchksz. This is the replacement for the environment variable NC_BLKSZ set in model run scripts and used by mpp_io. The default value is 64 KB. Any time a file is opened in fms2_io (nf90_open or nf90_create), the optional argument chunksize=ncchksz is passed to the NetCDF library. NetCDF attempts to use this value to control the blocksize utilized for reads and writes of data from the filesystem.

[TingLei-NOAA]

@jswhit2 Thanks! I will see if the chunkshape could be controlled too .

[TingLei-NOAA]

Dear All, An update from FMS developers on chunk size/shapes specification in FMS.
NOAA-GFDL/FMS#892.