HDF5 used in NetCDF-4 is not thread-safe in Julia

[danlooo]

This issue is to document that accessing HDF5 based data like NetCDF-4 will result in errors and killed processes in a multithreaded context. HDF5 can be compiled in a thread-safe way, e.g. as in PyTables/PyTables#776, but it's not the case in the Julia version. Please add locks to your own code instead. See also JuliaIO/HDF5.jl#835, JuliaIO/HDF5.jl#835 and HDF5/build_tarballs.jl

Example using file sresa1b_ncar_ccsm3-example.nc:

using YAXArrays
using NetCDF
ds = open_dataset("sresa1b_ncar_ccsm3-example.nc")
Threads.@threads for i in 1:Threads.nthreads()
    ds.tas[1, 1, 1]
end

will result in error

ERROR: TaskFailedException

    nested task error: NetCDF error code -43:
        NetCDF: Attribute not found

or in other contexts:

    major: Virtual Object Layer
    minor: Can't open object
  #003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
    major: Attribute
    minor: Can't open object
  #004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header

[danlooo]

Issue is now part ofthe read netcdf4 documentation.