Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework the compression library search #5085

Open
wants to merge 28 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
5b42956
Rework compression lib search to match documented process
byrnHDF Nov 7, 2024
e5e34cd
Merge branch 'HDFGroup:develop' into develop-comp-search
byrnHDF Nov 7, 2024
37feafb
Revert for legacy szip
byrnHDF Nov 7, 2024
18af724
Return the legacy find module
byrnHDF Nov 7, 2024
3433678
Remove the entire legacy SZIP find
byrnHDF Nov 7, 2024
9546458
Clarify which library is built
byrnHDF Nov 7, 2024
9deb725
Added release note.
byrnHDF Nov 8, 2024
13eca0f
Merge branch 'HDFGroup:develop' into develop-comp-search
byrnHDF Nov 11, 2024
dfc2333
Merge branch 'HDFGroup:develop' into develop-comp-search
byrnHDF Nov 20, 2024
ed9dfd6
Add release note about NOT FOUND lib error message.
byrnHDF Nov 20, 2024
9041e1d
Merge branch 'HDFGroup:develop' into develop-comp-search
byrnHDF Nov 21, 2024
cf3b44f
Convert default ZLIB option to OFF
byrnHDF Nov 21, 2024
5bcf394
Correct spelling
byrnHDF Nov 21, 2024
f912eeb
netcdf requires zlib
byrnHDF Nov 21, 2024
4d3a788
Match zlib setting to szip setting in workflows
byrnHDF Nov 21, 2024
9c658af
Revert accidental file check-in
byrnHDF Nov 21, 2024
d578c40
Update CMakeFilters.cmake
lrknox Nov 21, 2024
7183cc3
Merge branch 'HDFGroup:develop' into develop-comp-search
byrnHDF Nov 21, 2024
4ce8ada
Move variable declaration inside ifdef block
byrnHDF Nov 22, 2024
183524a
Committing clang-format changes
github-actions[bot] Nov 22, 2024
e214fa3
Merge branch 'HDFGroup:develop' into develop-comp-search
byrnHDF Nov 22, 2024
c015f17
Merge branch 'HDFGroup:develop' into develop-comp-search
byrnHDF Nov 25, 2024
cd902ad
Correct plugins ref
byrnHDF Nov 25, 2024
7af667f
Merge branch 'HDFGroup:develop' into develop-comp-search
byrnHDF Nov 25, 2024
6eb007f
Merge branch 'HDFGroup:develop' into develop-comp-search
byrnHDF Nov 26, 2024
912984d
Merge branch 'HDFGroup:develop' into develop-comp-search
byrnHDF Nov 26, 2024
195d982
Merge branch 'HDFGroup:develop' into develop-comp-search
byrnHDF Nov 27, 2024
999d6c8
Merge branch 'HDFGroup:develop' into develop-comp-search
byrnHDF Nov 27, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/aocc-cmake.yml
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ jobs:
CC=mpicc cmake -C $GITHUB_WORKSPACE/config/cmake/cacheinit.cmake -G Ninja \
-DCMAKE_BUILD_TYPE=${{ inputs.build_mode }} \
-DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF \
-DHDF5_ENABLE_ZLIB_SUPPORT:BOOL=OFF \
-DHDF5_ENABLE_PARALLEL:BOOL=ON \
-DHDF5_ENABLE_SUBFILING_VFD:BOOL=ON \
-DHDF5_BUILD_CPP_LIB:BOOL=OFF \
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/i386-cmake.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ jobs:
cmake -C ../config/cmake/cacheinit.cmake -G "Unix Makefiles" \
--log-level=VERBOSE \
-DCMAKE_BUILD_TYPE=${{ inputs.build_mode }} \
-DHDF5_ENABLE_ZLIB_SUPPORT:BOOL=OFF \
-DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF \
-DHDF5_BUILD_CPP_LIB:BOOL=OFF \
-DLIBAEC_USE_LOCALCONTENT:BOOL=OFF \
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/julia-cmake.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ jobs:
cd "${{ runner.workspace }}/build"
cmake -C $GITHUB_WORKSPACE/config/cmake/cacheinit.cmake -G Ninja \
-DCMAKE_BUILD_TYPE=${{ inputs.build_mode }} \
-DHDF5_ENABLE_ZLIB_SUPPORT:BOOL=OFF \
-DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF \
-DHDF5_ENABLE_PARALLEL:BOOL=OFF \
-DHDF5_BUILD_CPP_LIB:BOOL=OFF \
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/netcdf.yml
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ jobs:
-DHDF5_BUILD_DOC:BOOL=OFF \
-DLIBAEC_USE_LOCALCONTENT:BOOL=OFF \
-DZLIB_USE_LOCALCONTENT:BOOL=OFF \
-DHDF5_ENABLE_ZLIB_SUPPORT:BOOL=ON \
-DH5_NO_DEPRECATED_SYMBOLS:BOOL=OFF \
-DBUILD_TESTING:BOOL=OFF \
-DCMAKE_INSTALL_PREFIX:PATH=/usr/local \
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/nvhpc-cmake.yml
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ jobs:
cd "${{ runner.workspace }}/build"
cmake -C $GITHUB_WORKSPACE/config/cmake/cacheinit.cmake -G Ninja \
-DCMAKE_BUILD_TYPE=${{ inputs.build_mode }} \
-DHDF5_ENABLE_ZLIB_SUPPORT:BOOL=OFF \
-DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF \
-DHDF5_ENABLE_PARALLEL:BOOL=ON \
-DMPIEXEC_NUMPROC_FLAG:STRING=-np \
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/vol_adios2.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ jobs:
-DHDF5_ENABLE_PARALLEL:BOOL=ON \
-DHDF5_ENABLE_THREADSAFE:BOOL=ON \
-DHDF5_ALLOW_UNSUPPORTED:BOOL=ON \
-DHDF5_ENABLE_ZLIB_SUPPORT:BOOL=OFF \
-DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF \
${{ github.workspace }}/hdf5
cat src/libhdf5.settings
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/vol_async.yml
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ jobs:
-DHDF5_ENABLE_PARALLEL:BOOL=ON \
-DHDF5_ENABLE_THREADSAFE:BOOL=ON \
-DHDF5_ALLOW_UNSUPPORTED:BOOL=ON \
-DHDF5_ENABLE_ZLIB_SUPPORT:BOOL=OFF \
-DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF \
-DHDF5_VOL_ALLOW_EXTERNAL:STRING="GIT" \
-DHDF5_VOL_URL01:STRING="https://github.com/HDFGroup/vol-async.git" \
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/vol_cache.yml
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@ jobs:
-DHDF5_ENABLE_PARALLEL:BOOL=ON \
-DHDF5_ENABLE_THREADSAFE:BOOL=ON \
-DHDF5_ALLOW_UNSUPPORTED:BOOL=ON \
-DHDF5_ENABLE_ZLIB_SUPPORT:BOOL=OFF \
-DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF \
-DHDF5_VOL_ALLOW_EXTERNAL:STRING="GIT" \
-DHDF5_VOL_URL01:STRING="https://github.com/HDFGroup/vol-async.git" \
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/vol_ext_passthru.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ jobs:
-DBUILD_STATIC_LIBS=OFF \
-DHDF5_TEST_API:BOOL=ON \
-DHDF5_ENABLE_PARALLEL:BOOL=ON \
-DHDF5_ENABLE_ZLIB_SUPPORT:BOOL=OFF \
-DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF \
-DHDF5_VOL_ALLOW_EXTERNAL:STRING="GIT" \
-DHDF5_VOL_URL01:STRING="https://github.com/hpc-io/vol-external-passthrough.git" \
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/vol_log.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ jobs:
-DHDF5_ENABLE_PARALLEL:BOOL=ON \
-DHDF5_ENABLE_THREADSAFE:BOOL=ON \
-DHDF5_ALLOW_UNSUPPORTED:BOOL=ON \
-DHDF5_ENABLE_ZLIB_SUPPORT:BOOL=OFF \
-DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF \
${{ github.workspace }}/hdf5
cat src/libhdf5.settings
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/vol_rest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,8 +57,8 @@ jobs:
-DHDF5_BUILD_HL_LIB:BOOL=ON \
-DHDF5_TEST_API:BOOL=ON \
-DHDF5_ALLOW_UNSUPPORTED:BOOL=ON \
-DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF \
-DHDF5_ENABLE_ZLIB_SUPPORT:BOOL=OFF \
-DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF \
-DHDF5_VOL_ALLOW_EXTERNAL:STRING="GIT" \
-DHDF5_VOL_URL01:STRING="https://github.com/HDFGroup/vol-rest.git" \
-DHDF5_VOL_VOL-REST_BRANCH:STRING="master" \
Expand Down
33 changes: 20 additions & 13 deletions CMakeFilters.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,13 @@ endif ()
# Option for ZLib support
#-----------------------------------------------------------------------------
set(H5_ZLIB_FOUND FALSE)
option (HDF5_ENABLE_ZLIB_SUPPORT "Enable Zlib Filters" ON)
if(NOT DEFINED ZLIB_PACKAGE_NAME)
set(ZLIB_PACKAGE_NAME "zlib")
endif ()
if(NOT DEFINED ZLIBNG_PACKAGE_NAME)
set(ZLIBNG_PACKAGE_NAME "zlib-ng")
endif ()
option (HDF5_ENABLE_ZLIB_SUPPORT "Enable Zlib Filters" OFF)
if (HDF5_ENABLE_ZLIB_SUPPORT)
if (NOT H5_ZLIB_HEADER)
if (NOT ZLIB_USE_EXTERNAL)
Expand All @@ -88,11 +94,12 @@ if (HDF5_ENABLE_ZLIB_SUPPORT)
endif ()
set(ZLIB_FOUND FALSE)
if (HDF5_USE_ZLIB_STATIC)
set(ZLIB_SEACH_TYPE static)
set(ZLIB_SEARCH_TYPE static)
else ()
set(ZLIB_SEACH_TYPE shared)
set(ZLIB_SEARCH_TYPE shared)
endif ()
find_package (ZLIB NAMES ${PACKAGE_NAME} COMPONENTS ${ZLIB_SEACH_TYPE})
# Search pure Config mode first
find_package (ZLIB NAMES ${PACKAGE_NAME} OPTIONAL_COMPONENTS ${ZLIB_SEARCH_TYPE})
if (NOT ZLIB_FOUND)
if (CMAKE_VERSION VERSION_GREATER_EQUAL "3.24.0")
set(ZLIB_USE_STATIC_LIBS ${HDF5_USE_ZLIB_STATIC})
Expand Down Expand Up @@ -140,7 +147,7 @@ if (HDF5_ENABLE_ZLIB_SUPPORT)
message (VERBOSE "Filter HDF5_ZLIB is ON")
else ()
set (HDF5_ENABLE_ZLIB_SUPPORT OFF CACHE BOOL "" FORCE)
message (WARNING " ZLib support in HDF5 was enabled but not found")
message (FATAL_ERROR " ZLib support in HDF5 was enabled but not found")
endif ()
message(STATUS "H5_ZLIB_HEADER=${H5_ZLIB_HEADER}")
endif ()
Expand All @@ -149,7 +156,10 @@ endif ()
# Option for SzLib support
#-----------------------------------------------------------------------------
set(H5_SZIP_FOUND FALSE)
option (HDF5_ENABLE_SZIP_SUPPORT "Use SZip Filter" ON)
if(NOT DEFINED LIBAEC_PACKAGE_NAME)
set(LIBAEC_PACKAGE_NAME "libaec")
endif ()
option (HDF5_ENABLE_SZIP_SUPPORT "Use SZip Filter" OFF)
if (HDF5_ENABLE_SZIP_SUPPORT)
option (HDF5_ENABLE_SZIP_ENCODING "Use SZip Encoding" ON)
if (NOT SZIP_USE_EXTERNAL)
Expand All @@ -160,10 +170,8 @@ if (HDF5_ENABLE_SZIP_SUPPORT)
endif ()
set(libaec_USE_STATIC_LIBS ${HDF5_USE_LIBAEC_STATIC})
set(SZIP_FOUND FALSE)
find_package (SZIP NAMES ${LIBAEC_PACKAGE_NAME}${HDF_PACKAGE_EXT} COMPONENTS ${LIBAEC_SEARCH_TYPE})
if (NOT SZIP_FOUND)
find_package (SZIP) # Legacy find
endif ()
# Search pure Config mode, there is not a FindSZIP module available
find_package (SZIP NAMES ${LIBAEC_PACKAGE_NAME}${HDF_PACKAGE_EXT} OPTIONAL_COMPONENTS ${LIBAEC_SEARCH_TYPE})
set(H5_SZIP_FOUND ${SZIP_FOUND})
if (H5_SZIP_FOUND)
set (H5_SZIP_INCLUDE_DIR_GEN ${SZIP_INCLUDE_DIR})
Expand All @@ -173,8 +181,7 @@ if (HDF5_ENABLE_SZIP_SUPPORT)
else ()
if (HDF5_ALLOW_EXTERNAL_SUPPORT MATCHES "GIT" OR HDF5_ALLOW_EXTERNAL_SUPPORT MATCHES "TGZ")
EXTERNAL_SZIP_LIBRARY (${HDF5_ALLOW_EXTERNAL_SUPPORT} ${HDF5_ENABLE_SZIP_ENCODING})
message (VERBOSE "Filter SZIP is built")
message (VERBOSE "... with library AEC")
message (VERBOSE "Filter SZIP is built using library AEC")
set (LINK_COMP_LIBS ${LINK_COMP_LIBS} ${H5_SZIP_STATIC_LIBRARY})
endif ()
endif ()
Expand All @@ -193,6 +200,6 @@ if (HDF5_ENABLE_SZIP_SUPPORT)
endif ()
else ()
set (HDF5_ENABLE_SZIP_SUPPORT OFF CACHE BOOL "" FORCE)
message (WARNING "SZIP support in HDF5 was enabled but not found")
message (FATAL_ERROR "SZIP support in HDF5 was enabled but not found")
endif ()
endif ()
4 changes: 2 additions & 2 deletions config/cmake/scripts/HDF5options.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,8 @@ set (ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_ALLOW_EXTERNAL_SUPPORT:STRIN
#set(ENV{SZIP_ROOT} "some_location")
#set (ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DH5_SZIP_LIBRARY:FILEPATH=some_location/lib/szlib.lib -DH5_SZIP_INCLUDE_DIR:PATH=some_location/include")

### disable using ext zlib
#set (ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_ENABLE_ZLIB_SUPPORT:BOOL=OFF")
### enable using ext zlib
#set (ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_ENABLE_ZLIB_SUPPORT:BOOL=ON")
### disable using ext szip
#set (ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF")
#set (ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_ENABLE_SZIP_ENCODING:BOOL=OFF")
Expand Down
4 changes: 2 additions & 2 deletions release_docs/INSTALL_CMake.txt
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ Download from https://github.com/HDFGroup/hdf5/blob/develop/config/cmake/scripts
External compression plugin libraries from https://github.com/HDFGroup/hdf5_plugins:
hdf5_plugins.tar.gz

External compression szip and zlib libraries:
External compression szip and zlib libraries:0
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Accidental?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most likely yes.

ZLIB "https://github.com/madler/zlib/releases/download/v1.3.1/zlib-1.3.1.tar.gz"
ZLIBNG "https://github.com/zlib-ng/zlib-ng/archive/refs/tags/2.2.2.tar.gz"
LIBAEC "https://github.com/MathisRosenhauer/libaec/releases/download/v1.1.3/libaec-1.1.3.tar.gz"
Expand Down Expand Up @@ -939,7 +939,7 @@ if (BUILD_TESTING)
HDF5_ALLOW_EXTERNAL_SUPPORT "Allow External Library Building (NO GIT TGZ)" "NO"
HDF5_ENABLE_PLUGIN_SUPPORT "Enable PLUGIN Filters" OFF
HDF5_ENABLE_SZIP_SUPPORT "Use SZip Filter" OFF
HDF5_ENABLE_ZLIB_SUPPORT "Enable Zlib Filters" ON
HDF5_ENABLE_ZLIB_SUPPORT "Enable Zlib Filters" OFF

if (HDF5_USE_ZLIB_NG)
ZLIBNG_USE_EXTERNAL "Use External Library Building for ZLIBNG" OFF
Expand Down
14 changes: 13 additions & 1 deletion release_docs/RELEASE.txt
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ New Features
- Renamed the option: HDF5_ENABLE_Z_LIB_SUPPORT

The option has been renamed to HDF5_ENABLE_ZLIB_SUPPORT to be consistent
with the naming of other options.
with the naming of other options. Also, the option defaults to OFF.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely want to make sure this is very well called out for the next release

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Absolutely


- Added support for MinGW + MSYS2 when building with CMake

Expand Down Expand Up @@ -510,6 +510,18 @@ Bug Fixes since HDF5-2.0.0 release

Configuration
-------------
- Changed the zlib/szip compression find message to FATAL ERROR

The message was changed to indicate that zlib/szip compression was requested and
that it was not found. If an option is requested, not finding it should always
be an error.

- Removed the module search find_package for szip library

There is not a szip module file to use, so the find_package only uses
find_package in config mode. The choice then is to either build szip, with libaec,
inline or find a system installed szip library, built with CMake.

- Changed name of libhdf5hl_fortran installed by autotools to libhdf5_hl_fortran. The
new name is consistent with the name of the lib when installed by CMake and with the
other hl libs.
Expand Down
3 changes: 2 additions & 1 deletion test/dsets.c
Original file line number Diff line number Diff line change
Expand Up @@ -2845,13 +2845,14 @@ test_missing_filter(hid_t file)
size_t i, j; /* Local index variables */
herr_t ret; /* Generic return value */
const char *testfile = H5_get_srcdir_filename(FILE_DEFLATE_NAME); /* Corrected test file name */
H5CX_node_t api_ctx = {{0}, NULL}; /* API context node to push */
bool api_ctx_pushed = false; /* Whether API context pushed */

TESTING("dataset access with missing filter");

/* Unregister the deflate filter */
#ifdef H5_HAVE_FILTER_DEFLATE
H5CX_node_t api_ctx = {{0}, NULL}; /* API context node to push */

/* Verify deflate filter is registered currently */
if (H5Zfilter_avail(H5Z_FILTER_DEFLATE) != true) {
H5_FAILED();
Expand Down
Loading