Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GH-35274: [Java][CI] Enable GCS on MacOS #35277

Closed
wants to merge 19 commits into from
Closed

GH-35274: [Java][CI] Enable GCS on MacOS #35277

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
a05f58e
feat: enable gcs on macosx
davisusanibar Apr 21, 2023
acf8e93
Update cpp/build-support/run-test.sh
davisusanibar Apr 24, 2023
d6da99d
fix: Symbol not found: _curl_multi_poll
davisusanibar Apr 24, 2023
df2a7e0
Merge branch 'main' into GH-GCS-OSX
davisusanibar May 12, 2023
26d808d
fix: Symbol not found: _curl_multi_poll
davisusanibar May 12, 2023
3490e48
Merge branch 'main' into GH-GCS-OSX
davisusanibar Jun 5, 2023
2213140
fix: testbench, clean and rebase branch
davisusanibar Jun 5, 2023
ddca08f
Merge branch 'main' into GH-GCS-OSX
davisusanibar Jun 6, 2023
e1db91b
fix: define cmake options for GCS dependencies
davisusanibar Jun 6, 2023
6d75cbd
Merge branch 'main' into GH-GCS-OSX
davisusanibar Jun 6, 2023
6ae8fdd
Merge branch 'main' into GH-GCS-OSX
davisusanibar Jun 15, 2023
5038e14
Merge branch 'main' into GH-GCS-OSX
davisusanibar Jun 28, 2023
c5ad475
fix: update absl build to 20230125.3
davisusanibar Jun 28, 2023
828d689
fix: delete absl build version
davisusanibar Jun 29, 2023
5e996fb
Merge branch 'main' into GH-GCS-OSX
davisusanibar Jul 1, 2023
1deb06b
fix: solve conflicts
davisusanibar Jul 4, 2023
480a896
Merge branch 'main' into GH-GCS-OSX
davisusanibar Jul 14, 2023
49ee27c
Merge branch 'main' into GH-GCS-OSX
davisusanibar Aug 1, 2023
9acf7dc
Merge branch 'main' into GH-GCS-OSX
davisusanibar Sep 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions ci/scripts/java_jni_macos_build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ export ARROW_ACERO
export ARROW_DATASET
: ${ARROW_GANDIVA:=ON}
export ARROW_GANDIVA
: ${ARROW_GCS:=ON}
: ${ARROW_ORC:=ON}
export ARROW_ORC
: ${ARROW_PARQUET:=ON}
Expand All @@ -67,6 +68,7 @@ mkdir -p "${build_dir}/cpp"
pushd "${build_dir}/cpp"

cmake \
-Dabsl_SOURCE=BUNDLED \
-DARROW_ACERO=${ARROW_ACERO} \
-DARROW_BUILD_SHARED=OFF \
-DARROW_BUILD_TESTS=${ARROW_BUILD_TESTS} \
Expand All @@ -75,15 +77,18 @@ cmake \
-DARROW_DEPENDENCY_USE_SHARED=OFF \
-DARROW_GANDIVA=${ARROW_GANDIVA} \
-DARROW_GANDIVA_STATIC_LIBSTDCPP=ON \
-DARROW_GCS=${ARROW_GCS} \
-DARROW_ORC=${ARROW_ORC} \
-DARROW_PARQUET=${ARROW_PARQUET} \
-DARROW_S3=${ARROW_S3} \
-DARROW_USE_CCACHE=${ARROW_USE_CCACHE} \
-DARROW_WITH_SNAPPY=ON \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does Snappy need to be enabled here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At Arrow build, ARROW_WITH_SNAPPY is not added by default and GCS use that codec by default.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If ARROW_WITH_SNAPPY is required for ARROW_GCS, could you enable ARROW_WITH_SNAPPY automatically with ARROW_GCS?

diff --git a/cpp/cmake_modules/DefineOptions.cmake b/cpp/cmake_modules/DefineOptions.cmake
index e2ad9e7a8..247a190ae 100644
--- a/cpp/cmake_modules/DefineOptions.cmake
+++ b/cpp/cmake_modules/DefineOptions.cmake
@@ -335,7 +335,10 @@ takes precedence over ccache if a storage backend is configured" ON)
                 ARROW_WITH_UTF8PROC)
 
   define_option(ARROW_GCS
-                "Build Arrow with GCS support (requires the GCloud SDK for C++)" OFF)
+                "Build Arrow with GCS support (requires the GCloud SDK for C++)"
+                OFF
+                DEPENDS
+                ARROW_WITH_SNAPPY)
 
   define_option(ARROW_HDFS "Build the Arrow HDFS bridge" OFF)

-DAWSSDK_SOURCE=BUNDLED \
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} \
-DCMAKE_INSTALL_LIBDIR=lib \
-DCMAKE_INSTALL_PREFIX=${install_dir} \
-DCMAKE_UNITY_BUILD=${CMAKE_UNITY_BUILD} \
-Dgoogle_cloud_cpp_storage_SOURCE=BUNDLED \
-DGTest_SOURCE=BUNDLED \
-DPARQUET_BUILD_EXAMPLES=OFF \
-DPARQUET_BUILD_EXECUTABLES=OFF \
Expand Down
3 changes: 3 additions & 0 deletions cpp/build-support/run-test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,9 @@ rm -f $LOGFILE $LOGFILE.gz

pipe_cmd=cat

# Install the gcs testbench
sh $ROOT/../ci/scripts/install_gcs_testbench.sh default
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should add this here. Why do you need this change?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we enable ARROW_GCS some test need some software base that is installed by this shell. Just moved inside of MacOS scope.


function setup_sanitizers() {
# Sets environment variables for different sanitizers (it configures how) the run_tests. Function works.

Expand Down
4 changes: 2 additions & 2 deletions cpp/thirdparty/versions.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,8 @@
# `DEPENDENCIES` array (see the comment on top of the declaration for the
# format).

ARROW_ABSL_BUILD_VERSION=20211102.0
ARROW_ABSL_BUILD_SHA256_CHECKSUM=dcf71b9cba8dc0ca9940c4b316a0c796be8fab42b070bb6b7cab62b48f0e66c4
ARROW_ABSL_BUILD_VERSION=20230125.2
lidavidm marked this conversation as resolved.
Show resolved Hide resolved
ARROW_ABSL_BUILD_SHA256_CHECKSUM=9a2b5752d7bfade0bdeee2701de17c9480620f8b237e1964c1b9967c75374906
ARROW_AWS_C_AUTH_BUILD_VERSION=v0.6.22
ARROW_AWS_C_AUTH_BUILD_SHA256_CHECKSUM=691a6b4418afcd3dc141351b6ad33fccd8e3ff84df0e9e045b42295d284ee14c
ARROW_AWS_C_CAL_BUILD_VERSION=v0.5.20
Expand Down
1 change: 1 addition & 0 deletions java/Brewfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,4 @@

brew "openjdk@11"
brew "sccache"
brew "curl"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps keep this file in alphabetical order?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Juste deleted, that was added for testing purpose that at the end we add CURL_ROOT on the github.yml file