GH-35274: [Java][CI] Enable GCS on MacOS #35277

davisusanibar · 2023-04-21T20:28:46Z

Rationale for this change

Enables GCS on MacOS when building the Arrow Dataset for Java.

What changes are included in this PR?

CI build to enable GCS

Are these changes tested?

import org.apache.arrow.dataset.file.FileFormat;
import org.apache.arrow.dataset.file.FileSystemDatasetFactory;
import org.apache.arrow.dataset.jni.NativeMemoryPool;
import org.apache.arrow.dataset.scanner.ScanOptions;
import org.apache.arrow.dataset.scanner.Scanner;
import org.apache.arrow.dataset.source.Dataset;
import org.apache.arrow.dataset.source.DatasetFactory;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.memory.RootAllocator;
import org.apache.arrow.vector.ipc.ArrowReader;
import org.apache.arrow.vector.types.pojo.Schema;

public class Test {
  public static void main(String[] args) {
    String uri = "gs://anonymous@voltrondata-labs-datasets/nyc-taxi-tiny/year=2022/month=2/part-0.parquet"; // Google Cloud Storage
    ScanOptions options = new ScanOptions(/*batchSize*/ 32768);
    try (
        BufferAllocator allocator = new RootAllocator();
        DatasetFactory datasetFactory = new FileSystemDatasetFactory(allocator, NativeMemoryPool.getDefault(), FileFormat.PARQUET, uri);
        Dataset dataset = datasetFactory.finish();
        Scanner scanner = dataset.newScan(options);
        ArrowReader reader = scanner.scanBatches()
    ) {
      Schema schema = scanner.schema();
      System.out.println(schema); // Schema<vendor_name: Utf8, pickup_datetime: Timestamp(MILLISECOND, null), dropoff_datetime: Timestamp(MILLISECOND, null), passenger_count: Int(64, true), trip_distance: FloatingPoint(DOUBLE), pickup_longitude: FloatingPoint(DOUBLE), pickup_latitude: FloatingPoint(DOUBLE), rate_code: Utf8, store_and_fwd: Utf8, dropoff_longitude: FloatingPoint(DOUBLE), dropoff_latitude: FloatingPoint(DOUBLE), payment_type: Utf8, fare_amount: FloatingPoint(DOUBLE), extra: FloatingPoint(DOUBLE), mta_tax: FloatingPoint(DOUBLE), tip_amount: FloatingPoint(DOUBLE), tolls_amount: FloatingPoint(DOUBLE), total_amount: FloatingPoint(DOUBLE), improvement_surcharge: FloatingPoint(DOUBLE), congestion_surcharge: FloatingPoint(DOUBLE), pickup_location_id: Int(64, true), dropoff_location_id: Int(64, true)>
      while (reader.loadNextBatch()) {
        System.out.println("RowCount: " + reader.getVectorSchemaRoot().getRowCount()); //2979
      }
    } catch (Exception e) {
      e.printStackTrace();
    }
  }
}

Are there any user-facing changes?

No

Closes: [Java][Dataset][MacOS] Enable GCS arrow-java#202
GitHub Issue: ARROW-373: [C++] JSON serialization format for testing #202

github-actions · 2023-04-21T20:29:10Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

github-actions · 2023-04-21T20:33:14Z

Closes: [Java][Dataset][MacOS] Enable GCS arrow-java#202

github-actions · 2023-04-21T20:33:16Z

⚠️ GitHub issue apache/arrow-java#202 has been automatically assigned in GitHub to PR creator.

kou · 2023-04-21T22:08:33Z

@github-actions crossbow submit java-jars

github-actions · 2023-04-21T22:10:53Z

Revision: a05f58e

Submitted crossbow builds: ursacomputing/crossbow @ actions-24f328b4cf

Task	Status
java-jars

assignUser

A minor change

cpp/build-support/run-test.sh

davisusanibar · 2023-04-22T05:18:43Z

I am seeing error on MacOS aarch_64 with message dyld: lazy symbol binding failed: Symbol not found: _curl_multi_poll @kou / @assignUser Do you have any idea about the reason of?

Running arrow-gcsfs-test, redirecting output into /Users/voltrondata/github-actions-runner/_work/crossbow/crossbow/arrow/cpp-build/cpp/build/test-logs/arrow-gcsfs-test.txt (attempt 1/1)
/Users/voltrondata/github-actions-runner/_work/crossbow/crossbow/arrow/cpp/build-support/run-test.sh: line 91: 33793 Abort trap: 6           $TEST_EXECUTABLE "$@" > $LOGFILE.raw 2>&1
INFO:werkzeug:WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://localhost:63593
INFO:werkzeug:Press CTRL+C to quit
INFO:werkzeug: * Restarting with stat
Running main() from /Users/voltrondata/github-actions-runner/_work/crossbow/crossbow/arrow/cpp-build/cpp/googletest_ep-prefix/src/googletest_ep/googletest/src/gtest_main.cc
[==========] Running 99 tests from 3 test suites.
[----------] Global test environment set-up.
[----------] 26 tests from TestGCSFSGeneric
[ RUN      ] TestGCSFSGeneric.Empty
dyld: lazy symbol binding failed: Symbol not found: _curl_multi_poll
  Referenced from: /Users/voltrondata/github-actions-runner/_work/crossbow/crossbow/arrow/cpp-build/cpp/release/arrow-gcsfs-test
  Expected in: /usr/lib/libcurl.4.dylib

dyld: Symbol not found: _curl_multi_poll
  Referenced from: /Users/voltrondata/github-actions-runner/_work/crossbow/crossbow/arrow/cpp-build/cpp/release/arrow-gcsfs-test
  Expected in: /usr/lib/libcurl.4.dylib

...
...

Total Test time (real) = 121.27 sec

The following tests FAILED:
	 61 - arrow-gcsfs-test (Failed)
Errors while running CTest

assignUser · 2023-04-22T05:25:09Z

I think the installed curl version is too old. It looks like libcurl 4 is installed but multi pool was introduced in 7 https://curl.se/libcurl/c/curl_multi_poll.html

Co-authored-by: Jacob Wujciak-Jens <jacob@wujciak.de>

davisusanibar · 2023-04-24T16:14:33Z

@github-actions crossbow submit java-jars

github-actions · 2023-04-24T16:17:40Z

Revision: d6da99d

Submitted crossbow builds: ursacomputing/crossbow @ actions-fa53b98180

Task	Status
java-jars

cpp/thirdparty/versions.txt

assignUser · 2023-05-10T03:39:47Z

This should fix the m1 issues

--- a/dev/tasks/java-jars/github.yml
+++ b/dev/tasks/java-jars/github.yml
@@ -108,6 +108,7 @@ jobs:
           set -e
           # make brew Java available to CMake
           if [ "{{ arch }}" = "aarch_64" ]; then
+            export CURL_ROOT=$(brew --prefix curl)
             export JAVA_HOME=$(brew --prefix openjdk@11)/libexec/openjdk.jdk/Contents/Home
           fi
           arrow/ci/scripts/java_jni_macos_build.sh \

davisusanibar · 2023-06-06T03:57:50Z

@github-actions crossbow submit java-jars

github-actions · 2023-06-06T04:00:11Z

Revision: e1db91b

Submitted crossbow builds: ursacomputing/crossbow @ actions-7a70b20d3a

Task	Status
java-jars

davisusanibar · 2023-06-15T04:15:02Z

@lidavidm do you have some insight about what I could review to solve this error https://github.com/apache/arrow/actions/runs/5274483912/jobs/9538989549?pr=35277:

[70/122] Generating arrow-flight-glib/ArrowFlight-1.0.gir with a custom command (wrapped by meson to set env)
FAILED: arrow-flight-glib/ArrowFlight-1.0.gir 
env PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:/build/c_glib/meson-uninstalled /usr/bin/g-ir-scanner --quiet --no-libtool --namespace=ArrowFlight --nsversion=1.0 --warn-all --output arrow-flight-glib/ArrowFlight-1.0.gir --c-include=arrow-flight-glib/arrow-flight-glib.h --warn-all --include-uninstalled=./arrow-glib/Arrow-1.0.gir -I/arrow/c_glib/arrow-flight-glib -I/build/c_glib/arrow-flight-glib -I/arrow/c_glib/. -I/build/c_glib/. --filelist=/build/c_glib/arrow-flight-glib/libarrow-flight-glib.so.1300.0.0.p/ArrowFlight_1.0_gir_filelist --include=Arrow-1.0 --symbol-prefix=gaflight --identifier-prefix=GAFlight --pkg-export=arrow-flight-glib --cflags-begin -I/arrow/c_glib/. -I/build/c_glib/. -I/usr/local/include -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/gobject-introspection-1.0 -DARROW_NO_DEPRECATED_API --cflags-end --add-include-path=/build/c_glib/arrow-glib --add-include-path=/usr/share/gir-1.0 -L/build/c_glib/arrow-flight-glib --library arrow-flight-glib -L/build/c_glib/arrow-glib -L/usr/local/lib -L/usr/local/lib --extra-library=arrow_flight --extra-library=arrow --extra-library=arrow_acero --extra-library=gobject-2.0 --extra-library=glib-2.0 --extra-library=girepository-1.0 --sources-top-dirs /arrow/c_glib/ --sources-top-dirs /build/c_glib/ --warn-error
/usr/bin/ld: /usr/local/lib/libarrow_flight.so: undefined reference to `absl::lts_20230125::crc_internal::CrcCordState::CrcCordState()'
/usr/bin/ld: /usr/local/lib/libarrow_flight.so: undefined reference to `absl::lts_20230125::crc_internal::CrcCordState::operator=(absl::lts_20230125::crc_internal::CrcCordState&&)'
/usr/bin/ld: /usr/local/lib/libarrow_flight.so: undefined reference to `absl::lts_20230125::crc_internal::CrcCordState::CrcCordState(absl::lts_20230125::crc_internal::CrcCordState&&)'
/usr/bin/ld: /usr/local/lib/libarrow_flight.so: undefined reference to `absl::lts_20230125::base_internal::StrError[abi:cxx11](int)'
/usr/bin/ld: /usr/local/lib/libarrow_flight.so: undefined reference to `absl::lts_20230125::crc_internal::CrcCordState::~CrcCordState()'
/usr/bin/ld: /usr/local/lib/libarrow_flight.so: undefined reference to `absl::lts_20230125::crc_internal::CrcCordState::Checksum() const'
collect2: error: ld returned 1 exit status
linking of temporary binary failed: Command '['x86_64-linux-gnu-gcc', '-pthread', '-o', '/build/c_glib/tmp-introspectci4byn1o/ArrowFlight-1.0', '-DARROW_NO_DEPRECATED_API', '/build/c_glib/tmp-introspectci4byn1o/ArrowFlight-1.0.o', '-L.', '-Wl,-rpath,.', '-Wl,--no-as-needed', '-L/build/c_glib/arrow-flight-glib', '-Wl,-rpath,/build/c_glib/arrow-flight-glib', '-L/build/c_glib/arrow-glib', '-Wl,-rpath,/build/c_glib/arrow-glib', '-L/usr/local/lib', '-Wl,-rpath,/usr/local/lib', '-L/usr/local/lib', '-Wl,-rpath,/usr/local/lib', '-larrow-flight-glib', '-larrow_flight', '-larrow', '-larrow_acero', '-lgobject-2.0', '-lglib-2.0', '-lgirepository-1.0', '-lgio-2.0', '-lgobject-2.0', '-Wl,--export-dynamic', '-lgmodule-2.0', '-pthread', '-lglib-2.0']' returned non-zero exit status 1.
[71/122] Compiling C++ object gandiva-glib/libgandiva-glib.so.1300.0.0.p/expression.cpp.o
[72/122] Compiling C++ object arrow-flight-sql-glib/libarrow-flight-sql-glib.so.1300.0.0.p/server.cpp.o
ninja: build stopped: subcommand failed.

kou · 2023-06-15T05:56:52Z

Hmm. It's not reproduced on local...

It seems that the error message show that we need to add Abseil's CRC related modules to https://github.com/apache/arrow/blob/main/cpp/cmake_modules/ThirdpartyToolchain.cmake#L2856-L3756 .

BTW, can we use 20230125.3 https://github.com/abseil/abseil-cpp/releases/tag/20230125.3 ?

davisusanibar · 2023-06-30T15:39:54Z

@github-actions crossbow submit java-jars

github-actions · 2023-06-30T15:42:19Z

Revision: 828d689

Submitted crossbow builds: ursacomputing/crossbow @ actions-221f6dfca1

Task	Status
java-jars

davisusanibar · 2023-07-03T20:53:55Z

@lidavidm please could you help me to move this PR to draft?, there are some problems on the CI that I need to test and probably need to call it again and again. Thank you in advance

kou · 2023-07-03T21:14:59Z

You can find the "Convert to draft" link in the right side bar.

davisusanibar · 2023-07-03T21:23:29Z

You can find the "Convert to draft" link in the right side bar.

Thank you

davisusanibar · 2023-07-05T07:29:58Z

@github-actions crossbow submit java-jars

github-actions · 2023-07-05T07:32:22Z

Revision: 1deb06b

Submitted crossbow builds: ursacomputing/crossbow @ actions-abbebdf72b

Task	Status
java-jars

davisusanibar · 2023-07-15T00:19:22Z

@github-actions crossbow submit java-jars

github-actions · 2023-07-15T00:21:54Z

Revision: 480a896

Submitted crossbow builds: ursacomputing/crossbow @ actions-54eb93bef5

Task	Status
java-jars

davisusanibar · 2023-09-05T05:47:02Z

@github-actions crossbow submit java-jars

github-actions · 2023-09-05T05:49:29Z

Revision: 9acf7dc

Submitted crossbow builds: ursacomputing/crossbow @ actions-8d83452f61

Task	Status
java-jars

davisusanibar · 2024-03-05T13:12:37Z

In order to define a better alternative solution, the current pull request has been closed.

github-actions · 2024-11-26T19:23:08Z

⚠️ GitHub issue #35274 has no components, please add labels for components.

feat: enable gcs on macosx

a05f58e

davisusanibar requested review from assignUser, kou and raulcd as code owners April 21, 2023 20:28

github-actions bot added Component: C++ awaiting review Awaiting review labels Apr 21, 2023

davisusanibar mentioned this pull request Apr 21, 2023

GH-35245: [Java][Dataset][Linux] Enable GCS #35246

Merged

davisusanibar changed the title ~~feat: enable GCS on macosx~~ GH-35274: [Java][CI] Enable GCS on MacOS Apr 21, 2023

assignUser reviewed Apr 22, 2023

View reviewed changes

cpp/build-support/run-test.sh Outdated Show resolved Hide resolved

github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Apr 22, 2023

davisusanibar and others added 2 commits April 24, 2023 09:47

Update cpp/build-support/run-test.sh

acf8e93

Co-authored-by: Jacob Wujciak-Jens <jacob@wujciak.de>

fix: Symbol not found: _curl_multi_poll

d6da99d

davisusanibar requested a review from lidavidm as a code owner April 24, 2023 15:11

github-actions bot added the Component: Java label Apr 24, 2023

lidavidm reviewed Apr 24, 2023

View reviewed changes

cpp/thirdparty/versions.txt Outdated Show resolved Hide resolved

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Apr 24, 2023

davisusanibar added 2 commits May 12, 2023 12:09

Merge branch 'main' into GH-GCS-OSX

df2a7e0

fix: Symbol not found: _curl_multi_poll

26d808d

github-actions bot added the awaiting change review Awaiting change review label Jun 6, 2023

davisusanibar added 2 commits June 6, 2023 12:28

Merge branch 'main' into GH-GCS-OSX

6d75cbd

Merge branch 'main' into GH-GCS-OSX

6ae8fdd

davisusanibar added 3 commits June 28, 2023 11:13

Merge branch 'main' into GH-GCS-OSX

5038e14

fix: update absl build to 20230125.3

c5ad475

fix: delete absl build version

828d689

Merge branch 'main' into GH-GCS-OSX

5e996fb

davisusanibar marked this pull request as draft July 3, 2023 21:23

fix: solve conflicts

1deb06b

Merge branch 'main' into GH-GCS-OSX

480a896

davisusanibar added 2 commits August 1, 2023 16:09

Merge branch 'main' into GH-GCS-OSX

49ee27c

Merge branch 'main' into GH-GCS-OSX

9acf7dc

davisusanibar closed this Mar 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-35274: [Java][CI] Enable GCS on MacOS #35277

GH-35274: [Java][CI] Enable GCS on MacOS #35277

davisusanibar commented Apr 21, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Apr 21, 2023

github-actions bot commented Apr 21, 2023

github-actions bot commented Apr 21, 2023

kou commented Apr 21, 2023

github-actions bot commented Apr 21, 2023

assignUser left a comment

davisusanibar commented Apr 22, 2023

assignUser commented Apr 22, 2023

davisusanibar commented Apr 24, 2023

github-actions bot commented Apr 24, 2023

assignUser commented May 10, 2023 •

edited

Loading

davisusanibar commented Jun 6, 2023

github-actions bot commented Jun 6, 2023

davisusanibar commented Jun 15, 2023

kou commented Jun 15, 2023

davisusanibar commented Jun 30, 2023

github-actions bot commented Jun 30, 2023

davisusanibar commented Jul 3, 2023

kou commented Jul 3, 2023

davisusanibar commented Jul 3, 2023

davisusanibar commented Jul 5, 2023

github-actions bot commented Jul 5, 2023

davisusanibar commented Jul 15, 2023

github-actions bot commented Jul 15, 2023

davisusanibar commented Sep 5, 2023

github-actions bot commented Sep 5, 2023

davisusanibar commented Mar 5, 2024

github-actions bot commented Nov 26, 2024

GH-35274: [Java][CI] Enable GCS on MacOS #35277

GH-35274: [Java][CI] Enable GCS on MacOS #35277

Conversation

davisusanibar commented Apr 21, 2023 • edited by github-actions bot Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

github-actions bot commented Apr 21, 2023

github-actions bot commented Apr 21, 2023

github-actions bot commented Apr 21, 2023

kou commented Apr 21, 2023

github-actions bot commented Apr 21, 2023

assignUser left a comment

Choose a reason for hiding this comment

davisusanibar commented Apr 22, 2023

assignUser commented Apr 22, 2023

davisusanibar commented Apr 24, 2023

github-actions bot commented Apr 24, 2023

assignUser commented May 10, 2023 • edited Loading

davisusanibar commented Jun 6, 2023

github-actions bot commented Jun 6, 2023

davisusanibar commented Jun 15, 2023

kou commented Jun 15, 2023

davisusanibar commented Jun 30, 2023

github-actions bot commented Jun 30, 2023

davisusanibar commented Jul 3, 2023

kou commented Jul 3, 2023

davisusanibar commented Jul 3, 2023

davisusanibar commented Jul 5, 2023

github-actions bot commented Jul 5, 2023

davisusanibar commented Jul 15, 2023

github-actions bot commented Jul 15, 2023

davisusanibar commented Sep 5, 2023

github-actions bot commented Sep 5, 2023

davisusanibar commented Mar 5, 2024

github-actions bot commented Nov 26, 2024

davisusanibar commented Apr 21, 2023 •

edited by github-actions bot

Loading

assignUser commented May 10, 2023 •

edited

Loading