GH-34742: [Java] Split flight-sql-jdbc-driver to facilitate reuse #34678

wgtmac · 2023-03-22T06:51:35Z

Rationale for this change

flight-sql-jdbc-driver module contains a set of utilities to implement JDBC support backed by arrow vectors. However, it employs maven shade plugin which is hard to make it a dependecy of external projects.

What changes are included in this PR?

This patch splits flight-sql-jdbc-driver module into two separate modules: flight-sql-jdbc-core and flight-sql-jdbc-driver. flight-sql-jdbc-core contains almost every thing in the original module except the shade plugin. flight-sql-jdbc-driver only shades flight-sql-jdbc-core.

Are these changes tested?

Make sure all tests pass.

Are there any user-facing changes?

No.

Closes: [Java] Arrow-to-JDBC arrow-java#363
Closes: [Java] Split flight-sql-jdbc-driver to allow reuse of some internals #34742
GitHub Issue: [Java] Split flight-sql-jdbc-driver to allow reuse of some internals #34742
GitHub Issue: [Java] Split flight-sql-jdbc-driver to allow reuse of some internals #34742

github-actions · 2023-03-22T06:52:04Z

Closes: [Java] Arrow-to-JDBC arrow-java#363

github-actions · 2023-03-22T06:52:06Z

⚠️ GitHub issue apache/arrow-java#363 has been automatically assigned in GitHub to PR creator.

flight-sql-jdbc-driver module contains a set of utilities to implement JDBC support backed by arrow vectors. However, it employs maven shade plugin which is hard to use. This patch splits it into two separate modules.

wgtmac · 2023-03-22T08:38:10Z

@lidavidm Could you please take a look? Thanks!

lidavidm · 2023-03-22T19:03:15Z

@davisusanibar or @lwhite1 would either of you be able to take a first pass?

Also,

arrow/dev/tasks/tasks.yml

Lines 799 to 803 in ce0d20c

    
           java-jars: 
        
             # Build jar's that contains cpp libraries dependencies 
        
             ci: github 
        
             template: java-jars/github.yml 
        
             artifacts:

needs to be updated with new artifacts

wgtmac · 2023-03-23T01:51:55Z

arrow/dev/tasks/tasks.yml

Lines 799 to 803 in ce0d20c

java-jars:

# Build jar's that contains cpp libraries dependencies

ci: github

template: java-jars/github.yml

artifacts:

needs to be updated with new artifacts

Fixed tasks.yml, thanks @lidavidm!

BTW, though this PR looks very large by affecting hundreds of files, it contains only 200- lines of change as mostly are file relocation.

assignUser · 2023-03-23T01:54:07Z

@github-actions crossbow submit java-jars

github-actions · 2023-03-23T01:56:34Z

Revision: d382b46

Submitted crossbow builds: ursacomputing/crossbow @ actions-5a76b30651

Task	Status
java-jars

wgtmac · 2023-03-23T03:59:49Z

Revision: d382b46

Submitted crossbow builds: ursacomputing/crossbow @ actions-5a76b30651

Task Status
java-jars

                  flight-sql-jdbc-core-12.0.0-SNAPSHOT-cyclonedx.json [     OK]
                   flight-sql-jdbc-core-12.0.0-SNAPSHOT-cyclonedx.xml [     OK]
                     flight-sql-jdbc-core-12.0.0-SNAPSHOT-javadoc.jar [     OK]
                     flight-sql-jdbc-core-12.0.0-SNAPSHOT-sources.jar [     OK]
                       flight-sql-jdbc-core-12.0.0-SNAPSHOT-tests.jar [     OK]
                             flight-sql-jdbc-core-12.0.0-SNAPSHOT.jar [     OK]
                             flight-sql-jdbc-core-12.0.0-SNAPSHOT.pom [     OK]
                flight-sql-jdbc-driver-12.0.0-SNAPSHOT-cyclonedx.json [     OK]
                 flight-sql-jdbc-driver-12.0.0-SNAPSHOT-cyclonedx.xml [     OK]
                   flight-sql-jdbc-driver-12.0.0-SNAPSHOT-javadoc.jar [PENDING]
                   flight-sql-jdbc-driver-12.0.0-SNAPSHOT-sources.jar [     OK]
                     flight-sql-jdbc-driver-12.0.0-SNAPSHOT-tests.jar [     OK]
                           flight-sql-jdbc-driver-12.0.0-SNAPSHOT.jar [     OK]
                           flight-sql-jdbc-driver-12.0.0-SNAPSHOT.pom [     OK]
Error: Process completed with exit code 1.

flight-sql-jdbc-driver will not create flight-sql-jdbc-driver-12.0.0-SNAPSHOT-javadoc.jar because it only shades flight-sql-jdbc-core. So I removed it from tasks.yml.

@assignUser Should we rerun the crossbow build?

github-actions · 2023-03-23T05:14:02Z

Revision: 7067c95

Submitted crossbow builds: ursacomputing/crossbow @ actions-c5e181f11b

Task	Status
java-jars

wgtmac · 2023-03-23T07:44:36Z

Revision: 7067c95

Submitted crossbow builds: ursacomputing/crossbow @ actions-c5e181f11b

Task Status
java-jars

Downloaded from apache: https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/19.0/guava-parent-19.0.pom (9.9 kB at 1.2 MB/s)
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Arrow Java Root POM 12.0.0-SNAPSHOT:
[INFO] 
[INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 45.069 s]
[INFO] Arrow Format ....................................... SUCCESS [ 16.880 s]
[INFO] Arrow Memory ....................................... SUCCESS [  5.681 s]
[INFO] Arrow Memory - Core ................................ SUCCESS [ 26.060 s]
[INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 12.697 s]
[INFO] Arrow Memory - Netty ............................... SUCCESS [ 18.038 s]
[INFO] Arrow Vectors ...................................... FAILURE [10:15 min]
[INFO] Arrow Compression .................................. SKIPPED
[INFO] Arrow Tools ........................................ SKIPPED
[INFO] Arrow JDBC Adapter ................................. SKIPPED
[INFO] Arrow Plasma Client ................................ SKIPPED
[INFO] Arrow Flight ....................................... SKIPPED
[INFO] Arrow Flight Core .................................. SKIPPED
[INFO] Arrow Flight GRPC .................................. SKIPPED
[INFO] Arrow Flight SQL ................................... SKIPPED
[INFO] Arrow Flight SQL JDBC Driver Core .................. SKIPPED
[INFO] Arrow Flight SQL JDBC Driver ....................... SKIPPED
[INFO] Arrow Flight Integration Tests ..................... SKIPPED
[INFO] Arrow AVRO Adapter ................................. SKIPPED
[INFO] Arrow Algorithms ................................... SKIPPED
[INFO] Arrow Performance Benchmarks ....................... SKIPPED
[INFO] Arrow Java C Data Interface ........................ SKIPPED
[INFO] Arrow Orc Adapter .................................. SKIPPED
[INFO] Arrow Gandiva ...................................... SKIPPED
[INFO] Arrow Java Dataset ................................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  12:21 min
[INFO] Finished at: 2023-03-23T06:29:00Z
[INFO] ------------------------------------------------------------------------
Error:  Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:3.1.1:shade (default) on project arrow-vector: Execution default of goal org.apache.maven.plugins:maven-shade-plugin:3.1.1:shade failed: Plugin org.apache.maven.plugins:maven-shade-plugin:3.1.1 or one of its dependencies could not be resolved: Failed to collect dependencies at org.apache.maven.plugins:maven-shade-plugin:jar:3.1.1 -> org.ow2.asm:asm:jar:6.0: Failed to read artifact descriptor for org.ow2.asm:asm:jar:6.0: Could not transfer artifact org.ow2.asm:asm:pom:6.0 from/to apache (https://repo.maven.apache.org/maven2/): Connection reset -> [Help 1]
Error:  
Error:  To see the full stack trace of the errors, re-run Maven with the -e switch.
Error:  Re-run Maven using the -X switch to enable full debug logging.
Error:  
Error:  For more information about the errors and possible solutions, please read the following articles:
Error:  [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException
Error:  
Error:  After correcting the problems, you can resume the build with the command
Error:    mvn <args> -rf :arrow-vector
Error: Process completed with exit code 1.

Failure is in arrow-vector which seems unrelated.

kou · 2023-03-24T00:14:58Z

I re-ran the job and the job passed. :-)

wgtmac · 2023-03-24T01:23:45Z

cc @sunchao :)

assignUser

+1 on CI changes, can't comment on the rest :)

sunchao

Makes sense. It seems currently flight-sql-jdbc-driver just shade everything so it's difficult for downstream projects to manage dependencies (e.g., exclude certain dependency from flight-sql-jdbc-driver).

I'm not a regular reviewer for Arrow-Java though so it's better for @lidavidm to make the final call.

lidavidm

LGTM. However, our testing of the driver artifact itself is nonexistent. We should consider setting up a job that takes the built driver JAR and runs it against the dremio-oss Docker container or similar.

lidavidm · 2023-03-24T18:55:05Z

java/flight/flight-sql-jdbc-core/pom.xml

+
+    <artifactId>flight-sql-jdbc-core</artifactId>
+    <name>Arrow Flight SQL JDBC Driver Core</name>
+    <description>(Contrib/Experimental) A JDBC driver based on Arrow Flight SQL.</description>


nit: can we update this to reflect that this isn't the actual driver?

Slightly modified the description. Please correct me if there is any better description.

wgtmac · 2023-03-27T02:08:10Z

Do you have any idea why the flight-sql-jdbc-core test is easy to fail? @lidavidm

https://github.com/apache/arrow/actions/runs/4527686135/jobs/7973768071?pr=34678

lidavidm · 2023-03-27T02:11:02Z

The test probably needs some debugging, but I have very little time for Java work these days (it's mostly on my personal time). There haven't been substantial contributions to the driver since its initial contribution.

wgtmac · 2023-03-27T06:19:55Z

The test probably needs some debugging, but I have very little time for Java work these days (it's mostly on my personal time). There haven't been substantial contributions to the driver since its initial contribution.

I tried to debug it but cannot reproduce on my Mac. It seems to be unstable and now the tests are all green.

lidavidm · 2023-03-27T12:09:26Z

Thanks for looking. I filed apache/arrow-java#210 for anyone who wants to follow up.

lidavidm · 2023-03-27T12:11:02Z

I'll open a new issue for this, because the original issue is quite different.

github-actions · 2023-03-27T12:12:58Z

Closes: [Java] Split flight-sql-jdbc-driver to allow reuse of some internals #34742

github-actions · 2023-03-27T12:13:01Z

⚠️ GitHub issue #34742 has been automatically assigned in GitHub to PR creator.

ursabot · 2023-03-27T15:33:11Z

Benchmark runs are scheduled for baseline = 0a21a4c and contender = eaa1a1e. eaa1a1e is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.33% ⬆️0.0%] test-mac-arm
[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Failed ⬇️0.0% ⬆️0.0%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] eaa1a1ea ec2-t3-xlarge-us-east-2
[Failed] eaa1a1ea test-mac-arm
[Finished] eaa1a1ea ursa-i9-9960x
[Failed] eaa1a1ea ursa-thinkcentre-m75q
[Finished] 0a21a4c2 ec2-t3-xlarge-us-east-2
[Failed] 0a21a4c2 test-mac-arm
[Finished] 0a21a4c2 ursa-i9-9960x
[Finished] 0a21a4c2 ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

…se (apache#34678) ### Rationale for this change `flight-sql-jdbc-driver` module contains a set of utilities to implement JDBC support backed by arrow vectors. However, it employs maven shade plugin which is hard to make it a dependecy of external projects. ### What changes are included in this PR? This patch splits `flight-sql-jdbc-driver` module into two separate modules: `flight-sql-jdbc-core` and `flight-sql-jdbc-driver`. `flight-sql-jdbc-core` contains almost every thing in the original module except the shade plugin. `flight-sql-jdbc-driver` only shades `flight-sql-jdbc-core`. ### Are these changes tested? Make sure all tests pass. ### Are there any user-facing changes? No. * Closes: #20730 * Closes: apache#34742 Authored-by: Gang Wu <ustcwg@gmail.com> Signed-off-by: David Li <li.davidm96@gmail.com>

github-actions · 2024-11-26T19:23:23Z

⚠️ GitHub issue #34742 has been automatically assigned in GitHub to PR creator.

wgtmac requested a review from lidavidm as a code owner March 22, 2023 06:51

github-actions bot added Component: Java awaiting review Awaiting review labels Mar 22, 2023

wgtmac force-pushed the jdbc branch from 0cef01f to 3f422c7 Compare March 22, 2023 07:27

GH-20730: [Java] Split flight-sql-jdbc-driver to facilitate reuse

71f5795

flight-sql-jdbc-driver module contains a set of utilities to implement JDBC support backed by arrow vectors. However, it employs maven shade plugin which is hard to use. This patch splits it into two separate modules.

wgtmac force-pushed the jdbc branch from 3f422c7 to 71f5795 Compare March 22, 2023 07:47

fix dev/tasks/tasks.yml

d382b46

wgtmac requested review from assignUser, kou and raulcd as code owners March 23, 2023 01:47

remove flight-sql-jdbc-driver javadoc jar from dev/tasks/tasks.yml

7067c95

assignUser reviewed Mar 24, 2023

View reviewed changes

github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Mar 24, 2023

sunchao approved these changes Mar 24, 2023

View reviewed changes

github-actions bot added awaiting merge Awaiting merge and removed awaiting committer review Awaiting committer review labels Mar 24, 2023

lidavidm approved these changes Mar 24, 2023

View reviewed changes

wgtmac requested review from lidavidm and removed request for kou and raulcd March 26, 2023 14:20

wgtmac force-pushed the jdbc branch from cbee8c8 to d7b6ba6 Compare March 27, 2023 01:22

modify description to reflect driver and core module

a4f1207

wgtmac force-pushed the jdbc branch from d7b6ba6 to a4f1207 Compare March 27, 2023 05:28

lidavidm changed the title ~~GH-20730: [Java] Split flight-sql-jdbc-driver to facilitate reuse~~ GH-34742: [Java] Split flight-sql-jdbc-driver to facilitate reuse Mar 27, 2023

lidavidm merged commit eaa1a1e into apache:main Mar 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-34742: [Java] Split flight-sql-jdbc-driver to facilitate reuse #34678

GH-34742: [Java] Split flight-sql-jdbc-driver to facilitate reuse #34678

wgtmac commented Mar 22, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Mar 22, 2023

github-actions bot commented Mar 22, 2023

wgtmac commented Mar 22, 2023

lidavidm commented Mar 22, 2023

wgtmac commented Mar 23, 2023

assignUser commented Mar 23, 2023 •

edited

Loading

github-actions bot commented Mar 23, 2023

wgtmac commented Mar 23, 2023 •

edited

Loading

github-actions bot commented Mar 23, 2023

wgtmac commented Mar 23, 2023

kou commented Mar 24, 2023

wgtmac commented Mar 24, 2023

assignUser left a comment

sunchao left a comment

lidavidm left a comment

lidavidm Mar 24, 2023

wgtmac Mar 26, 2023

wgtmac commented Mar 27, 2023

lidavidm commented Mar 27, 2023

wgtmac commented Mar 27, 2023

lidavidm commented Mar 27, 2023

lidavidm commented Mar 27, 2023

github-actions bot commented Mar 27, 2023

github-actions bot commented Mar 27, 2023

ursabot commented Mar 27, 2023

github-actions bot commented Nov 26, 2024

GH-34742: [Java] Split flight-sql-jdbc-driver to facilitate reuse #34678

GH-34742: [Java] Split flight-sql-jdbc-driver to facilitate reuse #34678

Conversation

wgtmac commented Mar 22, 2023 • edited by github-actions bot Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

github-actions bot commented Mar 22, 2023

github-actions bot commented Mar 22, 2023

wgtmac commented Mar 22, 2023

lidavidm commented Mar 22, 2023

wgtmac commented Mar 23, 2023

assignUser commented Mar 23, 2023 • edited Loading

github-actions bot commented Mar 23, 2023

wgtmac commented Mar 23, 2023 • edited Loading

github-actions bot commented Mar 23, 2023

wgtmac commented Mar 23, 2023

kou commented Mar 24, 2023

wgtmac commented Mar 24, 2023

assignUser left a comment

Choose a reason for hiding this comment

sunchao left a comment

Choose a reason for hiding this comment

lidavidm left a comment

Choose a reason for hiding this comment

lidavidm Mar 24, 2023

Choose a reason for hiding this comment

wgtmac Mar 26, 2023

Choose a reason for hiding this comment

wgtmac commented Mar 27, 2023

lidavidm commented Mar 27, 2023

wgtmac commented Mar 27, 2023

lidavidm commented Mar 27, 2023

lidavidm commented Mar 27, 2023

github-actions bot commented Mar 27, 2023

github-actions bot commented Mar 27, 2023

ursabot commented Mar 27, 2023

github-actions bot commented Nov 26, 2024

wgtmac commented Mar 22, 2023 •

edited by github-actions bot

Loading

assignUser commented Mar 23, 2023 •

edited

Loading

wgtmac commented Mar 23, 2023 •

edited

Loading