Further CI adjustments for Java interoperability branch #169

nberth · 2024-08-16T07:12:54Z

No description provided.

nberth · 2024-08-16T07:21:58Z

@GitMensch At the moment the tests with a call to Java fail:

1264. run_java.at:23: testing CALL Java static void (void) ...
../../tests/run_java.at:43: $JAVAC Test.java
../../tests/run_java.at:44: $COMPILE prog.cob
../../tests/run_java.at:45: $COBCRUN_DIRECT ./prog
--- /dev/null	2024-08-16 07:15:45.000000000 +0000
+++ /d/a/gnucobol/gnucobol/_build/tests/testsuite.dir/at-groups/1264/stderr	2024-08-16 07:15:45.467925900 +0000
@@ -0,0 +1 @@
+libcob: prog.cob:5: error: Java interoperability module cannot be loaded: module libcobjni-1.dll not found
--- -	2024-08-16 07:15:45.576799400 +0000
+++ /d/a/gnucobol/gnucobol/_build/tests/testsuite.dir/at-groups/1264/stdout	2024-08-16 07:15:45.530426900 +0000
@@ -1,2 +1 @@
-Hello world!
 
1264. run_java.at:23: 1264. CALL Java static void (void) (run_java.at:23): FAILED (run_java.at:45)

#                             -*- compilation -*-
1265. run_java.at:51: testing CALL Java static void (void) (missing) ...
../../tests/run_java.at:64: $COMPILE prog.cob
../../tests/run_java.at:66: $COBCRUN_DIRECT ./prog
--- /dev/null	2024-08-16 07:15:45.000000000 +0000
+++ /d/a/gnucobol/gnucobol/_build/tests/testsuite.dir/at-groups/1265/stderr	2024-08-16 07:15:45.123619000 +0000
@@ -0,0 +1 @@
+libcob: prog.cob:5: error: Java interoperability module cannot be loaded: module libcobjni-1.dll not found
1265. run_java.at:51: 1265. CALL Java static void (void) (missing) (run_java.at:51): FAILED (run_java.at:66)

In install.log we have:

…
/opt/cobol/gnucobol/bin
/opt/cobol/gnucobol/bin/cob-config
/opt/cobol/gnucobol/bin/cobc.exe
/opt/cobol/gnucobol/bin/cobcrun.exe
/opt/cobol/gnucobol/bin/cobfile.exe
/opt/cobol/gnucobol/bin/gcdiff.exe
/opt/cobol/gnucobol/bin/libcob-5.dll
/opt/cobol/gnucobol/bin/libcobdb-1.dll
/opt/cobol/gnucobol/bin/libcobjni-1.dll
…

so that lib seems to be built. I'm not sure what part is missing for the dynamic lookup to operate properly.

.github/workflows/windows-java.yml

GitMensch · 2024-08-16T10:36:29Z

I'm not sure what part is missing for the dynamic lookup to operate properly.

It would be good if someone merges 1231387 and by doing so applying these changes also to cob_load_lib() (ideally in svn upstream, of course) - this should provide us with more details for this (and other) failure(s).

tests/atlocal.in

GitMensch

I'm opposed to the bashrc adjustment as we won't want to do that when installing GnuCOBOL either.
jvm.dll is always in $JAVA_HOME/bin/server, isn't it?

Wouldn't it be reasonable then to check if JAVA_HOME is set, and if yes use it concatenated with "bin/server", passing it to lt_dladdsearchdir()?
If we then cannot delay-load libcob-java,m we can also hint the user to set JAVA_HOME.

Maybe there's something similar that would be useful outside of WIN32 as well).

GitMensch · 2024-08-18T06:36:20Z

.github/workflows/windows-java.yml

+      - name: install
+        run: |
+          make -C _build install
+          find /opt/cobol > _build/install.log
+        shell: msys2 {0}
+
+      - name: Upload install.log
+        uses: actions/upload-artifact@v4
+        with:
+          name: install-${{ matrix.os }}.log
+          path: _build/install.log


There should be real good reason to have the tests only run after install (if I remember correctly we did not found another way on MacOS only) - please move install+install log at the end.

This workflow is adapted from another I found in the gc4 branch. I guess that one needs to be fixed as well.

all development is iterative - that's obviously true for CI definitions as well :-)
Thank you for fixing the gc3+gc4 with anything you find here (at least some action versioning and "no install before test" so far)

GitMensch · 2024-08-18T06:44:51Z

.github/workflows/windows-java.yml

+        shell: msys2 {0}
+        run: |
+          # Be sure to have `jvm.dll` in path for tests with JNI below
+          jvmdir="$(dirname $(find "$JAVA_HOME" -name jvm.dll))"


Why does dirname produce a non unix path (and if it doesn't - why executing cygpath then)?

.github/workflows/windows-java.yml

nberth · 2024-08-19T07:28:15Z

Thanks @GitMensch for your review.

jvm.dll is always in $JAVA_HOME/bin/server, isn't it?

Is it a question or an assertion? I don't know the answer, as I said I'm completely unfamiliar with how things are done on Windows.

Wouldn't it be reasonable then to check if JAVA_HOME is set, and if yes use it concatenated with "bin/server", passing it to lt_dladdsearchdir()? If we then cannot delay-load libcob-java,m we can also hint the user to set JAVA_HOME.

If $JAVA_HOME/bin/server is always where we can find jvm.dll, then yes I guess we can rely more on that variable if it is set.

GitMensch · 2024-08-19T08:16:29Z

Thanks @GitMensch for your review.

You're welcome. Thanks for taking care in the first place!

jvm.dll is always in $JAVA_HOME/bin/server, isn't it?

Is it a question or an assertion? I don't know the answer, as I said I'm completely unfamiliar with how things are done on Windows.

That seems more JVM related than Windows - it is what I've found in multiple Windows installs using JRE since 8 (aka 1.8.0) up top current ones with both x64 and x86 architectures... While I just have seen 1 very old installation that has it under $JAVA_HOME/bin/client... checking further led to https://www.oracle.com/java/technologies/ms-windows-install-64bit.html#jvm.dll, so these two seem to be the official paths.

While this can be all made up, the following answer of ChatGPT4 seems quite reasonable and, from a glance, matches what https://www.oracle.com/java/technologies/whitepaper.html#overview documents:

The jvm.dll files located in JAVA_HOME/bin/server and JAVA_HOME/bin/client represent different implementations of the Java Virtual Machine (JVM) provided by the Java Development Kit (JDK). Here are the key differences:

Client JVM (JAVA_HOME/bin/client/jvm.dll)

Optimization for Client-Side Applications: The client JVM is optimized for quick startup and responsive performance suitable for client-side applications, such as desktop applications.

Memory Usage: It typically uses less memory than the server JVM, making it more appropriate for environments where resources are limited.

Compilation Strategy: The client JVM uses a simpler, less aggressive compilation strategy. It focuses on reducing the time it takes to start up the application rather than on long-term performance optimization.

Garbage Collection: The client JVM uses a less aggressive garbage collection strategy, aiming to minimize the perceived pauses for applications.

Server JVM (JAVA_HOME/bin/server/jvm.dll)

Optimization for Server-Side Applications: The server JVM is designed for long-running server applications, such as web servers or application servers, where performance and throughput are more critical than startup time.

Memory Usage: It may use more memory as it aims to optimize for long-term performance rather than conserving resources.

Compilation Strategy: The server JVM employs a more aggressive compilation strategy, including advanced optimizations such as Just-In-Time (JIT) compilation. This results in better performance over time as the JVM can make more sophisticated optimizations based on the application's behavior.

Garbage Collection: The server JVM uses more sophisticated and aggressive garbage collection algorithms, which are designed to improve throughput and reduce long-term memory fragmentation.

Usage

Client JVM: Best for desktop applications where fast startup and responsiveness are important.

Server JVM: Ideal for server applications where long-term performance and throughput are critical.

How to Choose

The choice between the client and server JVM typically depends on the nature of the application you are running:

For short-lived, interactive applications, you might prefer the client JVM.

For long-running, resource-intensive applications, the server JVM would be the better choice.

Selection at Runtime

When you run a Java application, you can specify which JVM to use. If no specific JVM is mentioned, the JDK typically defaults to one based on the operating system and architecture. You can explicitly select the server or client JVM by specifying the -client or -server options when starting your Java application:
java -client -jar MyApp.jar
or
java -server -jar MyApp.jar
These options ensure that the appropriate jvm.dll is used, corresponding to the chosen JVM implementation.

which brings us to...

Wouldn't it be reasonable then to check if JAVA_HOME is set, and if yes use it concatenated with "bin/server", passing it to lt_dladdsearchdir()? If we then cannot delay-load libcob-java,m we can also hint the user to set JAVA_HOME.

If $JAVA_HOME/bin/server is always where we can find jvm.dll, then yes I guess we can rely more on that variable if it is set.

I'd say for now it would be reasonable to append both the server and the client directory (in this order) to the search path, as soon as JAVA_HOME is set. Please adjust it that way (doing that in the code for WIN32 and dropping it from atlocal).

For Linux this is the libvjm.so. Checking some Linux distros shows that (on older OS?) there's a subfolder in the system package manager installs containing the architecture but binary jdk downloads (and newer OS?) have those "directly" under $JAVA_HOME/lib/server/libjvm.so (or $JAVA_HOME/lib/client/libjvm.so).

Maybe that's overkill, but what do you think of the following:

if a made-up variable is set - use that (for example JVM_LIB_PATH [pointing to a "server" or "client" directory, we don't care) and prefix the search path by it
otherwise, if JAVA_HOME is set, append it to the search paths, both server and client in that order (for Windows using "bin" otherwise using "lib")

GitMensch · 2024-08-19T08:24:30Z

... second thought about "JVM_LIB_PATH": as people can self-prefix the system specific search dir we likely can just drop that extra variable and just append to the search path according to JAVA_HOME

nberth · 2024-08-19T09:18:59Z

Agreed ;-)

nberth · 2024-08-19T14:29:43Z

@GitMensch While investigating this I'm wondering whether one should always assume libltdl is used (as it provides lt_dladdsearchdir); other variants for similar systems do not appear do offer any search path facility. On WIN32 there a AddDllDirectory that seems available, but that [may affect DLL search order] (https://learn.microsoft.com/en-us/windows/win32/api/Winbase/nf-winbase-setdlldirectorya).

nberth · 2024-08-19T14:41:16Z

BTW configuring with --without-dl leads to a failing test:

#                             -*- compilation -*-
737. run_misc.at:4198: testing COB_PRE_LOAD with entry points ...
/home/nico/work/repos/gnucobol/tests/run_misc.at:4257: $COMPILE_MODULE prog.cob
/home/nico/work/repos/gnucobol/tests/run_misc.at:4258: $COMPILE_MODULE prog1.cob
/home/nico/work/repos/gnucobol/tests/run_misc.at:4259: $COMPILE main-prog.cob
/home/nico/work/repos/gnucobol/tests/run_misc.at:4260: COB_PRE_LOAD="prog"$PATHSEP"prog1" $COBCRUN_DIRECT ./main-prog
--- -	2024-08-19 16:36:39.341893156 +0200
+++ /tmp/b/tests/testsuite.dir/at-groups/737/stdout	2024-08-19 16:36:39.334951092 +0200
@@ -1,4 +1,4 @@
 12abc
-11
+55
 xxxxx
 
737. run_misc.at:4198: 737. COB_PRE_LOAD with entry points (run_misc.at:4198): FAILED (run_misc.at:4260)

Looking more and more like a rabbit hole.

GitMensch · 2024-08-19T15:30:32Z

Hm... that change gets bigger than expected...

Suggestion:

for USE_LIBDL:
- just ad the server + client directory via lt_dladdsearchdir()
otherwise pre-load if available:
- define a static lt_dlhandle jvm_handle = NULL;
- check if there exists a jvm.dll / libjvm.so in the relevant directories via access(path), on the first found do jvm_handle = lt_dlopen(path)
- on teardown: if (jvm_handle) { lt_dlclose (jvm_handle); jvm_handle = NULL; }

GitMensch

the solution looks nice to me (ChangeLogs entries missing including the separate one for the reasonable move to the systfines.h - if they are still needed in fileio.c, likely they can be kept in call.c only?)

Concerning the failing MacOS CI - is JAVA_HOME set there?

GitMensch · 2024-08-20T20:02:10Z

libcob/call.c

-#define lt_dlopenlcl(x)	lt_dlopen(x)
-#define lt_dlclose(x)	FreeLibrary(x)
+#define lt_dlopenlcl(x)		lt_dlopen(x)
+#define lt_dlclose(x)		FreeLibrary(x)
 #define	lt_dlinit()
 #define	lt_dlexit()
-#define lt_dlhandle	HMODULE
+#define lt_dlhandle		HMODULE
+#define lt_dladdsearchdir(x)	AddDllDirectory(x)


that's a WIN32 definition and we don't use that for this case - you can therefore revert those lines

nberth · 2024-08-21T09:24:13Z

https://github.com/OCamlPro/gnucobol/actions/runs/10486974200/job/29046364090?pr=169#step:8:230

I'll disable mingw32 as I'm unsable to find what package to install to get that on i686 arch. Unless @ddeclerck @GitMensch have an idea about it?

GitMensch · 2024-08-21T09:35:47Z

Better drop BDB, cjson and xml2 from all java CI definitions and explicit request --without-indexed --without-json --without-xml2 - those are (to be) tested in the "common" CI ones.

I personally would not use pacboy for the matrix but the "good old" standard one:

  strategy:
    matrix:
      include:
        - { sys: mingw64, env: x86_64 }
        - { sys: mingw32, env: i686 }
        - { sys: ucrt64,  env: ucrt-x86_64 }
        - { sys: clang64, env: clang-x86_64 }
  steps:
    - uses: msys2/setup-msys2@v2
      with:
        msystem: ${{matrix.sys}}
        install: mingw-w64-${{matrix.env}}-openssl

You can then also use the default dependencies from msys2 https://github.com/msys2/MINGW-packages/blob/67aee9a6de4f153729397c6a5b09fd8bd39786ca/mingw-w64-gnucobol/PKGBUILD#L22-L28 - but as noted, those dependencies should get into the "common" CI for msys2, along with adding that build matrix to #170.

GitMensch · 2024-08-21T11:46:42Z

As the failing Ubuntu one is not related to java-interop you may still merge that.
But please keep the distcheck (only on Ubuntu) as this validates we can distribute the package.

You want to use:

make -C _build distcheck DISTCHECK_CONFIGURE_FLAGS="--without-indexed --without-json --without-xml2 --with-java" TESTSUITEFLAGS="--jobs=$((${NPROC}+1))"

nberth · 2024-08-21T12:05:15Z

As the failing Ubuntu one is not related to java-interop you may still merge that. But please keep the distcheck (only on Ubuntu) as this validates we can distribute the package.

You want to use:

make -C _build distcheck DISTCHECK_CONFIGURE_FLAGS="--without-indexed --without-json --without-xml2 --with-java" TESTSUITEFLAGS="--jobs=$((${NPROC}+1))"

Many thanks for this suggestion! I suspected there would be one variable to pass configure args, but I though since it's only for Java interop. we could discard the distribution. I'll do the change ASAP.

GitMensch

@nberth here's the detailed review you've asked for :-)

Someday we'll have to make a decision if we want to keep the java tests separated or not; if we know that this doesn't work on some environment, then using a conditional there depending on the build matrix seems useful;
for the binary nightly packages (MSYS2, MSVC, possibly Ubuntu) a "full featured" build would be useful...

.github/workflows/ubuntu-java.yml

.github/workflows/windows-java.yml

GitMensch · 2024-08-21T14:23:21Z

.github/workflows/macos-java.yml

@@ -49,9 +44,7 @@ jobs:
      - name: bootstrap
        run: |
          sed -i '' 's/-undefined suppress//g' configure.ac


What is this doing and why? It seems to originate from @ddeclerck.
We should at least add a comment there.

"Recent" versions of XCode (>= 15 ?) include a new linker, which does not support the -undefined flag. This substitution removes that option, which would otherwise cause the testsuite to fail on MacOS. Until a definitive solution is found...

I guess this would need to be checked by configure.

configure uses this only in a single place - "newer" MacOS and a compiler claiming to be GCC:

case $host_os in rhapsody*) if test "$COB_USES_GCC" = yes; then COB_SHARED_OPT="-bundle -flat_namespace -undefined suppress" fi ;;

... so I've learned something new: https://en.wikipedia.org/wiki/Rhapsody_(operating_system)

I've also checked the used files from https://git.savannah.gnu.org/cgit/config.git and will update some "soon" but that won't help for this specific issue.

For that we can add a separate case entry (what is $host_os in the failing case?) or try if we can link successfully w/wo those options.

The libtool.m4 seems to distinguish using darwin versions as well:

# serial 58 LT_INIT … case $host_os in rhapsody* | darwin1.[[012]]) _lt_dar_allow_undefined='$wl-undefined ${wl}suppress' ;; darwin1.*) _lt_dar_allow_undefined='$wl-flat_namespace $wl-undefined ${wl}suppress' ;; darwin*) # darwin 5.x on # if running on 10.5 or later, the deployment target defaults # to the OS version, if on x86, and 10.4, the deployment # target defaults to 10.4. Don't you love it? case ${MACOSX_DEPLOYMENT_TARGET-10.0},$host in 10.0,*86*-darwin8*|10.0,*-darwin[[912]]*) _lt_dar_allow_undefined='$wl-undefined ${wl}dynamic_lookup' ;; 10.[[012]][[,.]]*) _lt_dar_allow_undefined='$wl-flat_namespace $wl-undefined ${wl}suppress' ;; 10.*|11.*) _lt_dar_allow_undefined='$wl-undefined ${wl}dynamic_lookup' ;; esac ;; esac …

(not sure that's relevant though)

GitMensch reviewed Aug 16, 2024

View reviewed changes

.github/workflows/windows-java.yml Outdated Show resolved Hide resolved

nberth force-pushed the java-interop-ci4windows branch from 6bd657a to d554c75 Compare August 16, 2024 10:33

GitMensch reviewed Aug 16, 2024

View reviewed changes

tests/atlocal.in Outdated Show resolved Hide resolved

nberth force-pushed the java-interop-ci4windows branch 6 times, most recently from 17aea01 to eb61422 Compare August 16, 2024 14:28

GitMensch requested changes Aug 18, 2024

View reviewed changes

GitMensch reviewed Aug 18, 2024

View reviewed changes

.github/workflows/windows-java.yml Show resolved Hide resolved

nberth force-pushed the java-interop-ci4windows branch from eb61422 to 0cb9ab3 Compare August 19, 2024 07:22

nberth force-pushed the java-interop-ci4windows branch from 0cb9ab3 to 78fbcf2 Compare August 19, 2024 07:41

nberth force-pushed the java-interop-ci4windows branch 3 times, most recently from f3852b2 to eb021a0 Compare August 19, 2024 12:00

nberth mentioned this pull request Aug 20, 2024

CI adjustments for gcos4gnucobol-3.x #170

Merged

nberth force-pushed the java-interop-ci4windows branch from 09a109d to 78c2fe5 Compare August 20, 2024 20:02

GitMensch requested changes Aug 20, 2024

View reviewed changes

nberth force-pushed the java-interop-ci4windows branch 2 times, most recently from eb8b752 to 14a44f4 Compare August 20, 2024 20:40

nberth force-pushed the java-interop-ci4windows branch 12 times, most recently from 17c7e68 to 3e22b7b Compare August 21, 2024 09:11

Use a broader build matrix for Windows' CI

1d90af0

nberth force-pushed the java-interop-ci4windows branch from 3e22b7b to 8c314e5 Compare August 21, 2024 10:20

nberth changed the title ~~Add Windows CI for Java interoperability branch~~ More CI adjustments for Java interoperability branch Aug 21, 2024

nberth changed the title ~~More CI adjustments for Java interoperability branch~~ Further CI adjustments for Java interoperability branch Aug 21, 2024

nberth force-pushed the java-interop-ci4windows branch from 0528d97 to 4426107 Compare August 21, 2024 11:00

nberth force-pushed the java-interop-ci4windows branch 2 times, most recently from 1c0a315 to ec81415 Compare August 21, 2024 12:11

nberth added 2 commits August 21, 2024 14:16

Disable BDB, CJSON, and XML2 on CIs for Java interoperability

7fa0de4

Drop install.log in CI for Java interoperability

499275d

nberth force-pushed the java-interop-ci4windows branch from ec81415 to 499275d Compare August 21, 2024 12:23

nberth requested a review from GitMensch August 21, 2024 12:31

nberth merged commit 4edf4a4 into OCamlPro:java-interop Aug 21, 2024
8 checks passed

nberth deleted the java-interop-ci4windows branch August 21, 2024 13:57

GitMensch reviewed Aug 21, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Further CI adjustments for Java interoperability branch #169

Further CI adjustments for Java interoperability branch #169

nberth commented Aug 16, 2024 •

edited

Loading

nberth commented Aug 16, 2024

GitMensch commented Aug 16, 2024

GitMensch left a comment

GitMensch Aug 18, 2024

nberth Aug 19, 2024

GitMensch Aug 19, 2024

GitMensch Aug 18, 2024

nberth commented Aug 19, 2024

GitMensch commented Aug 19, 2024

Client JVM (`JAVA_HOME/bin/client/jvm.dll`)

Server JVM (`JAVA_HOME/bin/server/jvm.dll`)

Usage

How to Choose

Selection at Runtime

GitMensch commented Aug 19, 2024

nberth commented Aug 19, 2024

nberth commented Aug 19, 2024 •

edited

Loading

nberth commented Aug 19, 2024

GitMensch commented Aug 19, 2024

GitMensch left a comment

GitMensch Aug 20, 2024

nberth commented Aug 21, 2024

GitMensch commented Aug 21, 2024 •

edited

Loading

GitMensch commented Aug 21, 2024 •

edited

Loading

nberth commented Aug 21, 2024

GitMensch left a comment

GitMensch Aug 21, 2024

ddeclerck Sep 2, 2024

nberth Sep 2, 2024

GitMensch Sep 2, 2024

nberth Sep 2, 2024

Further CI adjustments for Java interoperability branch #169

Further CI adjustments for Java interoperability branch #169

Conversation

nberth commented Aug 16, 2024 • edited Loading

nberth commented Aug 16, 2024

GitMensch commented Aug 16, 2024

GitMensch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nberth commented Aug 19, 2024

GitMensch commented Aug 19, 2024

Client JVM (JAVA_HOME/bin/client/jvm.dll)

Server JVM (JAVA_HOME/bin/server/jvm.dll)

Usage

How to Choose

Selection at Runtime

GitMensch commented Aug 19, 2024

nberth commented Aug 19, 2024

nberth commented Aug 19, 2024 • edited Loading

nberth commented Aug 19, 2024

GitMensch commented Aug 19, 2024

GitMensch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nberth commented Aug 21, 2024

GitMensch commented Aug 21, 2024 • edited Loading

GitMensch commented Aug 21, 2024 • edited Loading

nberth commented Aug 21, 2024

GitMensch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nberth commented Aug 16, 2024 •

edited

Loading

Client JVM (`JAVA_HOME/bin/client/jvm.dll`)

Server JVM (`JAVA_HOME/bin/server/jvm.dll`)

nberth commented Aug 19, 2024 •

edited

Loading

GitMensch commented Aug 21, 2024 •

edited

Loading

GitMensch commented Aug 21, 2024 •

edited

Loading