Improve libpython search

[devinrsmith]

It seems that jpyutil is unable to properly find libpython3.8.so.1.0 on Ubuntu 20.04 (see deephaven/deephaven-core#2641).

https://packages.ubuntu.com/focal/amd64/libpython3.8/filelist:

/usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.so
/usr/lib/x86_64-linux-gnu/libpython3.8.so.1
/usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0

Maybe there is a more canonical way to reference the currently running libpython?

[fabrice-ducos]

What about using python3-config?

[devinrsmith]

python3-config looks like a solution for the build stage - for users at runtime though, where python may be installed elsewhere, I think ultimately the logic in jpyutil._find_python_dll_file needs to be improved to account for other canonical places (if python3-config happens to be installed, maybe it could be useful as a fallback?).

[fabrice-ducos]

Hi Devin,

if you look at python3-config's source code, it seems to be a simple Python script of 74 lines (with only dependencies to standard Python modules). Maybe the logic of that script could be adapted and embedded into jpyutil._find_python_dll_file.

Another solution: that jpy ships with its own copy of python3-config

Do I miss something there?

[devinrsmith]

Nope - that sounds great. Do you have a link to the source repository? My quick googling didn't find it.

[devinrsmith]

Or is this something that cpython maintains?

[fabrice-ducos]

I just "cat'ed" (or type'd if you are on a Windows platform) the copy of python3-config installed on my platform.

If you prefer to get the code from the "source", you will find it in the cpython repository:
python-config.in

You will notice that the extension is .in, not .py
It's actually Python code, but the shebang line (#!) seems to be replaced at deployment time by the actual path to the Python interpreter.

[devinrsmith]

Here's a little gist, https://gist.github.com/devinrsmith/1d7ba0131c6bbe74ade1b7d293ae282e

I'm not able to replicate the issue @mofojed - we might need to look more closely at your setup?

[mofojed]

My python setup was out of date - I had a python command and python3 command, pointing to different installations:

➜  main git:(js-plugin-jetty) ✗ which python
/usr/bin/python
➜  main git:(js-plugin-jetty) ✗ python --version
Python 3.8.10
➜  main git:(js-plugin-jetty) ✗ which python3
/usr/local/bin/python3
➜  main git:(js-plugin-jetty) ✗ python3 --version
Python 3.8.2

Using the /usr/bin/python version 3.8.10 to create my venv resolved this issue. I cannot recall how these python versions got installed initially.

[devinrsmith]

If this happens again,

JPY_LOG_LEVEL=DEBUG /path/to/venv/bin/python -c 'import jpyutil; print(jpyutil._find_python_dll_file())'

would be useful debug information.

[devinrsmith]

https://cbs.centos.org/koji/rpminfo?rpmID=390332

I think virtual environments from "rh-python38 Software Collection" are incorrect; potentially, we need to include additional search paths, as in this case, it looks like the proper libpython is at

/opt/rh/rh-python38/root/usr/lib64/libpython3.8.so.rh-python38-1.0.

https://docs.python.org/3/library/sysconfig.html may be useful in these regards. In particular, potentially sysconfig.get_config_var("LDLIBRARY") / sysconfig.get_config_var("INSTSONAME"). Also, can see useful info using the module: python -m sysconfig.

Note: rh-python38 Software Collection is EOL, but just used as an example of how we may want to improve the search.

[devinrsmith]

In the case of the rh-python38, we have INSTSONAME = "libpython3.8.so.rh-python38-1.0", which if used as an additional search name, would have found the correct libpython.