Improve Python and R package testing

[bryanpaget]

The package testing we use does not work well. Code is here: https://github.com/StatCan/aaw-kubeflow-containers/blob/master/tests/general/test_packages.py

Example issues with the current test suite:

1. The Python package torch was not working for quite a while but test_packages.py was unable to catch this. Unable to import torch on pytorch image. #514
2. Recently we upgraded from Python 3.9 to Python 3.11 and SASPy was not usable but the test suite was unable to catch this. SASPy Config Needs Updating #533
3. Additionally, I've struggled multiple times with false positives and have added multiple entries into the exclude list, for instance, the package pyyaml is imported with import yaml and pillow is imported with import pil.

The exclusion list is growing:

# List of packages that cannot be tested in a standard way
EXCLUDED_PACKAGES = [
    # Binaries
    "tini",
    "python",
    "hdf5",
    "bzip2",
    'nodejs', 
    # Jupyterlab extensions
    'python-lsp-server',
    'jupyterlab-translate',
    'jupyter-dash',
    "jupyter-pluto-proxy",
    "jupyterlab-git",
    "jupyter-server-proxy",
    "jupyterlab-language-pack-fr-fr",
    "jupyterlab-lsp",
    "jupyter-lsp",
    # Other
    "conda-forge::blas[build=openblas]",
    "protobuf",
    "r-irkernel",
    "unixodbc",
    "bzip2",
    "openssl",
    "ca-certificates",
    "r-tidymodels",
    "gputil",
    "cudatoolkit",
    "cudnn",
    # Python
    # import PIL, not import pillow
    "pillow",
    # import yaml, not import pyyaml
    "pyyaml",
]

[bryanpaget]

We recently had an issue with JupyerLab-Git extension not working with Python 3.11. So in addition to having package import tests, we might also want JupyerLab extension tests.

Reference to issue (h/t @Souheil-Yazji):

• Authentication Issue - AttributeError: module 'asyncio' has no attribute 'coroutine' jupyterlab/jupyterlab-git#1247

[bryanpaget]

We may also want to have some example notebooks, which are run during testing, to ensure our "core offerings" just work.

Some example SASPy Notebooks can be found here:

• https://github.com/sassoftware/saspy-examples/blob/main/SAS_contrib/SASPy%20for%20Machine%20Learning.ipynb

Or maybe just take some of the core commands from some example notebooks, and add them to our test suite, but then make sure they fail when they should fail and pass when they should pass.