ci(main): bundle tests fail to build charms with zipfile EOFError

[orfeas-k]

Bug Description

As of yesterday, bundle tests v2 fail to build the charms with the error shown in logs below. This has been observed in multiple runs (example) with the same exact logs. Note that:

• each time the CI runs, it fails on a different charm
• building the charms (tried kfp-ui) locally with charmcraft pack works
• running the tests locally to see the output there too

To Reproduce

Run the CI from main.

Environment

Juju 3.5
Microk8s v1.25.16
charmcraft 2.6.0

Relevant Log Output

logs from running August 2024

::    ::   Successfully built setuptools wheel
::    ::   Installing collected packages: setuptools, wheel
::    ::   ERROR: Exception:
::    ::   Traceback (most recent call last):
::    ::     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
::    ::       status = self.run(options, args)
::    ::     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 421, in run
::    ::       installed = install_given_reqs(
::    ::     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/req/__init__.py", line 67, in install_given_reqs
::    ::       requirement.install(
::    ::     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 809, in install
::    ::       install_wheel(
::    ::     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/operations/install/wheel.py", line 606, in install_wheel
::    ::       unpack_file(wheel_path, unpacked_dir.path)
::    ::     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/utils/unpacking.py", line 249, in unpack_file
::    ::       unzip_file(
::    ::     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/utils/unpacking.py", line 139, in unzip_file
::    ::       shutil.copyfileobj(fp, destfp)
::    ::     File "/usr/lib/python3.8/shutil.py", line 205, in copyfileobj
::    ::       buf = fsrc_read(length)
::    ::     File "/usr/lib/python3.8/zipfile.py", line 940, in read
::    ::       data = self._read1(n)
::    ::     File "/usr/lib/python3.8/zipfile.py", line 1008, in _read1
::    ::       data += self._read2(n - len(data))
::    ::     File "/usr/lib/python3.8/zipfile.py", line 1043, in _read2
::    ::       raise EOFError
::    ::   EOFError
::    ::   ----------------------------------------
::    :: ERROR: Command errored out with exit status 2: /root/parts/charm/build/staging-venv/bin/python3 /root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-_x_j7juu/overlay --no-warn-script-location --no-binary anyio,attrs,certifi,charmed-kubeflow-chisme,charset-normalizer,deepdiff,exceptiongroup,h11,httpcore,httpx,idna,importlib-resources,jsonschema,lightkube,lightkube-models,markupsafe,ops,ordered-set,pip,pkgutil-resolve-name,pyrsistent,pyyaml,requests,ruamel-yaml,ruamel-yaml-clib,serialized-data-interface,setuptools,sniffio,tenacity,urllib3,websocket-client,wheel,zipp --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel Check the logs for full command output.
:: WARNING: Initial package installation failed. Falling back to older method, which may leave your charm in an un-runnable state.
:: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--upgrade', 'jinja2']
::    :: Collecting jinja2
::    ::   Downloading jinja2-3.1.4-py3-none-any.whl (133 kB)
::    :: Collecting MarkupSafe>=2.0
::    ::   Downloading MarkupSafe-2.1.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (26 kB)
::    :: Installing collected packages: MarkupSafe, jinja2
::    :: Successfully installed MarkupSafe-2.1.5 jinja2-3.1.4
:: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--upgrade', '--no-binary', ':all:', 'wheel', 'setuptools', 'pip']
::    :: Collecting wheel
::    ::   Using cached wheel-0.43.0.tar.gz (99 kB)
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: finished with status 'done'
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::     Preparing wheel metadata: started
::    ::     Preparing wheel metadata: finished with status 'done'
::    :: Collecting setuptools
::    ::   Using cached setuptools-69.5.1.tar.gz (2.3 MB)
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::   Installing backend dependencies: started
::    ::   Installing backend dependencies: finished with status 'done'
::    ::     Preparing wheel metadata: started
::    ::     Preparing wheel metadata: finished with status 'done'
::    :: Collecting pip
::    ::   Downloading pip-24.0.tar.gz (2.1 MB)
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: finished with status 'done'
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::     Preparing wheel metadata: started
::    ::     Preparing wheel metadata: finished with status 'done'
::    :: Building wheels for collected packages: wheel, setuptools, pip
::    ::   Building wheel for wheel (PEP 517): started
::    ::   Building wheel for wheel (PEP 517): finished with status 'done'
::    ::   Created wheel for wheel: filename=wheel-0.43.0-py3-none-any.whl size=65775 sha256=55c570405f142630c6b9f72fe09d9b67cf1477fcf543ae5b8dcb1f5b7377da81
::    ::   Stored in directory: /root/.cache/pip/wheels/d4/a3/b2/6faade3bb04cd720bb432a6c21ec7baa2542801334a4d47a66
::    ::   Building wheel for setuptools (PEP 517): started
::    ::   Building wheel for setuptools (PEP 517): finished with status 'done'
::    ::   Created wheel for setuptools: filename=setuptools-69.5.1-py3-none-any.whl size=894566 sha256=075e940cfa3659352ca81f87fabf8b4b75ccdbb5f99f2a71a4f77df1b6707fbc
::    ::   Stored in directory: /root/.cache/pip/wheels/57/b6/41/311c955c1ca3eed447666adf5ca9680ddcdd11dc785a746bf6
::    ::   Building wheel for pip (PEP 517): started
::    ::   Building wheel for pip (PEP 517): finished with status 'done'
::    ::   Created wheel for pip: filename=pip-24.0-py3-none-any.whl size=2110223 sha256=7176c5cddbc465dc78dddac6545a4635901a7c9a08a959f4f3bbb3ca0bee944a
::    ::   Stored in directory: /root/.cache/pip/wheels/c1/d6/d4/2f7765d82b17d8c25425875b2dce5af9f22746668c3add50f6
::    :: Successfully built wheel setuptools pip
::    :: Installing collected packages: wheel, setuptools, pip
::    ::   Attempting uninstall: setuptools
::    ::     Found existing installation: setuptools 44.0.0
::    ::     Uninstalling setuptools-44.0.0:
::    ::       Successfully uninstalled setuptools-44.0.0
::    ::   Attempting uninstall: pip
::    ::     Found existing installation: pip 20.0.2
::    ::     Uninstalling pip-20.0.2:
::    ::       Successfully uninstalled pip-20.0.2
::    :: Successfully installed pip-24.0 setuptools-69.5.1 wheel-0.43.0
:: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--upgrade', '--no-binary', ':all:', '--requirement=requirements.txt']
::    :: Collecting anyio==3.7.1 (from -r requirements.txt (line 7))
::    ::   Downloading anyio-3.7.1.tar.gz (142 kB)
::    ::      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 142.9/142.9 kB 3.9 MB/s eta 0:00:00
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: finished with status 'done'
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::   Installing backend dependencies: started
::    ::   Installing backend dependencies: finished with status 'done'
::    ::   Preparing metadata (pyproject.toml): started
::    ::   Preparing metadata (pyproject.toml): finished with status 'done'
::    :: Collecting attrs==23.1.0 (from -r requirements.txt (line 9))
::    ::   Downloading attrs-23.1.0.tar.gz (212 kB)
::    ::      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 212.9/212.9 kB 29.5 MB/s eta 0:00:00
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: finished with status 'error'
::    ::   error: subprocess-exited-with-error
::    ::
::    ::   × pip subprocess to install build dependencies did not run successfully.
::    ::   │ exit code: 1
::    ::   ╰─> [103 lines of output]
::    ::       Collecting hatchling
::    ::         Downloading hatchling-1.24.2.tar.gz (63 kB)
::    ::            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.0/64.0 kB 2.0 MB/s eta 0:00:00
::    ::         Getting requirements to build wheel: started
::    ::         Getting requirements to build wheel: finished with status 'done'
::    ::         Installing backend dependencies: started
::    ::         Installing backend dependencies: finished with status 'error'
::    ::         error: subprocess-exited-with-error
::    ::
::    ::         × pip subprocess to install backend dependencies did not run successfully.
::    ::         │ exit code: 1
::    ::         ╰─> [81 lines of output]
::    ::             Collecting packaging>=23.2
::    ::               Using cached packaging-24.0-py3-none-any.whl
::    ::             Collecting pathspec>=0.10.1
::    ::               Downloading pathspec-0.12.1.tar.gz (51 kB)
::    ::                  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 51.0/51.0 kB 895.3 kB/s eta 0:00:00
::    ::               Installing build dependencies: started
::    ::               Installing build dependencies: finished with status 'done'
::    ::               Getting requirements to build wheel: started
::    ::               Getting requirements to build wheel: finished with status 'done'
::    ::               Preparing metadata (pyproject.toml): started
::    ::               Preparing metadata (pyproject.toml): finished with status 'done'
::    ::             Collecting trove-classifiers
::    ::               Downloading trove-classifiers-2024.4.10.tar.gz (16 kB)
::    ::               Installing build dependencies: started
::    ::               Installing build dependencies: finished with status 'done'
::    ::               Getting requirements to build wheel: started
::    ::               Getting requirements to build wheel: finished with status 'done'
::    ::               Installing backend dependencies: started
::    ::               Installing backend dependencies: finished with status 'done'
::    ::               Preparing metadata (pyproject.toml): started
::    ::               Preparing metadata (pyproject.toml): finished with status 'done'
::    ::             Collecting tomli>=1.2.2
::    ::               Using cached tomli-2.0.1-py3-none-any.whl
::    ::             Collecting pluggy>=1.0.0
::    ::               Downloading pluggy-1.5.0.tar.gz (67 kB)
::    ::                  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 68.0/68.0 kB 3.5 MB/s eta 0:00:00
::    ::               Installing build dependencies: started
::    ::               Installing build dependencies: finished with status 'error'
::    ::               error: subprocess-exited-with-error
::    ::
::    ::               × pip subprocess to install build dependencies did not run successfully.
::    ::               │ exit code: 2
::    ::               ╰─> [38 lines of output]
::    ::                   Collecting setuptools>=45.0
::    ::                     Using cached setuptools-69.5.1-py3-none-any.whl
::    ::                   Collecting setuptools-scm>=6.2.3 (from setuptools-scm[toml]>=6.2.3)
::    ::                     Using cached setuptools_scm-8.1.0-py3-none-any.whl
::    ::                   Collecting packaging>=20 (from setuptools-scm>=6.2.3->setuptools-scm[toml]>=6.2.3)
::    ::                     Using cached packaging-24.0-py3-none-any.whl
::    ::                   Collecting typing-extensions (from setuptools-scm>=6.2.3->setuptools-scm[toml]>=6.2.3)
::    ::                     Using cached typing_extensions-4.11.0-py3-none-any.whl
::    ::                   Collecting tomli>=1 (from setuptools-scm>=6.2.3->setuptools-scm[toml]>=6.2.3)
::    ::                     Using cached tomli-2.0.1-py3-none-any.whl
::    ::                   Installing collected packages: typing-extensions, tomli, setuptools, packaging, setuptools-scm
::    ::                   ERROR: Exception:
::    ::                   Traceback (most recent call last):
::    ::                     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
::    ::                       status = run_func(*args)
::    ::                     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 245, in wrapper
::    ::                       return func(self, options, args)
::    ::                     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 452, in run
::    ::                       installed = install_given_reqs(
::    ::                     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/req/__init__.py", line 72, in install_given_reqs
::    ::                       requirement.install(
::    ::                     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 856, in install
::    ::                       install_wheel(
::    ::                     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/operations/install/wheel.py", line 725, in install_wheel
::    ::                       _install_wheel(
::    ::                     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/operations/install/wheel.py", line 585, in _install_wheel
::    ::                       file.save()
::    ::                     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/operations/install/wheel.py", line 384, in save
::    ::                       shutil.copyfileobj(f, dest)
::    ::                     File "/usr/lib/python3.8/shutil.py", line 205, in copyfileobj
::    ::                       buf = fsrc_read(length)
::    ::                     File "/usr/lib/python3.8/zipfile.py", line 940, in read
::    ::                       data = self._read1(n)
::    ::                     File "/usr/lib/python3.8/zipfile.py", line 1008, in _read1
::    ::                       data += self._read2(n - len(data))
::    ::                     File "/usr/lib/python3.8/zipfile.py", line 1043, in _read2
::    ::                       raise EOFError
::    ::                   EOFError
::    ::                   [end of output]
::    ::
::    ::               note: This error originates from a subprocess, and is likely not a problem with pip.
::    ::             error: subprocess-exited-with-error
::    ::
::    ::             × pip subprocess to install build dependencies did not run successfully.
::    ::             │ exit code: 2
::    ::             ╰─> See above for output.
::    ::
::    ::             note: This error originates from a subprocess, and is likely not a problem with pip.
::    ::             [end of output]
::    ::
::    ::         note: This error originates from a subprocess, and is likely not a problem with pip.
::    ::       error: subprocess-exited-with-error
::    ::
::    ::       × pip subprocess to install backend dependencies did not run successfully.
::    ::       │ exit code: 1
::    ::       ╰─> See above for output.
::    ::
::    ::       note: This error originates from a subprocess, and is likely not a problem with pip.
::    ::       [end of output]
::    ::
::    ::   note: This error originates from a subprocess, and is likely not a problem with pip.
::    :: error: subprocess-exited-with-error
::    ::
::    :: × pip subprocess to install build dependencies did not run successfully.
::    :: │ exit code: 1
::    :: ╰─> See above for output.
::    ::
::    :: note: This error originates from a subprocess, and is likely not a problem with pip.
:: Traceback (most recent call last):
::   File "/snap/charmcraft/3454/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 260, in _install_dependencies
::     _process_run(
::   File "/snap/charmcraft/3454/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 408, in _process_run
::     raise RuntimeError(f"Subprocess command {cmd} execution failed with retcode {retcode}")
:: RuntimeError: Subprocess command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=anyio,attrs,certifi,charmed-kubeflow-chisme,charset-normalizer,deepdiff,exceptiongroup,h11,httpcore,httpx,idna,importlib-resources,jsonschema,lightkube,lightkube-models,markupsafe,ops,ordered-set,pip,pkgutil-resolve-name,pyrsistent,pyyaml,requests,ruamel-yaml,ruamel-yaml-clib,serialized-data-interface,setuptools,sniffio,tenacity,urllib3,websocket-client,wheel,zipp', '--requirement=requirements.txt', 'pip', 'setuptools', 'wheel'] execution failed with retcode 1
::
:: During handling of the above exception, another exception occurred:
::
:: Traceback (most recent call last):
::   File "/snap/charmcraft/3454/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 477, in <module>
::     main()
::   File "/snap/charmcraft/3454/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 472, in main
::     builder.build_charm()
::   File "/snap/charmcraft/3454/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 101, in build_charm
::     self.handle_dependencies()
::   File "/snap/charmcraft/3454/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 354, in handle_dependencies
::     self._install_dependencies(staging_venv_dir)
::   File "/snap/charmcraft/3454/lib/python3.10/site-packages/charmcraft/instrum.py", line 152, in _f
::     return func(*args, **kwargs)
::   File "/snap/charmcraft/3454/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 289, in _install_dependencies
::     _process_run(cmd)
::   File "/snap/charmcraft/3454/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 408, in _process_run
::     raise RuntimeError(f"Subprocess command {cmd} execution failed with retcode {retcode}")
:: RuntimeError: Subprocess command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--upgrade', '--no-binary', ':all:', '--requirement=requirements.txt'] execution failed with retcode 1
Parts processing error: Failed to run the build script for part 'charm'.
Failed to build charm for bases index '0'.
Full execution log: '/home/runner/.local/state/charmcraft/log/charmcraft-20240509-152917.976217.log'

Additional Context

No response

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5656.

This message was autogenerated

[orfeas-k]

Although this failed for 4 consecutive times, reran this now and it succeeded https://github.com/canonical/kfp-operators/actions/runs/9019490085/job/24814815779?pr=456.

[orfeas-k]

This has also happened in v1 tests actually

[orfeas-k]

We see this in different charms as well e.g. istio-pilot

Full logs

Added 'test-istio' model on microk8s/localhost with credential 'microk8s' for user 'admin'
integration-tls-provider: install_deps> python -I -m pip install -r requirements-integration.txt
integration-tls-provider: freeze> python -m pip freeze --all
integration-tls-provider: aiohttp==3.[8](https://github.com/canonical/istio-operators/actions/runs/9384901845/job/25841748232?pr=434#step:4:9).5,aiosignal==1.3.1,anyio==4.0.0,asttokens==2.4.0,async-timeout==4.0.3,asyncio==3.4.3,attrs==23.1.0,backcall==0.2.0,bcrypt==4.0.1,beautifulsoup4==4.12.2,cachetools==5.3.1,certifi==2023.7.22,cffi==1.15.1,charset-normalizer==3.2.0,cryptography==41.0.3,decorator==5.1.1,exceptiongroup==1.1.3,executing==1.2.0,frozenlist==1.4.0,google-auth==2.22.0,h11==0.14.0,httpcore==0.17.3,httpx==0.24.1,hvac==1.2.0,idna==3.4,iniconfig==2.0.0,ipdb==0.13.13,ipython==8.12.2,jedi==0.1[9](https://github.com/canonical/istio-operators/actions/runs/9384901845/job/25841748232?pr=434#step:4:10).0,Jinja2==3.1.2,juju==3.4.0.0,kubernetes==27.2.0,lightkube==0.14.0,lightkube-models==1.28.1.4,macaroonbakery==1.3.1,MarkupSafe==2.1.3,matplotlib-inline==0.1.6,multidict==6.0.4,mypy-extensions==1.0.0,oauthlib==3.2.2,packaging==23.1,paramiko==2.12.0,parso==0.8.3,pexpect==4.8.0,pickleshare==0.7.5,pip==24.0,pluggy==1.3.0,prompt-toolkit==3.0.39,protobuf==3.20.3,ptyprocess==0.7.0,pure-eval==0.2.2,pyasn1==0.5.0,pyasn1-modules==0.3.0,pycparser==2.21,Pygments==2.16.1,pyhcl==0.4.5,pymacaroons==0.13.0,PyNaCl==1.5.0,pyRFC3339==1.1,pytest==7.4.2,pytest-asyncio==0.21.1,pytest-operator==0.29.0,python-dateutil==2.8.2,pytz==2023.3.post1,PyYAML==6.0.1,requests==2.31.0,requests-oauthlib==1.3.1,rsa==4.9,setuptools==69.5.1,six==1.16.0,sniffio==1.3.0,soupsieve==2.5,stack-data==0.6.2,tenacity==8.2.3,tomli==2.0.1,toposort==1.[10](https://github.com/canonical/istio-operators/actions/runs/9384901845/job/25841748232?pr=434#step:4:11),traitlets==5.9.0,typing-inspect==0.9.0,typing_extensions==4.7.1,urllib3==1.26.16,wcwidth==0.2.6,websocket-client==1.6.2,websockets==8.1,wheel==0.43.0,yarl==1.9.2
integration-tls-provider: commands[0]> pytest --show-capture=no --log-cli-level=INFO -vvs --tb=native --model test-istio tests/test_bundle_tls_provider.py
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.4.2, pluggy-1.3.0 -- /home/runner/work/istio-operators/istio-operators/.tox/integration-tls-provider/bin/python
cachedir: .tox/integration-tls-provider/.pytest_cache
rootdir: /home/runner/work/istio-operators/istio-operators
configfile: pyproject.toml
plugins: anyio-4.0.0, asyncio-0.21.1, operator-0.29.0
asyncio: mode=strict
collecting ... collected 2 items

tests/test_bundle_tls_provider.py::test_build_and_deploy_istio_charms 
-------------------------------- live log setup --------------------------------
INFO     pytest_operator.plugin:plugin.py:675 Connecting to existing model github-pr-8e71e-microk8s:test-istio on unspecified cloud
-------------------------------- live log call ---------------------------------
INFO     pytest_operator.plugin:plugin.py:526 Using tmp_path: /home/runner/work/istio-operators/istio-operators/.tox/integration-tls-provider/tmp/pytest/test-istio0
INFO     pytest_operator.plugin:plugin.py:975 Building charm istio-gateway
INFO     pytest_operator.plugin:plugin.py:526 Using tmp_path: /home/runner/work/istio-operators/istio-operators/.tox/integration-tls-provider/tmp/pytest/test-istio0
INFO     pytest_operator.plugin:plugin.py:975 Building charm istio-pilot
INFO     pytest_operator.plugin:plugin.py:982 Charm build for istio-gateway completed with errors (return code=1) in 156.92s
FAILED
tests/test_bundle_tls_provider.py::test_tls_configuration XFAIL
------------------------------ live log teardown -------------------------------
INFO     pytest_operator.plugin:plugin.py:783 Model status:

Model       Controller                Cloud/Region        Version  SLA          Timestamp
test-istio  github-pr-8e71e-microk8s  microk8s/localhost  3.5.1    unsupported  13:28:51Z


INFO     pytest_operator.plugin:plugin.py:789 Juju error logs:


INFO     pytest_operator.plugin:plugin.py:855 Forgetting main...


=================================== FAILURES ===================================
______________________ test_build_and_deploy_istio_charms ______________________
Traceback (most recent call last):
  File "/home/runner/work/istio-operators/istio-operators/.tox/integration-tls-provider/lib/python3.8/site-packages/_pytest/runner.py", line 341, in from_call
    result: Optional[TResult] = func()
  File "/home/runner/work/istio-operators/istio-operators/.tox/integration-tls-provider/lib/python3.8/site-packages/_pytest/runner.py", line 262, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/home/runner/work/istio-operators/istio-operators/.tox/integration-tls-provider/lib/python3.8/site-packages/pluggy/_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "/home/runner/work/istio-operators/istio-operators/.tox/integration-tls-provider/lib/python3.8/site-packages/pluggy/_manager.py", line [11](https://github.com/canonical/istio-operators/actions/runs/9384901845/job/25841748232?pr=434#step:4:12)5, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/istio-operators/istio-operators/.tox/integration-tls-provider/lib/python3.8/site-packages/pluggy/_callers.py", line 152, in _multicall
    return outcome.get_result()
  File "/home/runner/work/istio-operators/istio-operators/.tox/integration-tls-provider/lib/python3.8/site-packages/pluggy/_result.py", line 114, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/runner/work/istio-operators/istio-operators/.tox/integration-tls-provider/lib/python3.8/site-packages/pluggy/_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/istio-operators/istio-operators/.tox/integration-tls-provider/lib/python3.8/site-packages/_pytest/runner.py", line 177, in pytest_runtest_call
    raise e
  File "/home/runner/work/istio-operators/istio-operators/.tox/integration-tls-provider/lib/python3.8/site-packages/_pytest/runner.py", line 169, in pytest_runtest_call
    item.runtest()
  File "/home/runner/work/istio-operators/istio-operators/.tox/integration-tls-provider/lib/python3.8/site-packages/_pytest/python.py", line 1792, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/home/runner/work/istio-operators/istio-operators/.tox/integration-tls-provider/lib/python3.8/site-packages/pluggy/_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "/home/runner/work/istio-operators/istio-operators/.tox/integration-tls-provider/lib/python3.8/site-packages/pluggy/_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/istio-operators/istio-operators/.tox/integration-tls-provider/lib/python3.8/site-packages/pluggy/_callers.py", line 152, in _multicall
    return outcome.get_result()
  File "/home/runner/work/istio-operators/istio-operators/.tox/integration-tls-provider/lib/python3.8/site-packages/pluggy/_result.py", line 114, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/runner/work/istio-operators/istio-operators/.tox/integration-tls-provider/lib/python3.8/site-packages/pluggy/_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/istio-operators/istio-operators/.tox/integration-tls-provider/lib/python3.8/site-packages/_pytest/python.py", line 194, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/home/runner/work/istio-operators/istio-operators/.tox/integration-tls-provider/lib/python3.8/site-packages/pytest_asyncio/plugin.py", line 532, in inner
    _loop.run_until_complete(task)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/runner/work/istio-operators/istio-operators/tests/test_bundle_tls_provider.py", line 33, in test_build_and_deploy_istio_charms
    istio_charms = await ops_test.build_charms(f"{charms_path}-gateway", f"{charms_path}-pilot")
  File "/home/runner/work/istio-operators/istio-operators/.tox/integration-tls-provider/lib/python3.8/site-packages/pytest_operator/plugin.py", line 1021, in build_charms
    charms = await asyncio.gather(
  File "/home/runner/work/istio-operators/istio-operators/.tox/integration-tls-provider/lib/python3.8/site-packages/pytest_operator/plugin.py", line 1004, in build_charm
    raise RuntimeError(
RuntimeError: Failed to build charm charms/istio-gateway:
Packing the charm.
Launching environment to pack for base name='ubuntu' channel='20.04' architectures=['amd64'] (may take a while the first time but it's reusable)
Packing the charm
Packing the charm.
Building charm in '/root'
Running step PULL for part 'charm'
Execute action
Running step BUILD for part 'charm'
Execute action
:: + env -i LANG=C.UTF-8 LC_ALL=C.UTF-8 PATH=/snap/charmcraft/3454/libexec/charmcraft:/snap/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin SNAP=/snap/charmcraft/3454 SNAP_ARCH=amd64 SNAP_NAME=charmcraft SNAP_VERSION=2.6.0 /snap/charmcraft/3454/bin/python3 -u -I /snap/charmcraft/3454/lib/python3.10/site-packages/charmcraft/charm_builder.py --builddir /root/parts/charm/build --installdir /root/parts/charm/install --entrypoint /root/parts/charm/build/src/charm.py -p wheel -p setuptools -p pip -r requirements.txt
:: Starting charm builder
:: Translated .jujuignore 2 '.git' => '.*/\\.git\\Z'
:: Translated .jujuignore 3 '.svn' => '.*/\\.svn\\Z'
:: Translated .jujuignore 4 '.hg' => '.*/\\.hg\\Z'
:: Translated .jujuignore 5 '.bzr' => '.*/\\.bzr\\Z'
:: Translated .jujuignore 6 '.tox' => '.*/\\.tox\\Z'
:: Translated .jujuignore 8 '/build/' => '/build\\Z'
:: Translated .jujuignore 9 '/revision' => '/revision\\Z'
:: Translated .jujuignore 10 '/venv' => '/venv\\Z'
:: Translated .jujuignore [12](https://github.com/canonical/istio-operators/actions/runs/9384901845/job/25841748232?pr=434#step:4:13) '.jujuignore' => '.*/\\.jujuignore\\Z'
:: Translated .jujuignore 1 '/venv\n' => '/venv\\Z'
:: Translated .jujuignore 2 '*.py[cod]\n' => '.*/[^/]*\\.py[cod]\\Z'
:: Translated .jujuignore 3 '*.charm\n' => '.*/[^/]*\\.charm\\Z'
:: Translated .jujuignore 4 '.tox\n' => '.*/\\.tox\\Z'
:: Translated .jujuignore 5 '__pycache__\n' => '.*/__pycache__\\Z'
:: Translated .jujuignore 1 '/staging-venv' => '/staging\\-venv\\Z'
:: Collected charmlib dependencies: set()
:: Building charm in '/root/parts/charm/install'
:: Linking in generic paths
:: Ignoring file because of rules: '.jujuignore'
:: Creating the dispatch mechanism
:: Creating the 'upgrade-charm' hook script pointing to dispatch
:: Creating the 'install' hook script pointing to dispatch
:: Creating the 'start' hook script pointing to dispatch
:: Handling dependencies
:: Current dependencies hash: 'd2b50d61430c50e878b34ed171619f2580b229af'
:: Dependencies directory not found
:: Installing dependencies
:: Running external command ['python3', '-m', 'venv', '/root/parts/charm/build/staging-venv']
:: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', 'pip@https://files.pythonhosted.org/packages/ba/19/e63fb4e0d20e48bd2167bb7e857abc0e21679e24805ba921a224df8977c0/pip-23.2.1.tar.gz']
::    :: Requirement already satisfied: pip@ https://files.pythonhosted.org/packages/ba/19/e63fb4e0d20e48bd2167bb7e857abc0e21679e24805ba921a224df8977c0/pip-23.2.1.tar.gz from https://files.pythonhosted.org/packages/ba/19/e63fb4e0d20e48bd2167bb7e857abc0e21679e24805ba921a224df8977c0/pip-23.2.1.tar.gz in ./staging-venv/lib/python3.8/site-packages (20.0.2)
:: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=anyio,attrs,certifi,charset-normalizer,exceptiongroup,h11,httpcore,httpx,idna,importlib-resources,jinja2,jsonschema,lightkube,lightkube-models,markupsafe,oci-image,ops,pip,pkgutil-resolve-name,pyrsistent,pyyaml,requests,serialized-data-interface,setuptools,sniffio,urllib3,websocket-client,wheel,zipp', '--requirement=requirements.txt', 'pip', 'setuptools', 'wheel']
::    :: Requirement already satisfied: pip in ./staging-venv/lib/python3.8/site-packages (20.0.2)
::    :: Requirement already satisfied: setuptools in ./staging-venv/lib/python3.8/site-packages (44.0.0)
::    :: Collecting wheel
::    ::   Downloading wheel-0.43.0.tar.gz (99 kB)
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: finished with status 'done'
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::     Preparing wheel metadata: started
::    ::     Preparing wheel metadata: finished with status 'done'
::    :: Collecting anyio==4.0.0
::    ::   Downloading anyio-4.0.0.tar.gz (153 kB)
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: finished with status 'done'
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::   Installing backend dependencies: started
::    ::   Installing backend dependencies: finished with status 'done'
::    ::     Preparing wheel metadata: started
::    ::     Preparing wheel metadata: finished with status 'done'
::    :: Collecting attrs==23.1.0
::    ::   Downloading attrs-23.1.0.tar.gz (212 kB)
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: finished with status 'done'
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::     Preparing wheel metadata: started
::    ::     Preparing wheel metadata: finished with status 'done'
::    :: Collecting certifi==2023.7.22
::    ::   Downloading certifi-2023.7.22.tar.gz (159 kB)
::    :: Collecting charset-normalizer==3.2.0
::    ::   Downloading charset-normalizer-3.2.0.tar.gz (97 kB)
::    :: Collecting exceptiongroup==1.1.3
::    ::   Downloading exceptiongroup-1.1.3.tar.gz (23 kB)
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: finished with status 'done'
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::     Preparing wheel metadata: started
::    ::     Preparing wheel metadata: finished with status 'done'
::    :: Collecting h11==0.14.0
::    ::   Downloading h11-0.14.0.tar.gz (100 kB)
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: finished with status 'done'
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::     Preparing wheel metadata: started
::    ::     Preparing wheel metadata: finished with status 'done'
::    :: Collecting httpcore==0.17.3
::    ::   Downloading httpcore-0.17.3.tar.gz (62 kB)
::    :: Collecting httpx==0.24.1
::    ::   Downloading httpx-0.24.1.tar.gz (81 kB)
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: finished with status 'done'
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::     Preparing wheel metadata: started
::    ::     Preparing wheel metadata: finished with status 'done'
::    :: Collecting idna==3.4
::    ::   Downloading idna-3.4.tar.gz (183 kB)
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: finished with status 'done'
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::     Preparing wheel metadata: started
::    ::     Preparing wheel metadata: finished with status 'done'
::    :: Collecting importlib-resources==6.0.1
::    ::   Downloading importlib_resources-6.0.1.tar.gz (38 kB)
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: finished with status 'done'
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::   Installing backend dependencies: started
::    ::   Installing backend dependencies: finished with status 'done'
::    ::     Preparing wheel metadata: started
::    ::     Preparing wheel metadata: finished with status 'done'
::    :: Collecting jinja2==3.1.2
::    ::   Downloading Jinja2-3.1.2.tar.gz (268 kB)
::    ::     ERROR: Command errored out with exit status 1:
::    ::      command: /root/parts/charm/build/staging-venv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-j4xc92x7/jinja2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-j4xc92x7/jinja2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-j4xc92x7/jinja2/pip-egg-info
::    ::          cwd: /tmp/pip-install-j4xc92x7/jinja2/
::    ::     Complete output (35 lines):
::    ::     Traceback (most recent call last):
::    ::       File "<string>", line 1, in <module>
::    ::       File "/tmp/pip-install-j4xc92x7/jinja2/setup.py", line 4, in <module>
::    ::         setup(
::    ::       File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/setuptools/__init__.py", line 145, in setup
::    ::         return distutils.core.setup(**attrs)
::    ::       File "/usr/lib/python3.8/distutils/core.py", line 121, in setup
::    ::         dist.parse_config_files()
::    ::       File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/setuptools/dist.py", line 701, in parse_config_files
::    ::         parse_configuration(self, self.command_options,
::    ::       File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/setuptools/config.py", line 121, in parse_configuration
::    ::         meta.parse()
::    ::       File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/setuptools/config.py", line 426, in parse
::    ::         section_parser_method(section_options)
::    ::       File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/setuptools/config.py", line 399, in parse_section
::    ::         self[name] = value
::    ::       File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/setuptools/config.py", line 184, in __setitem__
::    ::         value = parser(value)
::    ::       File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/setuptools/config.py", line 515, in _parse_version
::    ::         version = self._parse_attr(value, self.package_dir)
::    ::       File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/setuptools/config.py", line 349, in _parse_attr
::    ::         module = import_module(module_name)
::    ::       File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
::    ::         return _bootstrap._gcd_import(name[level:], package, level)
::    ::       File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
::    ::       File "<frozen importlib._bootstrap>", line 991, in _find_and_load
::    ::       File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
::    ::       File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
::    ::       File "<frozen importlib._bootstrap_external>", line 848, in exec_module
::    ::       File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
::    ::       File "/tmp/pip-install-j4xc92x7/jinja2/src/jinja2/__init__.py", line 8, in <module>
::    ::         from .environment import Environment as Environment
::    ::       File "/tmp/pip-install-j4xc92x7/jinja2/src/jinja2/environment.py", line 14, in <module>
::    ::         from markupsafe import Markup
::    ::     ModuleNotFoundError: No module named 'markupsafe'
::    ::     ----------------------------------------
::    :: ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
:: WARNING: Initial package installation failed. Falling back to older method, which may leave your charm in an un-runnable state.
:: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--upgrade', '--no-binary', ':all:', 'wheel', 'setuptools', 'pip']
::    :: Collecting wheel
::    ::   Using cached wheel-0.43.0.tar.gz (99 kB)
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: finished with status 'done'
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::     Preparing wheel metadata: started
::    ::     Preparing wheel metadata: finished with status 'done'
::    :: Collecting setuptools
::    ::   Using cached setuptools-70.0.0.tar.gz (2.3 MB)
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::   Installing backend dependencies: started
::    ::   Installing backend dependencies: finished with status 'done'
::    ::     Preparing wheel metadata: started
::    ::     Preparing wheel metadata: finished with status 'done'
::    :: Collecting pip
::    ::   Downloading pip-24.0.tar.gz (2.1 MB)
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: finished with status 'done'
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::     Preparing wheel metadata: started
::    ::     Preparing wheel metadata: finished with status 'done'
::    :: Building wheels for collected packages: wheel, setuptools, pip
::    ::   Building wheel for wheel (PEP 517): started
::    ::   Building wheel for wheel (PEP 517): finished with status 'done'
::    ::   Created wheel for wheel: filename=wheel-0.43.0-py3-none-any.whl size=65775 sha256=55c570405f142630c6b9f72fe09d9b67cf1477fcf543ae5b8dcb1f5b7377da81
::    ::   Stored in directory: /root/.cache/pip/wheels/d4/a3/b2/6faade3bb04cd720bb432a6c21ec7baa254280[13](https://github.com/canonical/istio-operators/actions/runs/9384901845/job/25841748232?pr=434#step:4:14)34a4d47a66
::    ::   Building wheel for setuptools (PEP 517): started
::    ::   Building wheel for setuptools (PEP 517): finished with status 'done'
::    ::   Created wheel for setuptools: filename=setuptools-70.0.0-py3-none-any.whl size=863432 sha256=dc1c3e41e77a13dcc52aaf9051edcf91da0b06fceb9007f89d4a84e2e4294d7e
::    ::   Stored in directory: /root/.cache/pip/wheels/20/5f/b8/845eeb9e697798706931bd3ebc34b40c3bd3e3d6167407bb77
::    ::   Building wheel for pip (PEP 517): started
::    ::   Building wheel for pip (PEP 517): finished with status 'done'
::    ::   Created wheel for pip: filename=pip-24.0-py3-none-any.whl size=2110223 sha256=741c1090d56683aca93c5c585d26dab507548a7e6afa75e59f9e76d1920a2359
::    ::   Stored in directory: /root/.cache/pip/wheels/c1/d6/d4/2f7765d82b17d8c25425875b2dce5af9f22746668c3add50f6
::    :: Successfully built wheel setuptools pip
::    :: Installing collected packages: wheel, setuptools, pip
::    ::   Attempting uninstall: setuptools
::    ::     Found existing installation: setuptools 44.0.0
::    ::     Uninstalling setuptools-44.0.0:
::    ::       Successfully uninstalled setuptools-44.0.0
::    ::   Rolling back uninstall of setuptools
::    ::   Moving to /root/parts/charm/build/staging-venv/bin/easy_install
::    ::    from /tmp/pip-uninstall-b6c35q6n/easy_install
::    ::   Moving to /root/parts/charm/build/staging-venv/bin/easy_install-3.8
::    ::    from /tmp/pip-uninstall-b6c35q6n/easy_install-3.8
::    ::   Moving to /root/parts/charm/build/staging-venv/lib/python3.8/site-packages/__pycache__/
::    ::    from /root/parts/charm/build/staging-venv/lib/python3.8/site-packages/~_pycache__
::    ::   Moving to /root/parts/charm/build/staging-venv/lib/python3.8/site-packages/easy_install.py
::    ::    from /tmp/pip-uninstall-_cmer5js/easy_install.py
::    ::   Moving to /root/parts/charm/build/staging-venv/lib/python3.8/site-packages/setuptools-44.0.0.dist-info/
::    ::    from /root/parts/charm/build/staging-venv/lib/python3.8/site-packages/~etuptools-44.0.0.dist-info
::    ::   Moving to /root/parts/charm/build/staging-venv/lib/python3.8/site-packages/setuptools/
::    ::    from /root/parts/charm/build/staging-venv/lib/python3.8/site-packages/~etuptools
::    :: ERROR: Exception:
::    :: Traceback (most recent call last):
::    ::   File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
::    ::     status = self.run(options, args)
::    ::   File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 421, in run
::    ::     installed = install_given_reqs(
::    ::   File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/req/__init__.py", line 67, in install_given_reqs
::    ::     requirement.install(
::    ::   File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 809, in install
::    ::     install_wheel(
::    ::   File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/operations/install/wheel.py", line 606, in install_wheel
::    ::     unpack_file(wheel_path, unpacked_dir.path)
::    ::   File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/utils/unpacking.py", line 249, in unpack_file
::    ::     unzip_file(
::    ::   File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/utils/unpacking.py", line 139, in unzip_file
::    ::     shutil.copyfileobj(fp, destfp)
::    ::   File "/usr/lib/python3.8/shutil.py", line 205, in copyfileobj
::    ::     buf = fsrc_read(length)
::    ::   File "/usr/lib/python3.8/zipfile.py", line 940, in read
::    ::     data = self._read1(n)
::    ::   File "/usr/lib/python3.8/zipfile.py", line 1008, in _read1
::    ::     data += self._read2(n - len(data))
::    ::   File "/usr/lib/python3.8/zipfile.py", line 1043, in _read2
::    ::     raise EOFError
::    :: EOFError
:: Traceback (most recent call last):
::   File "/snap/charmcraft/3454/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 260, in _install_dependencies
::     _process_run(
::   File "/snap/charmcraft/3454/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 408, in _process_run
::     raise RuntimeError(f"Subprocess command {cmd} execution failed with retcode {retcode}")
:: RuntimeError: Subprocess command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=anyio,attrs,certifi,charset-normalizer,exceptiongroup,h11,httpcore,httpx,idna,importlib-resources,jinja2,jsonschema,lightkube,lightkube-models,markupsafe,oci-image,ops,pip,pkgutil-resolve-name,pyrsistent,pyyaml,requests,serialized-data-interface,setuptools,sniffio,urllib3,websocket-client,wheel,zipp', '--requirement=requirements.txt', 'pip', 'setuptools', 'wheel'] execution failed with retcode 1
::
:: During handling of the above exception, another exception occurred:
::
:: Traceback (most recent call last):
::   File "/snap/charmcraft/3454/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 477, in <module>
::     main()
::   File "/snap/charmcraft/3454/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 472, in main
::     builder.build_charm()
::   File "/snap/charmcraft/3454/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 101, in build_charm
::     self.handle_dependencies()
::   File "/snap/charmcraft/3454/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 354, in handle_dependencies
::     self._install_dependencies(staging_venv_dir)
::   File "/snap/charmcraft/3454/lib/python3.10/site-packages/charmcraft/instrum.py", line 152, in _f
::     return func(*args, **kwargs)
::   File "/snap/charmcraft/3454/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 283, in _install_dependencies
::     _process_run(cmd)
::   File "/snap/charmcraft/3454/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 408, in _process_run
::     raise RuntimeError(f"Subprocess command {cmd} execution failed with retcode {retcode}")
:: RuntimeError: Subprocess command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--upgrade', '--no-binary', ':all:', 'wheel', 'setuptools', 'pip'] execution failed with retcode 2
Parts processing error: Failed to run the build script for part 'charm'.
Failed to build charm for bases index '0'.
Full execution log: '/home/runner/.local/state/charmcraft/log/charmcraft-20240605-132615.576[14](https://github.com/canonical/istio-operators/actions/runs/9384901845/job/25841748232?pr=434#step:4:15)2.log'


=========================== short test summary info ============================
FAILED tests/test_bundle_tls_provider.py::test_build_and_deploy_istio_charms
=================== 1 failed, 1 xfailed in [15](https://github.com/canonical/istio-operators/actions/runs/9384901845/job/25841748232?pr=434#step:4:16)7.93s (0:02:37) ===================
integration-tls-provider: exit 1 (158.71 seconds) /home/runner/work/istio-operators/istio-operators> pytest --show-capture=no --log-cli-level=INFO -vvs --tb=native --model test-istio tests/test_bundle_tls_provider.py pid=[16](https://github.com/canonical/istio-operators/actions/runs/9384901845/job/25841748232?pr=434#step:4:17)652
  integration-tls-provider: FAIL code 1 ([17](https://github.com/canonical/istio-operators/actions/runs/9384901845/job/25841748232?pr=434#step:4:18)4.96=setup[16.25]+cmd[158.71] seconds)
  evaluation failed :( (175.03 seconds)

[DnPlas]

Moving this issue to canonical/bundle-kubeflow as it is happening for more than one charm.

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-6081.

This message was autogenerated

[orfeas-k]

Debugging

• pip install --no-binary=:all: --requirement=requirements.txt for istio-pilot in a fresh LXC container with python 3.8.10, pip 24.1 is completing without errors, after installing charm bin dependencies with apt-get install rustc cargo libffi-dev libssl-dev pkg-config

• Opened some PRs to play around with setuptools installation in charm-python-packages, in case there was an incompatibility between the versions installed for the charm. The rationale there was that:

  1. building fails when installing those packages (including setuptools)
  2. earlier, when installing the setuptools charm-python-package, it uses version 44.0.0
  3. later, it tries to install setuptools=>45.0

  However, even when it installed a newer version of setuptools, that didn't change much

• Building the charm in a fresh LXC container using charmcraft from latest/edge is successful. Same with charmcraft from latest/candidate

• In the CI though, switching charmcraft from latest/edge (runs) to latest/candidate (runs) seems to fix the issue and the charm is being built. Thus, I sent this PR fix(ci): use charmcraft from latest/candidate in integration tests istio-operators#512 to use charmcraft from latest/candidate. Let the charmcraft team in Matrix know about this and asked for pointers

• That being said, integration test case test_enable_ingress_auth fails https://github.com/canonical/istio-operators/actions/runs/10388478915/job/28764213343?pr=512#step:8:21 (both on Microk8s 1.25 and 1.26)
  There are some indicators that istio-pilot could be misbehaving (although having been built successfully)

   istio-pilot/0* maintenance executing 10.1.138.74 (stop) stopping charm software
   ...
   istio-pilot/0* unknown lost 10.1.138.74 agent lost, see 'juju show-status-log istio-pilot/0'
  

  Downloading the logs artifact the only "error" there is

   `unit-istio-pilot-0: 14:24:02 INFO unit.istio-pilot/0.juju-log ingress:6: Removing gateway due to errors in processing the ingress-auth relation.`
  

[orfeas-k]

Workaround

The issue was worked around in #531 by using charmcraft from latest/candidate in integration tests and backporting changes that free disk space in runner #519 and updating cert_handler library to v1 #517.

Cause

Doing a little more debugging, it looks like the issue could be caused because we are building two charms at the same time, since:

• draft: only charmcraft pack charms istio-operators#533 runs charmcraft pack for both charms sequentially (and skip tests) and both builds succeed (but take around 1h to complete)
• tests: build charms sequentially istio-operators#534 modified the tests to build the charms sequentially as well and this also succeeded (but took 1h as well).

Solution

In the future, we could hit this again when charmcraft from latest/edge is promoted. If this happens, we should allocate time to work on a larger effort to refactor our CI as described e.g. in #766 or canonical/istio-operators#412, which would probably eliminate this issue as well.