Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

actions/upload-artifacts@v3 is being removed (5 Dec 2024), affects Linux build-wheels jobs #21616

Open
huonw opened this issue Nov 5, 2024 · 2 comments
Labels
bug category:internal CI, fixes for not-yet-released features, etc.
Milestone

Comments

@huonw
Copy link
Contributor

huonw commented Nov 5, 2024

Describe the bug

According to https://github.blog/changelog/2024-11-05-notice-of-breaking-changes-for-github-actions/#artifacts-v3-brownouts, the actions/upload-artifacts@v3 behaviour is being deprecated and removed. Timeline:

  • brown-out: November 14, 9am – 10am EST
  • brown-out: November 21, 9am – 5pm EST
  • removal: December 5

This affects the Linux build-wheels jobs which run in a container, and require node16 compatibility (only available in v3, not v4, of this action).

helper.upload_log_artifacts(name="wheels-and-pex", node16_compat=bool(container)),

We'll need to remove the usage of this from all active branches, or CI will fail.

Options:

  1. quick-hack: stop uploading the logs at all
  2. better: use the container only for the build, either by running the whole pants invocation in it, or using a docker environment within pants
  3. ???

Pants version
N/A

OS
N/A

Additional info
Related discussion happened in: #21133

@huonw huonw added bug category:internal CI, fixes for not-yet-released features, etc. labels Nov 5, 2024
@huonw huonw added this to the 2.22.x milestone Nov 5, 2024
@tdyas
Copy link
Contributor

tdyas commented Nov 6, 2024

Can we just upgrade to v4 of actions/upload-artifact?

@huonw
Copy link
Contributor Author

huonw commented Nov 6, 2024

I think that requires node20 which isn't compatible with the container used (see #21133 and #21172), although feel free to dig into the git history and/or experiment more!

cburroughs added a commit that referenced this issue Nov 16, 2024
While trying to land #21528 to was observed that the wheel building jobs
were failing during git checkout with wonderful errors like
`/__e/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found
(required by /__e/node20/bin/node)`. This appears to be the same
symptoms as actions/checkout#1809 which
pointed the the deprecation notice at
<https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/>.
If this all sounds familiar that is because we *mostly* cleaned this up
in #21133, but kept a fewer uses of the older actions for
`manylinux2014` compatibility. However, from the notice:

"To opt out of this and continue using Node16 while it is still
available in the runner, you can choose to set
ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true
as an ‘env’ in their workflow or as an environment variable on your
runner machine. This will *only work until we upgrade the runner
removing Node16 later in the spring*. (emphasis added)"

From the wheel job failures during #21528 and my attempts to fiddle with
all of `ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION`,
`ACTIONS_RUNNER_FORCED_INTERNAL_NODE_VERSION`,
`ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION` I think the removal time
promised for the Spring has finally come, but I'm not that familiar with
GitHub Actions and could be missing something.

As a consequence the version of both `actions/upload-artifacts` and
`actions/checkout` is bumped to `v4`. To make this testable now and in
the future, wheels are now build on ci config changes.

Why is bumping to `manylinux_2_28` (the next oldest manylinux) not a big
deal? The "2.28" refers to glibc
[2.28](https://sourceware.org/legacy-ml/libc-alpha/2018-08/msg00003.html)
in 2018. That is older than Debian stable and if you need to use
software that old you are presumably paying someone for an enterprise
distribution.

NOTE: I think we should bump the manylinux version in `main` regardless,
but if my read if the situation is correct we may need to backport this
any active release line. Without this or a more complex change we would
also be unable to release after December 5th per #21616.

ref #21195 #21616
benjyw pushed a commit to benjyw/pants that referenced this issue Nov 18, 2024
While trying to land pantsbuild#21528 to was observed that the wheel building jobs
were failing during git checkout with wonderful errors like
`/__e/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found
(required by /__e/node20/bin/node)`. This appears to be the same
symptoms as actions/checkout#1809 which
pointed the the deprecation notice at
<https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/>.
If this all sounds familiar that is because we *mostly* cleaned this up
in pantsbuild#21133, but kept a fewer uses of the older actions for
`manylinux2014` compatibility. However, from the notice:

"To opt out of this and continue using Node16 while it is still
available in the runner, you can choose to set
ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true
as an ‘env’ in their workflow or as an environment variable on your
runner machine. This will *only work until we upgrade the runner
removing Node16 later in the spring*. (emphasis added)"

From the wheel job failures during pantsbuild#21528 and my attempts to fiddle with
all of `ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION`,
`ACTIONS_RUNNER_FORCED_INTERNAL_NODE_VERSION`,
`ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION` I think the removal time
promised for the Spring has finally come, but I'm not that familiar with
GitHub Actions and could be missing something.

As a consequence the version of both `actions/upload-artifacts` and
`actions/checkout` is bumped to `v4`. To make this testable now and in
the future, wheels are now build on ci config changes.

Why is bumping to `manylinux_2_28` (the next oldest manylinux) not a big
deal? The "2.28" refers to glibc
[2.28](https://sourceware.org/legacy-ml/libc-alpha/2018-08/msg00003.html)
in 2018. That is older than Debian stable and if you need to use
software that old you are presumably paying someone for an enterprise
distribution.

NOTE: I think we should bump the manylinux version in `main` regardless,
but if my read if the situation is correct we may need to backport this
any active release line. Without this or a more complex change we would
also be unable to release after December 5th per pantsbuild#21616.

ref pantsbuild#21195 pantsbuild#21616
benjyw added a commit that referenced this issue Nov 19, 2024
While trying to land #21528 to was observed that the wheel building jobs
were failing during git checkout with wonderful errors like
``/__e/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found
(required by /__e/node20/bin/node)``. This appears to be the same
symptoms as actions/checkout#1809 which
pointed the the deprecation notice at

<https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/>.
If this all sounds familiar that is because we *mostly* cleaned this up
in #21133, but kept a fewer uses of the older actions for
`manylinux2014` compatibility. However, from the notice:

"To opt out of this and continue using Node16 while it is still
available in the runner, you can choose to set
ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true
as an ‘env’ in their workflow or as an environment variable on your
runner machine. This will *only work until we upgrade the runner
removing Node16 later in the spring*. (emphasis added)"

From the wheel job failures during #21528 and my attempts to fiddle with
all of `ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION`,
`ACTIONS_RUNNER_FORCED_INTERNAL_NODE_VERSION`,
`ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION` I think the removal time
promised for the Spring has finally come, but I'm not that familiar with
GitHub Actions and could be missing something.

As a consequence the version of both `actions/upload-artifacts` and
`actions/checkout` is bumped to `v4`. To make this testable now and in
the future, wheels are now build on ci config changes.

Why is bumping to `manylinux_2_28` (the next oldest manylinux) not a big
deal? The "2.28" refers to glibc

[2.28](https://sourceware.org/legacy-ml/libc-alpha/2018-08/msg00003.html)
in 2018. That is older than Debian stable and if you need to use
software that old you are presumably paying someone for an enterprise
distribution.

NOTE: I think we should bump the manylinux version in `main` regardless,
but if my read if the situation is correct we may need to backport this
any active release line. Without this or a more complex change we would
also be unable to release after December 5th per #21616.

ref #21195 #21616

Co-authored-by: cburroughs <chris.burroughs@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug category:internal CI, fixes for not-yet-released features, etc.
Projects
None yet
Development

No branches or pull requests

2 participants