[Bug?]: Post-release changes to 1.22.22 breaking checksum calculation for packages built from source using both Yarn 1.x and 2+

[Esaron]

Self-service

• I'd be willing to implement a fix

Describe the bug

Copying from Yarn 1.x issue:

Looks like a few days ago, 2024/05/24, the latest version of Yarn classic (1.22.22) was updated after already having been released to the community. This caused all checksums calculated for packages built from source (IE for dependencies pointing at specific GitHub refs, etc) to change. At my company specifically, when we did a routine cache-busting operation, all of our CI/CD builds started failing and we had to spend the day before a US holiday scrambling to figure out what happened and make sure we could release code next week. Setting aside the issue of whether releases should be updated without a version change after being in the wild for months, does it even make sense for the checksums of packages to change when the dependency itself hasn't changed at all, and only the yarn-classic.cjs file downloaded during the packing process has?

To reproduce

It's impossible to provide this because the Yarn 1.22.22 release changed from what it used to be and I don't know how to get the old version anymore. You need to have a yarn.lock file with a built-from-source dependency version that has an old checksum generated with the previously released Yarn 1.22.22. If you have that, you just need to run:

yarn cache clear; yarn install and you will receive the error: package@https://github.com/foo/bar.git#commit=[sha]: The remote archive doesn't match the expected checksum

Environment

System:
    OS: macOS 14.1
    CPU: (14) arm64 Apple M3 Max
  Binaries:
    Node: 18.20.1 - /private/var/folders/bq/mxpnyn_x0bs4l9d01_8z_jn80000gp/T/xfs-c3abd307/node
    Yarn: 4.2.2 - /private/var/folders/bq/mxpnyn_x0bs4l9d01_8z_jn80000gp/T/xfs-c3abd307/yarn
    npm: 10.5.0 - /opt/homebrew/opt/node@18/bin/npm

Additional context

No response

[Esaron]

One thing I've noticed is that before we upgraded to Yarn 4.x, the old version of the lockfile did not include integrity keys for dependencies pointing directly at refs in GitHub, while the new lockfile does include checksum keys for them.

[arcanis]

Why did you open this thread on both repos? Is this related to modern releases at all?

[Esaron]

I opened on both repos because it seems to be specific behavior with Yarn 2+ builds when packing classic packages.

See my latest comment from the other issue here.

Let's just continue the conversation here, since I imagine this issue doesn't affect Yarn classic (I checked our old yarn.lock file from back when we still used classic in March, and there doesn't appear to be the same chance for conflicts since the integrity key wasn't included for this package that now has a checksum key; see my earlier comment in this issue).

[arcanis]

which appears to be included in the zip

This is the issue. We add /.yarn to the npmignore before packing, but in the case of ember-ajax-fetch the npmignore didn't have a trailing newline. That caused the .yarn folder to not be excluded as it should. We need to explicitly add an extra newline just in case.

[f1sherman]

Any update on this issue?