ci: merge cached Nix stores

[develop7]

Introduces a nix cache warm-up job, which prefetches all the derivations we ever need to the Nix store, which is cached right away. Such cache is reused then in further actions, reducing in-job downloads to zero.

Before this PR every nix-involving job (13 of them on Linux) took 3 to 4 minutes to do "nix setup" due to downloading 121 to 1409 paths, 300 to 1360 MB big. While these jobs cache Nix store here and there, there's a handful of them, multiplied by 2.3-ish GB in size, and thus they exceed the upper cache limit for GH Actions of 10 GB.
After merging this PR, a cold run takes ~6 minutes to warm up the cache, then every nix-involving job takes 1:30 mins to unpack the cache and have all the derivations installed. A warm run takes 1 minute of a no-op cache warm up step and the same 1:30 to unpack the cache. This produces one Nix store cache per OS of 2.3 GB in size, which fit the storage nicely and would probably be a benefit for every build, for main and PR branches as well.

TODO: make the loadtest workflow use a common cache as well, maybe by extracting the "warm up" job to a separate workflow.

[develop7]

@laurenceisla @steve-chavez could you guys adopt this PR so we could test if purging individual caches actually works (and cache space is saved) because it looks like I'm being hit by the lack of permissions again (see here)?

[develop7]

In example workflow at c-n-a repository this parameter is set in "merge caches" jobs only, so I've followed it. I think the reason was to produce the individual cache still, so the "merge" job had caches to merge from.

…

On Wed, Nov 29, 2023, 17:06 Laurence Isla ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In .github/actions/setup-nix/action.yaml <#3071 (comment)>: > @@ -23,7 +23,6 @@ runs: key: cache-nix-${{ runner.os }}-id-${{ inputs.cache-id }}-${{ hashFiles('nix/**/*.nix') }} restore-keys: | cache-nix-${{ runner.os }}-common- - restore-key-hit: true Why is this removed? If active a cache won't be created if it hits the key, does that cause a problem? — Reply to this email directly, view it on GitHub <#3071 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAV6IN2D2NPFYM2SRBNMVLYG5MORAVCNFSM6AAAAAA75572YOVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTONJVGY2DQOJZGI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[develop7]

Okay, I've extracted the job to a separate workflow, now it should be merged to main to be triggered (see the note at https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_run)

@laurenceisla @steve-chavez

[develop7]

Another approach I can think of is to prepopulate nix store with all the related derivations in a separate job or even workflow, cache it and reuse this cache in subsequent jobs.

Let me try this.

[develop7]

So prepopulating job takes 6 minutes, produces one cache 2.3G big, all the "nix setup" steps in the nix-involving jobs on Linux take 1:30 mins tops.

[laurenceisla]

Looks great! Took 8 mins to complete all the tests, and that's mostly due to the macOS build. GJ!

[laurenceisla]

It's faster now! But I noticed these logs are starting to show (they're also present in this PR's Actions), e.g.:

/usr/bin/tar: ../../../../../nix/store/jad0xjhpmbb7hhsi3r0sp9llbhcy1qqb-aws-c-mqtt-0.8.14/include/aws/mqtt/v5/mqtt5_listener.h: Cannot open: File exists
/usr/bin/tar: ../../../../../nix/store/jad0xjhpmbb7hhsi3r0sp9llbhcy1qqb-aws-c-mqtt-0.8.14/include/aws/mqtt/v5/mqtt5_listener.h: Cannot open: File exists
/usr/bin/tar: ../../../../../nix/store/jad0xjhpmbb7hhsi3r0sp9llbhcy1qqb-aws-c-mqtt-0.8.14/include/aws/mqtt/v5/mqtt5_client.h: Cannot open: File exists
/usr/bin/tar: ../../../../../nix/store/jad0xjhpmbb7hhsi3r0sp9llbhcy1qqb-aws-c-mqtt-0.8.14/include/aws/mqtt/v5/mqtt5_types.h: Cannot open: File exists
...

https://github.com/PostgREST/postgrest/actions/runs/7105291857/job/19342398874#step:3:68

That didn't happen before:

https://github.com/PostgREST/postgrest/actions/runs/7092294352/job/19303274920#step:3:68

I believe the Jobs could be even faster if that didn't happen. Not sure where the problem is though, the only difference I see for that step is that restore-key-hit is false now.

[develop7]

But I noticed these logs are starting to show (they're also present in this PR's Actions), e.g.:

Noticed those too it appears to be fine: https://github.com/nix-community/cache-nix-action/#limitations

[laurenceisla]

Ah, it's a known issue then. OK, let's leave it at that.