Best way to cache multi-stage + multi-platform image builds

[AngellusMortis]

Docs request. Basically what is the best way to cache intermediate layers for a multi-stage build for multiple platforms? Is there a way to run two completely different commands, one per platform, and then push them to the same resulting tag?

We are building multi-stage and multi-platform images in Github Actions using docker/build-push-action and multiple buildkit builder nodes (Workflow job runs on a linux/amd64 node and then we add a linux/arm64 node after we initialize the builder). Basically as shown here: https://github.com/docker/buildx#building-multi-platform-images

Name:   builder-ae1e5dfb-7e25-4a10-a6dd-d0912305decf
Driver: docker-container

Nodes:
Name:      builder-ae1e5dfb-7e25-4a10-a6dd-d0912305decf0
Endpoint:  unix:///var/run/docker.sock
Status:    running
Flags:     --debug
Buildkit:  v0.10.5
Platforms: linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386

Name:      arm64-builder
Endpoint:  ssh://1.1.1.1
Status:    running
Flags:     --debug
Buildkit:  v0.10.5
Platforms: linux/arm64*

We are building docker images for a commit, then testing it and promoting it if tests pass. There is also a cache-from argument pulling a cache from the current branch build cache and the main branch build cache. Then cache-to back to the current branch build cache.

The issue we are seeing is that the intermediate layers for the linux/arm64 platform never seem to cache, resulting in the linux/arm64 image essentially always rebuilding.

The full command that gets ran by Github Actions:

/usr/bin/docker buildx build \
    --build-arg GIT_REF=eaabf7c57706641d1c1f1fc384ec14e9333c3413 \
    --cache-from ghcr.io/org/repo:buildcache-branch,ghcr.io/org/repo:buildcache-master \
    --cache-to type=registry,ref=ghcr.io/org/repo:buildcache-branch,mode=max \
    --file Dockerfile \
    --iidfile /tmp/docker-build-push-cM103X/iidfile \
    --platform linux/amd64,linux/arm64 \
    --tag ghcr.io/org/repo:branch \
    --tag ghcr.io/org/repo:commit-eaabf7c57706641d1c1f1fc384ec14e9333c3413 \
    --target prod \
    --metadata-file /tmp/docker-build-push-cM103X/metadata-file \
    --push .

[jedevc]

👋 Heya, sorry it took so long to get to this!

This issue sounds a lot like this: #1044. Essentially, while buildx has special logic for linking together separate outputs again after push, there's no similar logic for linking the caches. We also discussed this on the community slack some time back.

As a workaround in the meantime, you could perform multiple builds - because buildkit's own cache will apply for the first build, you can do your whole build + export, then have a special build step just to export linux/amd64 cache, then another special build step just to export the linux/arm64 cache. As long as these are on the same set of builders, the special steps won't need to rebuild the cache, but will just push it to the target destination - then you can setup your --cache-from to point to both of those separate sources.

[AngellusMortis]

So something like this?

docker buildx --cache-from image:buildcache-arm64 --cache-from image:buildcache-amd64 -t image:latest --push .
# only run on arm64
docker buildx --cache-from image:buildcache-arm64 --cache-to image:buildcache-arm64 .
# only run on amd64
docker buildx --cache-from image:buildcache-amd64 --cache-to image:buildcache-amd64 .

[AngellusMortis]

It appears that worked like a charm. Thanks!

[pecigonzalo]

Unfortunately this only works when there is a single buildkit builder, but if you have a pool of them (like in the example StatefulSet or Deployment for Kuberentes) you might land on a builder that does not have it.

[masip85]

I can't see the correlation between this...

wave Heya, sorry it took so long to get to this!

This issue sounds a lot like this: #1044. Essentially, while buildx has special logic for linking together separate outputs again after push, there's no similar logic for linking the caches. We also discussed this on the community slack some time back.

As a workaround in the meantime, you could perform multiple builds - because buildkit's own cache will apply for the first build, you can do your whole build + export, then have a special build step just to export linux/amd64 cache, then another special build step just to export the linux/arm64 cache. As long as these are on the same set of builders, the special steps won't need to rebuild the cache, but will just push it to the target destination - then you can setup your --cache-from to point to both of those separate sources.

... and the code @AngellusMortis provides. Could you explain a little bit further?

[AngellusMortis]

The docker build cache can only support a single build/platform at a time. Doing a multi-platform build and then caching the build (--cache-to) will result in only one of the two build caches being kept. It is a race condition for which gets saved.

i.e. the following command will only create a final build cache for amd64 or arm64, never both.

docker buildx build \
    --cache-from image:buildcache \
    --cache-to type=registry,ref=image:buildcache,mode=max \
    --platform linux/amd64,linux/arm64 \
    --t image:latest --push .

However, the current builder nodes still have both caches. So, you can run the build three times, once to push the final image, one to push the amd64 build cache and once to push the arm64 build cache. The second two builds will automatically use the local build caches and cache straight through and only push the build cache up:

docker buildx \
    --cache-from image:buildcache-arm64 
    --cache-from image:buildcache-amd64
    --platform linux/amd64,linux/arm64
    -t image:latest --push .
# only run on arm64
docker buildx 
    --cache-from image:buildcache-arm64 
    --cache-to image:buildcache-arm64
    --platform linux/arm64
    .
# only run on amd64
docker buildx \
     --cache-from image:buildcache-amd64 
     --cache-to image:buildcache-amd64 
     --platform linux/amd64
     .

[masip85]

Ah, ok. I was wrong about one concept. Local caches aren't type=local? The ones where you specify the path I mean. You specify registry, which .. can be and stored only locally? Os are them just a transient until push?

(i am working locally by the way,but I want to push final multi platform image with its cache layers )

I still have a doubts.

• buildcache is an arbitrary tag? Will it be uploaded to separate image?
• Why 2 latest commands export cache without push?
• Do I must execute this

 docker buildx \
    --cache-from image:buildcache-arm64 
    --cache-from image:buildcache-amd64
    --platform linux/amd64,linux/arm64
    -t image:latest --push .

always after this? :

docker buildx build \
    --cache-from image:buildcache \
    --cache-to type=registry,ref=image:buildcache,mode=max \
    --platform linux/amd64,linux/arm64 \
    --t image:latest --push .

[AngellusMortis]

No, by "local" I mean the cache built into the builder itself. If you take a simple Dockerfile and build it twice it in a row, the second time it will just cache straight through and not build anything new. That is because the builder keeps its own cache outside of what --cache-to/--cache-form args you provide. That is why the --no-cache arg exists.

For everything else, it looks like you are just missing something about how build caches and the registry build cache works. I recommend reading the docs for build caches which has a bunch of examples of how the "registry" build cache works. Then also look at linked ticket (#1044). Then come back here and look at the answer and my comments again.

I can assure you the approach I mentioned does work. We have been using it for over 6 months now (ever since my original reply in November).

[pecigonzalo]

docker buildx 
    --cache-from image:buildcache-arm64 
    --cache-to image:buildcache-arm64
    --platform linux/arm64
    .

this is missing build and --load right? or --push to a fake target? Using --push feels incorrect, as we already pushed before.

[jaape]

docker buildx 
    --cache-from image:buildcache-arm64 
    --cache-to image:buildcache-arm64
    --platform linux/arm64
    .

this is missing build and --load right? or --push to a fake target? Using --push feels incorrect, as we already pushed before.

The examples provided earlier may have been in the incorrect order.
Here is a revised version for pushing a multi-platform image while using build cache

1. Build arm64 image, use cache if available, do not push

docker buildx build --cache-from=type=registry,ref=image:buildcache-arm64 --cache-to=type=registry,ref=image:buildcache-arm64 --platform linux/arm64 --load .

2. Build amd64 image, use cache if available, do not push

docker buildx build --cache-from=type=registry,ref=image:buildcache-amd64 --cache-to=type=registry,ref=image:buildcache-amd64 --platform linux/amd64 --load .

3. Build the final image, use cache from above builds, push

docker buildx build --cache-from=type=registry,ref=image:buildcache-arm64 --cache-from=type=registry,ref=image:buildcache-amd64 -t image:buildcache --output=type=image --platform linux/arm64,linux/amd64 --push .

All the commands should go through a single buildkit builder as the builder building the final image needs to have the first two cached builds. Subsequent builds can use a different builder, as the cache is pushed in the last stage.

[ajnozari]

For anyone dealing with this on a GitHub action, you can use a matrix to prefix the cache.

Works quite well for me, although support on your specific registry might vary.

matrix:
    build: [
        {platform: linux/amd64, cache: cache-amd64},
        {platform: linux/arm/v7, cache: cache-armv7},
        {platform: linux/arm/v8, cache: cache-armv7},
        {platform: linux/arm64, cache: cache-arm64},
    ]