ECR login succeeds, but push fails with HEAD request 401 unauthorized

[aoyama-val]

Orb version

9.1.0

What happened

version: 2.1
orbs:
  aws-ecr: circleci/aws-ecr@9.1.0
  aws-cli: circleci/aws-cli@4.1.3
workflows:
  build:
    jobs:
      - aws-ecr/build_and_push_image:
          context: *************
          repo: *************
          region: ${AWS_DEFAULT_REGION}
          tag: "${CIRCLE_SHA1}"
          auth:
            - aws-cli/setup:
              role_arn: ${AWS_IAM_ROLE_ARN}

#16 exporting to image
#16 exporting layers
#16 exporting layers 5.9s done
#16 exporting manifest sha256:6df314dcc346194eaf73963e15521ca9679586d95080a6efae886615b26306c0 0.0s done
#16 exporting config sha256:2069114fce5394cc9474b71dceb31fe1bbf781a27e349eeb61c8b7fa990f4a5e 0.0s done
#16 exporting attestation manifest sha256:d205a952b67cd4fd29396d25bbb4008368d2553bfbdb139a943572d1d7573b0d 0.0s done
#16 exporting manifest list sha256:d4b739fe2b3923dda7fe06893123cbfa061ef98eecf03d7dee037b7b414f281f 0.0s done
#16 pushing layers
#16 pushing layers 0.6s done
#16 ERROR: failed to push ************.dkr.ecr.**************.amazonaws.com/ticketing-rails:d4a654fb86a8e1c7ef4a912c49a96a73921aaea7: unexpected status from HEAD request to https://************.dkr.ecr.**************.amazonaws.com/v2/ticketing-rails/blobs/sha256:1c656b3fb894779c46614514306b4b427ed930d4f4bddd8d085b475bc49263d3: 401 Unauthorized
------
 > exporting to image:
------

 2 warnings found (use --debug to expand):
 - LegacyKeyValueFormat: "ENV key=value" should be used instead of legacy "ENV key value" format (line 2)
 - LegacyKeyValueFormat: "ENV key=value" should be used instead of legacy "ENV key value" format (line 12)
ERROR: failed to solve: failed to push ************.dkr.ecr.**************.amazonaws.com/*************:d4a654fb86a8e1c7ef4a912c49a96a73921aaea7: unexpected status from HEAD request to https://************.dkr.ecr.**************.amazonaws.com/v2/*************/blobs/sha256:1c656b3fb894779c46614514306b4b427ed930d4f4bddd8d085b475bc49263d3: 401 Unauthorized

Exited with code exit status 1

Expected behavior

Push succeeds.

[gonrial]

@aoyama-val can you check that the role stored in AWS_IAM_ROLE_ARN has permissions to push to that ECR repo?
If you have permissions, but it still not working, can you list the permissions associated to that role?

[aoyama-val]

The role has these permissions at least:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "ecr:BatchCheckLayerAvailability",
            "Resource": [
                "arn:aws:ecr:*****"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "ecr:CompleteLayerUpload",
                "ecr:UploadLayerPart",
                "ecr:InitiateLayerUpload",
                "ecr:PutImage",
                "ecr:BatchGetImage"
            ],
            "Resource": [
                "arn:aws:ecr:*****"
            ]
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": "ecr:GetAuthorizationToken",
            "Resource": "*"
        }
    ]
}

The config was working fine with aws-ecr@8.2.1 + aws-cli@3.1.4, however I faced the issue when tried to upgrade the orbs.

Thanks.

[aoyama-val]

I added Allow All permission to the role, but still push failed with the same error.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Action": "*",
            "Resource": "*"
        }
    ]
}

Also I changed the auth method from OIDC to access key/secret, and it worked fine. So I guess something about OIDC is wrong.

[gonrial]

@aoyama-val what is the definition for the role that you are using for OIDC? Pay special attention at the Condition section

[aoyama-val]

Carefully checked it but everything seemed correct. Also tried deleting the whole Condition, but nothing changed.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::XXXXXXXXXXXX:oidc-provider/oidc.circleci.com/org/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"
            },
            "Action": "sts:AssumeRoleWithWebIdentity"
        }
    ]
}

[dgreda]

I'm facing the same problem. I've been using version 9.0.1 and it was working, but as soon as I upgraded to 9.1.0 it's no longer working.

It happens on the attempt to push the image.

#10 ERROR: failed to push ************.dkr.ecr.*********.amazonaws.com/***: unexpected status from HEAD request to https://************.dkr.ecr.*********.amazonaws.com/v2/***: 401 Unauthorized

[dreik]

Same issue, starting v9.0.3

[sylwit]

Confirm downgrading to v9.0.1 fix the issue

[marboledacci]

This is more likely a problem with the aws-cli orb, could you do a aws-cli/setup command alone and validate the authentication worked?

You can do something like this:

- aws-cli/setup:
     role_arn: ${ROLE_ARN}
     post-steps:
            - run:
                name: "Validation"
                command: aws sts get-caller-identity

A combination of versions that should work using OIDC is: circleci/aws-cli@5.1.0 and circleci/aws-ecr-orb@9.3.0.
Please check if it works using these versions.

[sylwit]

https://github.com/CircleCI-Public/aws-ecr-orb/blob/master/src/commands/build_and_push_image.yml#L185

auth: shouldn't be required if it's already set previously.

[marboledacci]

@sylwit That's right, the auth shouldn't be required. Is that causing something related to his issue?

[b-jan]

Hello, we face the same issue trying to migrate from circleci/aws-ecr@8.2 to circleci/aws-ecr@9.3.2 with the new auth parameter.

unexpected status from HEAD request... : 401 Unauthorized

Did you guys find a fix?
We cannot understand how to do the migration correctly.

[marboledacci]

@b-jan how are you doing the authentication to ECR? Are you enabling credentials helper?

[jeandlv]

Thanks @marboledacci,

We fixed our issue by adding these parameters :

 use_credentials_helper: false
 extra_build_args: '--provenance=false'

We added extra_build_args: '--provenance=false' to fix this issue.

[rafaelnogueira1]

@aoyama-val were you able to resolve this issue? I'm facing the same situation after updating.

[benmurden]

If you just want things to work the way they did before, wherever you use aws-cli/setup, explicitly export the credential values:

- aws-ecr/build_and_push_image:
    auth:
      - aws-cli/setup:
          role_arn: arn:aws:iam::<account_id>:role/role_name
      - run:
          name: "Export AWS credentials"
          command: |
            echo "export AWS_ACCESS_KEY_ID=$AWS_CLI_STR_ACCESS_KEY_ID" >> $BASH_ENV
            echo "export AWS_SECRET_ACCESS_KEY=$AWS_CLI_STR_SECRET_ACCESS_KEY" >> $BASH_ENV
            echo "export AWS_SESSION_TOKEN=$AWS_CLI_STR_SESSION_TOKEN" >> $BASH_ENV
      - run:
          name: "Validation"
          command: aws sts get-caller-identity
...

Explanation

This is due to changes in the aws-cli orb in which it no longer exports the environment variables with their standard names. It is supposed to set up a config profile instead, but for whatever reason that wasn't working for me. They also do this to prevent issues with multiple credentials in a single job, so if you need that, be aware this may mess things up.

I also added a validation step to make sure this is actually succeeding and not pushing the issue down the road after doing a whole Docker image build.