GET request is continuously triggered after the DELETE request is issued

[arunkpskpm]

What happened?

I’m automating the onboarding of accounts and other resources to Prisma Cloud using provider-http. As an initial step, I successfully created a role. However, when I tried to delete the managed resource (MR), it got stuck in the delete stage. After reviewing the pod logs, I noticed that a GET call is continuously triggered after the DELETE request. The external resource (the role) is deleted correctly, but the MR remains stuck. The GET/OBSERVE call is failing because the external resource was already deleted as part of the original DELETE request. I came across a similar issue reported here. I've tried several configurations in expectedResponseCheck, but none of them have worked so far.

pod logs

{"level":"info","ts":"2024-11-11T07:36:04+05:30","logger":"provider-http","msg":"http request sent: {\"method\":\"GET\",\"url\":\"https://api.eu.prismacloud.io/user/role/ce8d2265-048c-43bc-9341-07da17e7bd03\",\"headers\":{\"Content-Type\":[\"application/json\"],\"x-redlock-auth\":[\"{{response-token:crossplane-system:token}}\"]}}","request":"add-role"}
{"level":"info","ts":"2024-11-11T07:36:06+05:30","logger":"provider-http","msg":"http request sent: {\"method\":\"DELETE\",\"url\":\"https://api.eu.prismacloud.io/user/role/ce8d2265-048c-43bc-9341-07da17e7bd03\",\"headers\":{\"Content-Type\":[\"application/json\"],\"x-redlock-auth\":[\"{{response-token:crossplane-system:token}}\"]}}","request":"add-role"}
{"level":"info","ts":"2024-11-11T07:36:06+05:30","logger":"provider-http","msg":"http request sent: {\"method\":\"GET\",\"url\":\"https://api.eu.prismacloud.io/user/role/ce8d2265-048c-43bc-9341-07da17e7bd03\",\"headers\":{\"Content-Type\":[\"application/json\"],\"x-redlock-auth\":[\"{{response-token:crossplane-system:token}}\"]}}","request":"add-role"}
{"level":"info","ts":"2024-11-11T07:36:07+05:30","logger":"provider-http","msg":"http request sent: {\"method\":\"GET\",\"url\":\"https://api.eu.prismacloud.io/user/role/ce8d2265-048c-43bc-9341-07da17e7bd03\",\"headers\":{\"Content-Type\":[\"application/json\"],\"x-redlock-auth\":[\"{{response-token:crossplane-system:token}}\"]}}","request":"add-role"}
{"level":"info","ts":"2024-11-11T07:36:12+05:30","logger":"provider-http","msg":"http request sent: {\"method\":\"GET\",\"url\":\"https://api.eu.prismacloud.io/user/role/ce8d2265-048c-43bc-9341-07da17e7bd03\",\"headers\":{\"Content-Type\":[\"application/json\"],\"x-redlock-auth\":[\"{{response-token:crossplane-system:token}}\"]}}","request":"add-role"}
{"level":"info","ts":"2024-11-11T07:36:20+05:30","logger":"provider-http","msg":"http request sent: {\"method\":\"GET\",\"url\":\"https://api.eu.prismacloud.io/user/role/ce8d2265-048c-43bc-9341-07da17e7bd03\",\"headers\":{\"Content-Type\":[\"application/json\"],\"x-redlock-auth\":[\"{{response-token:crossplane-system:token}}\"]}}","request":"add-role"}
{"level":"info","ts":"2024-11-11T07:37:03+05:30","logger":"provider-http","msg":"http request sent: {\"method\":\"GET\",\"url\":\"https://api.eu.prismacloud.io/user/role/ce8d2265-048c-43bc-9341-07da17e7bd03\",\"headers\":{\"Content-Type\":[\"application/json\"],\"x-redlock-auth\":[\"{{response-token:crossplane-system:token}}\"]}}","request":"add-role"}
{"level":"info","ts":"2024-11-11T07:37:35+05:30","logger":"provider-http","msg":"http request sent: {\"method\":\"GET\",\"url\":\"https://api.eu.prismacloud.io/user/role/ce8d2265-048c-43bc-9341-07da17e7bd03\",\"headers\":{\"Content-Type\":[\"application/json\"],\"x-redlock-auth\":[\"{{response-token:crossplane-system:token}}\"]}}","request":"add-role"}
{"level":"info","ts":"2024-11-11T07:38:36+05:30","logger":"provider-http","msg":"http request sent: {\"method\":\"GET\",\"url\":\"https://api.eu.prismacloud.io/user/role/ce8d2265-048c-43bc-9341-07da17e7bd03\",\"headers\":{\"Content-Type\":[\"application/json\"],\"x-redlock-auth\":[\"{{response-token:crossplane-system:token}}\"]}}","request":"add-role"}
{"level":"info","ts":"2024-11-11T07:39:19+05:30","logger":"provider-http","msg":"http request sent: {\"method\":\"POST\",\"body\":\"{\\n  \\\"username\\\": \\\"{{ prismacloud-secrets:crossplane-system:accessKeyId }}\\\",\\n  \\\"password\\\": \\\"{{ prismacloud-secrets:crossplane-system:secretKey }}\\\"\\n}\\n\",\"url\":\"https://api.eu.prismacloud.io/login\",\"headers\":{\"Accept\":[\"application/json; charset=UTF-8\"],\"Content-Type\":[\"application/json; charset=UTF-8\"]}}","disposableRequest":"obtain-jwt-token"}
{"level":"info","ts":"2024-11-11T07:39:37+05:30","logger":"provider-http","msg":"http request sent: {\"method\":\"GET\",\"url\":\"https://api.eu.prismacloud.io/user/role/ce8d2265-048c-43bc-9341-07da17e7bd03\",\"headers\":{\"Content-Type\":[\"application/json\"],\"x-redlock-auth\":[\"{{response-token:crossplane-system:token}}\"]}}","request":"add-role"}
{"level":"info","ts":"2024-11-11T07:40:37+05:30","logger":"provider-http","msg":"http request sent: {\"method\":\"GET\",\"url\":\"https://api.eu.prismacloud.io/user/role/ce8d2265-048c-43bc-9341-07da17e7bd03\",\"headers\":{\"Content-Type\":[\"application/json\"],\"x-redlock-auth\":[\"{{response-token:crossplane-system:token}}\"]}}","request":"add-role"}
{"level":"info","ts":"2024-11-11T07:42:53+05:30","logger":"provider-http","msg":"http request sent: {\"method\":\"GET\",\"url\":\"https://api.eu.prismacloud.io/user/role/ce8d2265-048c-43bc-9341-07da17e7bd03\",\"headers\":{\"Content-Type\":[\"application/json\"],\"x-redlock-auth\":[\"{{response-token:crossplane-system:token}}\"]}}","request":"add-role"}
{"level":"info","ts":"2024-11-11T07:43:54+05:30","logger":"provider-http","msg":"http request sent: {\"method\":\"GET\",\"url\":\"https://api.eu.prismacloud.io/user/role/ce8d2265-048c-43bc-9341-07da17e7bd03\",\"headers\":{\"Content-Type\":[\"application/json\"],\"x-redlock-auth\":[\"{{response-token:crossplane-system:token}}\"]}}","request":"add-role"}

describe output after delete request is made

iehje9m@AMC-D6YC47CD9H examples % k describe request.http.crossplane.io/add-role
Name:         add-role
Namespace:
Labels:       <none>
Annotations:  crossplane.io/external-create-pending: 2024-11-11T07:16:14+05:30
              crossplane.io/external-create-succeeded: 2024-11-11T07:16:15+05:30
              crossplane.io/external-name: add-role
API Version:  http.crossplane.io/v1alpha2
Kind:         Request
Metadata:
  Creation Timestamp:             2024-11-11T01:46:14Z
  Deletion Grace Period Seconds:  0
  Deletion Timestamp:             2024-11-11T02:05:07Z
  Finalizers:
    finalizer.managedresource.crossplane.io
  Generation:        2
  Resource Version:  2195897
  UID:               dc70ddad-ab1c-4269-bd67-243b0d6e48bc
Spec:
  Deletion Policy:  Delete
  For Provider:
    Expected Response Check:
      Logic:  if .response.body.name == .payload.body.name
 then true
 else false
 end

      Type:  CUSTOM
    Headers:
      Content - Type:
        application/json
      X - Redlock - Auth:
        {{response-token:crossplane-system:token}}
    Mappings:
      Action:  CREATE
      Body:    {
name: .payload.body.name,
roleType: .payload.body.roleType,
accountGroupIds: ["55555f8c-8867-4a9a-b608-1bab38db8c40"],
description: .payload.body.description
}

      Headers:
        Content - Type:
          application/json
        X - Redlock - Auth:
          {{response-token:crossplane-system:token}}
      Method:  POST
      URL:     .payload.baseUrl + "/user/role"
      Action:  OBSERVE
      Method:  GET
      URL:     (.payload.baseUrl + "/user/role/" + (.response.body.id|tostring))
      Action:  UPDATE
      Body:    {
name: .payload.body.name,
roleType: .payload.body.roleType,
accountGroupIds: ["55555f8c-8867-4a9a-b608-1bab38db8c40"],
description: .payload.body.description
}

      Method:  PUT
      URL:     (.payload.baseUrl + "/user/role/" + (.response.body.id|tostring))
      Method:  DELETE
      URL:     (.payload.baseUrl + "/user/role/" + (.response.body.id|tostring))
    Payload:
      Base URL:  https://api.eu.prismacloud.io
      Body:      {
  "name": "test role1",
  "roleType": "Owners",
  "description": "Created from crossplane"
}

  Management Policies:
    *
  Provider Config Ref:
    Name:  http-conf
Status:
  Cache:
    Last Updated:  2024-11-11T02:06:04Z
    Response:
      Body:  {"id":"ce8d2265-048c-43bc-9341-07da17e7bd03","name":"test role1","description":"Created from crossplane","lastModifiedBy":"OnboardingUser","lastModifiedTs":1731289575107,"accountGroupIds":["55555f8c-8867-4a9a-b608-1bab38db8c40"],"resourceListIds":[],"codeRepositoryIds":[],"associatedUsers":[],"restrictDismissalAccess":false,"permissionGroup":null,"additionalAttributes":{"onlyAllowCIAccess":false,"onlyAllowComputeAccess":false,"hasDefenderPermissions":false,"onlyAllowReadAccess":false},"accountGroups":[],"resourceLists":[],"codeRepositories":[],"roleType":"Owners"}
      Headers:
        Cache - Control:
          no-cache, no-store, max-age=0, must-revalidate
        Connection:
          keep-alive
        Content - Security - Policy:
          default-src 'self' https:; font-src 'self' https: data:; img-src 'self' https: data:; object-src 'none'; script-src https:; style-src 'self' https: 'unsafe-inline'
        Content - Type:
          application/json
        Date:
          Mon, 11 Nov 2024 02:06:04 GMT
        Expires:
          0
        Pragma:
          no-cache
        Referrer - Policy:
          no-referrer
        Strict - Transport - Security:
          max-age=31536000 ; includeSubDomains
        Trace - Id:
          882dff959fcae095eced096960b5f692
        Vary:
          Origin
          Access-Control-Request-Method
          Access-Control-Request-Headers
          accept-encoding
          x-redlock-auth
          Origin
        X - Content - Type - Options:
          nosniff
        X - Download - Options:
          noopen
        X - Frame - Options:
          DENY
        X - Permitted - Cross - Domain - Policies:
          none
        X - Xss - Protection:
          0
      Status Code:  200
  Conditions:
    Last Transition Time:  2024-11-11T02:06:06Z
    Reason:                Available
    Status:                True
    Type:                  Ready
    Last Transition Time:  2024-11-11T02:06:06Z
    Message:               observe failed:  failed updating status: HTTP GET request failed with status code: 400
    Reason:                ReconcileError
    Status:                False
    Type:                  Synced
  Failed:                  22
  Request Details:
    Headers:
      Content - Type:
        application/json
      X - Redlock - Auth:
        {{response-token:crossplane-system:token}}
    Method:  GET
    URL:     https://api.eu.prismacloud.io/user/role/ce8d2265-048c-43bc-9341-07da17e7bd03
  Response:
    Body:  {"timestamp":"2024-11-11T02:26:45.265714384Z","status":400,"error":"INVALID ID","message":"invalid_id","path":"GET:/api/v1/user/role/ce8d2265-048c-43bc-9341-07da17e7bd03"}
    Headers:
      Cache - Control:
        no-cache, no-store, max-age=0, must-revalidate
      Connection:
        keep-alive
      Content - Security - Policy:
        default-src 'self' https:; font-src 'self' https: data:; img-src 'self' https: data:; object-src 'none'; script-src https:; style-src 'self' https: 'unsafe-inline'
      Content - Type:
        application/json
      Date:
        Mon, 11 Nov 2024 02:26:45 GMT
      Expires:
        0
      Pragma:
        no-cache
      Referrer - Policy:
        no-referrer
      Strict - Transport - Security:
        max-age=31536000 ; includeSubDomains
      Trace - Id:
        46be61ad0b33ba2b097f744a73c8e838
        46be61ad0b33ba2b097f744a73c8e838
      Vary:
        Origin
        Access-Control-Request-Method
        Access-Control-Request-Headers
        accept-encoding
        x-redlock-auth
        Origin
      X - Content - Type - Options:
        nosniff
      X - Download - Options:
        noopen
      X - Frame - Options:
        DENY
      X - Permitted - Cross - Domain - Policies:
        none
      X - Redlock - Status:
        [{"severity":"error","i18nKey":"invalid_id"}]
      X - Xss - Protection:
        0
    Status Code:  400
Events:
  Type     Reason                         Age                 From                                Message
  ----     ------                         ----                ----                                -------
  Normal   CreatedExternalResource        40m                 managed/request.http.crossplane.io  Successfully requested creation of external resource
  Normal   DeletedExternalResource        20m                 managed/request.http.crossplane.io  Successfully requested deletion of external resource
  Warning  CannotObserveExternalResource  13s (x22 over 20m)  managed/request.http.crossplane.io  failed updating status: HTTP GET request failed with status code: 400
iehje9m@AMC-D6YC47CD9H examples %

How can we reproduce it?

Please find the manifest which used for role creation.

apiVersion: http.crossplane.io/v1alpha2
kind: Request
metadata:
  name: add-role
spec:
  deletionPolicy: Delete
  forProvider:
    # insecureSkipTLSVerify: true
    waitTimeout: 5m
    headers:
      Content-Type:
        - application/json
      x-redlock-auth:
        - "{{response-token:crossplane-system:token}}"
    payload:
      baseUrl: 'https://api.eu.prismacloud.io'
      body: |
        {
          "name": "test role1",
          "roleType": "Owners",
          "description": "Created from crossplane"
        }
    mappings:
      - action: CREATE
        method: POST
        body: |
          {
          name: .payload.body.name,
          roleType: .payload.body.roleType,
          accountGroupIds: ["55555f8c-8867-4a9a-b608-1bab38db8c40"],
          description: .payload.body.description
          }
        url: .payload.baseUrl + "/user/role"
        headers:
          Content-Type:
            - application/json
          x-redlock-auth:
            - "{{response-token:crossplane-system:token}}"

      - action: OBSERVE
        method: GET
        url: (.payload.baseUrl + "/user/role/" + (.response.body.id|tostring))

      - action: UPDATE
        method: PUT
        url: (.payload.baseUrl + "/user/role/" + (.response.body.id|tostring))
        body: |
          {
          name: .payload.body.name,
          roleType: .payload.body.roleType,
          accountGroupIds: ["55555f8c-8867-4a9a-b608-1bab38db8c40"],
          description: .payload.body.description
          }
          
      - method: DELETE
        url: (.payload.baseUrl + "/user/role/" + (.response.body.id|tostring))
    expectedResponseCheck:
      type: CUSTOM
      logic: |
        if .response.body.name == .payload.body.name
         then true 
         else false 
         end
  providerConfigRef:
    name: http-conf

What environment did it happen in?

Crossplane version: v1.17.2

Include at least the version or commit of Crossplane you were running. Consider
also including your:

• Cloud provider or hardware configuration
• Kubernetes version (use kubectl version)

iehje9m@AMC ~ % kubectl version
Client Version: v1.31.0
Kustomize Version: v5.4.2
Server Version: v1.31.0

• Kubernetes distribution (e.g. Tectonic, GKE, OpenShift) - Kind
• OS (e.g. from /etc/os-release) - MacOS
• Kernel (e.g. uname -a)

Cc: @barunavo

[arielsepton]

Hi,

Just to explain, the provider sends a GET request after a DELETE to confirm the resource was deleted. Right now, it only supports cases where this GET returns a 404 status. In your case, it looks like the GET is returning a 400, which is causing the issue. I understand the need for more flexibility here.

[arunkpskpm]

Thank you so much for the enhancement release @arielsepton 👍