external-name annotation on client lost

[TomBillietKlarrio]

Hi,

We have quite some openid clients we create trough crossplane. However, from time to time, they seem to loose the external-name annotation in the metadata section, causing crossplane to give errors on those objects as it will try to create new ones in keycloak and can't do that.
We're unsure why this happens, we seems to trigger it sometimes when we restart some k8s nodes during a software release, but haven't been able to pinpoint the exact root.
I've extracted some logs that are related to a client at the moment it got corrupted. It looks like it looses the object from the cache, and tries to rebuild it? But then why would it remove the external-name annotation? Any help is appreciated.

2024-10-02T12:54:09Z	DEBUG	provider-keycloak	Async create ended.	{"trackerUID": "6be476ee-b482-413b-80e4-920959348371", "resourceName": "XXXXX", "gvk": "openidclient.keycloak.crossplane.io/v1alpha1, Kind=Client", "error": "async create failed: failed to create the resource: [{0 error sending POST request to /auth/admin/realms/poc-dsh/clients: 409 Conflict. Response body: {\"errorMessage\":\"Client XXXXX already exists\"}  []}]", "tfID": ""}
2024-10-02T12:54:09Z	DEBUG	provider-keycloak	Calling the inner handler for Update event.	{"gvk": "openidclient.keycloak.crossplane.io/v1alpha1, Kind=Client", "name": "XXXXX", "queueLength": 1}
2024-10-02T12:54:09Z	DEBUG	provider-keycloak	Successfully requested creation of external resource	{"controller": "managed/openidclient.keycloak.crossplane.io/v1alpha1, kind=client", "request": {"name":"XXXXX"}, "uid": "6be476ee-b482-413b-80e4-920959348371", "version": "561154598", "external-name": "", "external-name": ""}
2024-10-02T12:54:09Z	DEBUG	provider-keycloak	Creating the external resource	{"uid": "6be476ee-b482-413b-80e4-920959348371", "name": "XXXXX", "gvk": "openidclient.keycloak.crossplane.io/v1alpha1, Kind=Client"}
2024-10-02T12:54:09Z	DEBUG	provider-keycloak	Async create starting...	{"trackerUID": "6be476ee-b482-413b-80e4-920959348371", "resourceName": "XXXXX", "gvk": "openidclient.keycloak.crossplane.io/v1alpha1, Kind=Client", "tfID": ""}
2024-10-02T12:54:09Z	DEBUG	provider-keycloak	Reconciling	{"controller": "managed/openidclient.keycloak.crossplane.io/v1alpha1, kind=clientdefaultscopes", "request": {"name":"XXXXX"}}
2024-10-02T12:54:09Z	DEBUG	provider-keycloak	Diff detected	{"uid": "6be476ee-b482-413b-80e4-920959348371", "name": "XXXXX", "gvk": "openidclient.keycloak.crossplane.io/v1alpha1, Kind=Client", "instanceDiff": "*terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{\"access_token_lifespan\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"access_type\":*terraform.ResourceAttrDiff{Old:\"\", New:\"CONFIDENTIAL\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"admin_url\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"backchannel_logout_session_required\":*terraform.ResourceAttrDiff{Old:\"\", New:\"true\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"base_url\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"client_authenticator_type\":*terraform.ResourceAttrDiff{Old:\"\", New:\"client-secret\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"client_id\":*terraform.ResourceAttrDiff{Old:\"\", New:\"XXXXX\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"client_offline_session_idle_timeout\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"client_offline_session_max_lifespan\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"client_secret\":*terraform.ResourceAttrDiff{Old:\"\", New:\"b613f5bd-8066-4695-b6f2-483ad720df59\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:true, Type:0x0}, \"client_session_idle_timeout\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"client_session_max_lifespan\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"consent_required\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"consent_screen_text\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"description\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"direct_access_grants_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"display_on_consent_screen\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"true\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"exclude_session_state_from_auth_response\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"frontchannel_logout_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"full_scope_allowed\":*terraform.ResourceAttrDiff{Old:\"\", New:\"false\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"implicit_flow_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"import\":*terraform.ResourceAttrDiff{Old:\"\", New:\"false\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"name\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"oauth2_device_authorization_grant_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"false\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"realm_id\":*terraform.ResourceAttrDiff{Old:\"\", New:\"poc-dsh\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"resource_server_id\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"root_url\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"service_account_user_id\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"service_accounts_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"true\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"standard_flow_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"use_refresh_tokens\":*terraform.ResourceAttrDiff{Old:\"\", New:\"true\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"use_refresh_tokens_client_credentials\":*terraform.ResourceAttrDiff{Old:\"\", New:\"false\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"valid_post_logout_redirect_uris.#\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"valid_redirect_uris.#\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"web_origins.#\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, RawConfig:cty.NilVal, RawState:cty.NilVal, RawPlan:cty.NilVal, Meta:map[string]interface {}(nil)}"}
2024-10-02T12:54:09Z	DEBUG	provider-keycloak	Observing the external resource	{"uid": "6be476ee-b482-413b-80e4-920959348371", "name": "XXXXX", "gvk": "openidclient.keycloak.crossplane.io/v1alpha1, Kind=Client"}
2024-10-02T12:54:09Z	DEBUG	provider-keycloak	Instance state not found in cache, reconstructing...	{"uid": "6be476ee-b482-413b-80e4-920959348371", "name": "XXXXX", "gvk": "openidclient.keycloak.crossplane.io/v1alpha1, Kind=Client"}
2024-10-02T12:54:08Z	DEBUG	provider-keycloak	Calling the inner handler for Update event.	{"gvk": "openidclient.keycloak.crossplane.io/v1alpha1, Kind=Client", "name": "XXXXX", "queueLength": 0}
2024-10-02T12:54:08Z	DEBUG	provider-keycloak	Connecting to the service provider	{"uid": "6be476ee-b482-413b-80e4-920959348371", "name": "XXXXX", "gvk": "openidclient.keycloak.crossplane.io/v1alpha1, Kind=Client"}
2024-10-02T12:54:08Z	DEBUG	provider-keycloak	Reconciling	{"controller": "managed/openidclient.keycloak.crossplane.io/v1alpha1, kind=client", "request": {"name":"XXXXX"}}
2024-10-02T12:54:08Z	DEBUG	provider-keycloak	Calling the inner handler for Update event.	{"gvk": "openidclient.keycloak.crossplane.io/v1alpha1, Kind=Client", "name": "XXXXX", "queueLength": 0}
2024-10-02T12:52:06Z	DEBUG	provider-keycloak	Calling the inner handler for Create event.	{"gvk": "openidclient.keycloak.crossplane.io/v1alpha1, Kind=Client", "name": "XXXXX", "queueLength": 5}
2024-10-02T12:43:07Z	DEBUG	provider-keycloak	External resource is up to date	{"controller": "managed/openidclient.keycloak.crossplane.io/v1alpha1, kind=clientdefaultscopes", "request": {"name":"XXXXX"}, "uid": "a4727f32-e53c-4a2f-8aee-329349046778", "version": "561044047", "external-name": "poc-dsh/41653235-ca40-4412-b629-61d1d0d0ef2a", "requeue-after": "2024-10-02T12:53:07Z"}
2024-10-02T12:43:07Z	DEBUG	provider-keycloak	Reconciling	{"controller": "managed/openidclient.keycloak.crossplane.io/v1alpha1, kind=clientdefaultscopes", "request": {"name":"XXXXX"}}
2024-10-02T12:43:04Z	DEBUG	provider-keycloak	External resource is up to date	{"controller": "managed/openidclient.keycloak.crossplane.io/v1alpha1, kind=client", "request": {"name":"XXXXX"}, "uid": "6be476ee-b482-413b-80e4-920959348371", "version": "561043923", "external-name": "41653235-ca40-4412-b629-61d1d0d0ef2a", "requeue-after": "2024-10-02T12:53:04Z"}
2024-10-02T12:43:04Z	DEBUG	provider-keycloak	Observing the external resource	{"uid": "6be476ee-b482-413b-80e4-920959348371", "name": "XXXXX", "gvk": "openidclient.keycloak.crossplane.io/v1alpha1, Kind=Client"}

[Breee]

Hm, i need to investigate on that in a test cluster.
If restarting nodes triggers this, then i should be able to reproduce it.
But we should also be able to reproduce it then by killing the provider pod right? (You can also start multiple replicas for HA, maybe that changes it) - However, that should not matter because the real state should be stored in ETCD and not in memory.

[TomBillietKlarrio]

I did try quite a lot of different things to reproduce it, but have not been able to find a root cause what exactly triggers it unfortunately. Just killing the keycloak-crossplane (or crossplane) pod does not trigger it. We're not indeed trying to run 2 instances for the keycloak-crossplane provider to see if that helps