Unwanted updates happening on AKS managed kubernetes loadbalancer

[nishant221]

/kind bug

[Before submitting an issue, have you checked the Troubleshooting Guide self-managed & managed?]

What steps did you take and what happened:
[A clear and concise description of what the bug is.]

We created an AKS cluster via AzureASOManagedCluster APIs of CAPZ, and while observing activity logs of Azure , we are seeing some unwanted updates in Azure managed "kubernetes" loadbalancer (screenshots below)

After some time the above change gets auto-reverted as below:

What did you expect to happen:
These unwanted updates should not be triggered.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• cluster-api-provider-azure version: v1.16.1
• Kubernetes version: (use kubectl version): v1.28.9
• OS (e.g. from /etc/os-release):

[nojnhuh]

It looks to me that somehow two different API versions are being shown, and both are representing the same underlying resource. That noHealthyBackendsBehavior field seems to only have been introduced in the 2024-01-01 API version, so it would make sense that a 2022-09-01 representation doesn't include that field: https://github.com/search?q=repo%3AAzure%2Fazure-rest-api-specs%20noHealthyBackendsBehavior&type=code

And CAPZ doesn't touch the kubernetes load balancer directly since it's managed by AKS, so if anything this seems like a general AKS bug. I think it's more likely though to be a quirk in how the activity log is rendering the resource's change history.

/remove-kind bug
/kind support

[dtzar]

@nishant221 - have you tried enabling CAPZ preview API version? That would get ASO to utilize 2024-04-02-preview version of API. If this works, we likely need to update the stable version of the API from ASO.

[dtzar]

See Azure/azure-service-operator#4327

[dtzar]

@nishant221 - you can change to the preview API version for AzureASOManagedControlPlane via manually editing this value to what is available in ASO.

[nishant221]

@dtzar We are already specifying preview API version for ManagedCluster as follows:

but when I see on the management ManagedCluster cluster, it is getting created with a different API version (Assuming this is getting mutated by ASO)

[nojnhuh]

API version is not an inherent property of Kubernetes resources. The same resource can be represented by several different API versions: https://kubernetes.io/docs/concepts/overview/kubernetes-api/#api-groups-and-versioning

ASO's versioning scheme isn't compatible with Kubernetes's assumptions about API versions w.r.t. how Kubernetes will pick a default version, so you'll usually end up seeing an older version than you might have used to create the resource if you do a plain kubectl get managedcluster vs. specifying the version you want like kubectl get managedclusters.v1api20240402preview.containerservice.azure.com. Both would return the exact same set of resources though. See Azure/azure-service-operator#4147 for more context.

Overall, I still don't see where any resources are unexpectedly being mutated by either CAPZ, ASO, or AKS.