Let's chat about pain points about Kubernetes upgrades with Cluster API

[fabriziopandini]

As per our discussion during the May 3rd office hours, it would be great to collect feedback about the user experience on Kubernetes upgrades with Cluster API.

The discussion has been triggered by the idea of streamlined upgrades presented in this KEP, but we reached the conclusion that we would like to get a better understanding of the community sentiment around this topic before moving forward.

A couple of questions we are interested in (but any kind of feedback is more than welcome!):

How often do you upgrade the Kubernetes version on your clusters? Do you use CAPI to upgrade or something else? If something else, how do you upgrade clusters? If you use CAPI, what are the steps you follow? Do you use ClusterClass? Would you be interested in something like LTS support and/or skipping version upgrades for worker nodes? What are currently your biggest pain points related to upgrading?

[tobiasgiese]

I am writing on behalf of the Kubernetes platform team from Mercedes Benz.

How often do you upgrade the Kubernetes version on your clusters?

Usually we try to have the most recent Kubernetes version minus one. Currently we're upgrading to Kubernetes v1.26. This means that we try to upgrade our Kubernetes clusters three times a year (nearly 1000 clusters in 3 weeks 😅 ).

Do you use CAPI to upgrade or something else?

We are using CAPI to upgrade, yes. For sure we have a custom pipeline to deploy the CAPI resources.

If you use CAPI, what are the steps you follow?

• We have multiple nightly build clusters to test the update.
• During the update progress we first upgrade the Cluster infrastructure to have these changes in place
• After the Cluster infrastructure has been reconciled successfully, we upgrade the KubeadmControlPlane
• And at the last phase and after a successful upgrade of the KCP we start the MachineDeployment upgrade

Do you use ClusterClass?

Currently not. We're planning to use ClusterClass, but only if it's possible to migrate existing clusters into ClusterClass.

Would you be interested in something like LTS support and/or skipping version upgrades for worker nodes?

We are already able to kind of skip a Kubernetes version.
How do we do this?

• As example we take the current Kubernetes version v1.24 and the desired version v1.26
• First we update the KCP to v1.25 to not violate the Kubernetes skew policy
• After a successful update of the KCP we update to the desired Kubernetes v1.26
• As worker nodes can update to the Kubernetes version after the next we are able to update the MD directly to v1.26

What are currently your biggest pain points related to upgrading?

Tbh, we don't have any pain points with upgrading our clusters, except for the CC migration problem.

Btw, I just published a repository with a proof of concept how we have migrated our legacy Kubernetes clusters into Cluster API 🙂
https://github.com/tobiasgiese/cluster-api-migration

_{Tobias Giese tobias.giese@mercedes-benz.com, Mercedes-Benz Tech Innovation GmbH, legal info/Impressum}

[tobiasgiese]

One addition (thanks for the hint @seanschneeweiss): we have issues with the reachability of the control plane load balancer during rolling upgrade.

To fix this issue we have in our downstream forks some changes in the cluster-api and cluster-api-provider-openstack controllers.
We create the load balancer listeners of the new control planes with state down and store the information in conditions. As soon as the node is provisioned -- and thus the kube-apiserver is reachable -- we set the state to up and update the conditions. In this case the kube-apiserver is reachable all the time.
Additionally, with the new etcd learner mode feature (see kubernetes/kubeadm#1793) which is available since Kubernetes v1.27 there is not one kube-apiserver request missing.

[mdbooth]

@tobiasgiese Do we need to get the API LB fix merged in CAPO, or is it specific to your environment?

[tobiasgiese]

@mdbooth the issue is that the LB listener fix is not just in CAPO, but also in CAPI and CAPI Runtime SDK (in our case a custom controller as the Runtime SDK was not available at the time we have fixed this issue). Further, the load balancer provider issue should fix it and thus I think this change won't be implemented in the current CAPI code.

But maybe @sbueringer or @chrischdi can say more about this, both might still know our fix 🙂

[chrischdi]

I can give context on that:

• The healthcheck we did use (or @tobiasgiese uses) is TCP based (due to limitations on IaaS side).
• This means as soon as the port is reachable, the load balancer will forward traffic to the load balancer member

This creates some not perfect scenarios, e.g.:

• the kube-apiserver starts and opens up its port, while not being ready (e.g. it depends on etcd container which is not yet started, or apiserver itself starting up)
• lb healthcheck doing its tcp probe(s) and starts forwarding traffic
• so requests will get forwarded but result in errors

What we did at that time is: hook into the logic (and I think this included customizing CAPO) to:

• when joining a KCP machine: create the load balancer members in disabled mode
• after we know that the KCP machine is fine to reach traffic: enable the load balancer members pointing to this machine

[chrischdi]

How often do you upgrade the Kubernetes version on your clusters?

Maybe to add here: to catch-up to the one minor version behind upstream state, there were times where a minor version was skipped (either completely by hacking around in CAPI or whatever we did use for provisioning) or by upgrading KCP two times (v1.x -> v1.y -> v1.z) and workers directly to the target version afterwards (v1.x -> v1.z).

Edit: @tobiasgiese feel free to correct me if I'm wrong 👍

[rudimk]

At Porter, we manage more than a hundred K8s clusters - most of them are EKS, with about 10% comprising GKE and AKS clusters. Much of what I'm writing below pertains to how we manage cluster upgrades without CAPI at the moment, and how we see CAPI making our upgrade process better. This also assumes we're able to migrate all our managed clusters(EKS/GKE/AKS) into CAPA/CAPG/CAPZ over the next couple months.

How often do you upgrade the Kubernetes version on your clusters?

This has changed over time - until December 2022, we were only upgrading clusters when AWS would announce EOS for a particular version. January 2023 is when we elected to ensure all clusters are as close to the latest supported version as possible, by leapfrogging over versions. We're now looking at settling into a six-month upgrade cycle, where each cycle involves upgrading all clusters by atleast 2 versions, to the latest supported version.

Do you use CAPI to upgrade or something else? If something else, how do you upgrade clusters?

We don't use CAPI for upgrades just yet, but we hope to change that over the next couple months. At the moment, upgrades are carried out by writing custom Golang code which is responsible for upgrading EKS control planes over multiple versions, as well as updating launch templates for self-managed nodegroups and then running instance refresh ops. While all our clusters were creating using Terraform, the drift between TF state and the actual infrastructure is too large for us to risk Terraform-based upgrades. I've personally also experimented with running upgrades via Crossplane - the experiment shows promise, although the ability to run reliable instance refresh ops is not possible with Crossplane.

If you use CAPI, what are the steps you follow?

If we're able to bring in existing EKS clusters into CAPA, then based on the docs here, we imagine the process would look a lot like this:

1. Upgrade charts/components like Nginx Ingress, CA and so on, as well as user-specific chart templates to account for API version changes.
2. Bump up the cluster version in the AWSManagedControlPlane spec to the desired version.
3. Observe CAPA upgrade the EKS control plane across multiple versions, and then subsequently upgrade the managed nodegroups' launch templates to use newer AMIs.

The actual steps will probably be automated - script chart/template/API object version upgrades ,then call the CAPI manager cluster to update the version in the AWSManagedControlPlane object. In effect, we foresee version upgrades being a far more automated process than how we do it now.

Do you use ClusterClass?

Not just yet, but I like the concept of a ClusterClass - while we have a set, defined cluster architecture, formalising it and actually using that formalisation in order to reconcile topology at scale sounds like an attractive proposition.

Would you be interested in something like LTS support and/or skipping version upgrades for worker nodes?

Since we primarily operate on top of managed K8s offerings, the idea of a LTS release doesn't truly exist for us since cloud providers tend to treat each version the same. What does make a lot more sense for us - staying ahead by upgrading to the latest versions available. That's currently possible with our in-house upgrade process, and it seems like CAPA allows for jumping over versions.

What are currently your biggest pain points related to upgrading?

1. Automating the entire process. We've made some strides there, and CAPA/CAPG/CAPZ will end up being the culmination of this whole exercise.
2. It would be very cool if we could also automate Helm chart updates for system components like Nginx and so on as well as user-defined charts. But I don't see that as something in CAPI's ambit anyway. Using a Helm Controller to manage Helm charts doesn't make much sense because we want to allow for charts to have different configuration values across clusters, and running continuous reconciliation against charts seems like a bad idea.

[fabriziopandini]

Thank you all folks for the super valuable insight!
I think that ultimately CAPI should be aligned on Kubernetes, and thus ideally we should start supporting upgrades according to the process described in kubernetes/enhancements#3936.

This will give a new valuable tool to CAPI users, but ultimately, let everyone choose if to use it or not