How to deal with removal of v1beta1 CRD removals in Kubernetes 1.22

[camilamacedo86]

📢 Kubernetes 1.22 API removals

What is the big deal? Kubernetes 1.22 removes some long deprecated APIs, among the CustomResourceDefinitions in the version v1beta1. Consequently will not allow using these APIs anymore. What this means for contributors:

• If your operator uses the removed APIs, such as v1beta1 CRDs, it will not work on 1.22+:

the incompatible versions will appear in the OLM console (by consuming quay.io/operatorhubio/catalog)and users will be able to install the operator versions distributed on it. Then, it will fail in ways that may not be obvious to the user.

• The users of your Operator will be able to check events raised in the previous K8S version warning that the Operator installed uses deprecated APIs. More information on this is available in this blog post.
• To make your operator compatible with Kubernetes 1.22 and vendors that use this version such as OpenShift 4.9, usually you need to simply migrate your CRDs to version v1 which is the most commonly used. However, following detailed guidance over how to upgrade your projects and ensure that they’re workable in the latest versions.
• If your operator uses webhooks, OLM will handle the creation / migration of your webhooks to version v1, however, please ensure that you also update your source code.
• Let your Operator users know that your upgrade versions do not work on K8s versions < 1.16 by setting minKubeVersion: 1.16.0 in the CSV.

IMPORTANT: If you want to publish versions that are not compatible with 1.22+ then, you must set the annotation operatorhub.io/ui-metadata-max-k8s-version: 1.21 to let your users informed.

💡 How to make your project compatible with k8s 1.22?

Due to the number of options available to build Operators, it is hard to provide direct guidance on updating your operator to support Kubernetes 1.22. Recent versions of the OperatorSDK greater than 1.0.0 and Kubebuilder greater than 3.0.0 scaffold your project with the latest versions of these APIs (all that is generated by tools only). See the guides to upgrade your projects with OperatorSDK Golang, Ansible, Helm or the Kubebuilder one. For APIs other than the ones mentioned above, you will have to check your code for usage of removed API versions and upgrade to newer APIs. The details of this depend on your codebase.

Following a collection of how to's and tips to help you find if your project is using these removed APIs, how to update them, and how to test your changes. We hope that helps you out.

🔥 Checking the usage

• By looking in the code source:

You can grep your project's code to looking for all usage of these APIs. All APIs removed on k8s 1.22 / OCP 4.9 can be found in Removed APIs by release - 1.22, e.g:

$ grep -rni apiextensions.k8s.io/v1beta1 *
bundle/manifests/cache.example.com_memcacheds.yaml:1:apiVersion: apiextensions.k8s.io/v1beta1
config/crd/patches/cainjection_in_memcacheds.yaml:3:apiVersion: apiextensions.k8s.io/v1beta1
config/crd/patches/webhook_in_memcacheds.yaml:3:apiVersion: apiextensions.k8s.io/v1beta1
config/crd/bases/cache.example.com_memcacheds.yaml:3:apiVersion: apiextensions.k8s.io/v1beta1

• By using OperatorSDK(latest versions only) to validate the manifest in your bundle:

$  operator-sdk bundle validate <your bundle path> --select-optional suite=operatorframework --optional-values=k8s-version=1.22

$  operator-sdk bundle validate ./bundle --select-optional suite=operatorframework --optional-values=k8s-version=1.22
... 
ERRO[0000] Error: Value memcached-operator.v0.0.1: this bundle uses APIs that were deprecated and removed in v1.22. More info: https://kubernetes.io/docs/reference/using-api/deprecation-guide/#v1-22. Migrate the API(s) for CRD: (["memcacheds.cache.example.com"]) 

NOTE This check is only able to look at the manifests shipped inside of the bundle. Suppose your operator uses the removed APIs, but they are not shipped on the bundle, OR they depend on a project that uses them and will not work on 1.22+. In that case, unfortunately, we can not identify these scenarios. You will need to determine if this is your case by code review and QE testing on the clusters.

• By checking the events/alerts

If you deploy your Operator on the cluster and perform regression tests, the APIs will raise alerts/events to notify you about the usage of APIs that will be removed in the next versions:

$ kubectl get events

More information on this is available in this blog post.

NOTE: Events are also recorded in the Kubernetes API audit logs. Once the logs have been downloaded, deprecation events can easily be searched with:

$ grep "removed-release.*1.22" audit*.log

🔨 Updating your project

Now, you need to ensure that your project will no longer use these removed APIs on 1.22 and start to use the latest version(s).

For the k8s manifests, you need to re-generate them using the new versions. However, you might be importing them into your codebase. Then, you ought to replace the imports, e.g.: replace theapiused/v1beta1 with its latest version theapiused/v1 as well.

In the most common scenarios, projects are affected by using the k8s API apiextensions.k8s.io/v1beta1 for CRDs. In this way, follows an example and tips to migrate your CRDs from v1beta1 to (v1 format).

Tips and example(s) over how to migrate only CRDs

If you are using OperatorSDK CLI tool version >= v0.18.x < 1.0.0 you can:

$ operator-sdk generate crds --crd-version=v1
INFO[0000] Running CRD generator.                       
INFO[0000] CRD generation complete.

If you have been generating the manifests with controller-gen or manually then, you can use its upper version >= v0.4.1 and re-generate them:

$ controller-gen crd:trivialVersions=true,preserveUnknownFields=false rbac:roleName=manager-role  paths="./..." output:crd:artifacts:config=<add here the output-path>

OR

$ controller-gen crd:crdVersions=v1,preserveUnknownFields=false rbac:roleName=manager-role paths="./..." output:crd:artifacts:config=<add here the output-path>

Here’s a full example:

$ go get sigs.k8s.io/controller-tools/cmd/controller-gen@v0.4.1
$ controller-gen --version
Version: v0.4.1
$ controller-gen crd:trivialVersions=true,preserveUnknownFields=false rbac:roleName=manager-role paths="./..." output:crd:artifacts:config=deploy/crds/

NOTE To understand the above option preserveUnknownFields=false see this k8s doc.

See that in a live cluster, you can invoke Kubernetes’ conversion functionality by applying a crd.v1beta1 and then kubectl get a crd.v1 to view its converted format.

Now, you have your CRD using the latest version to be shipped on the new bundle version.

NOTE: You are probably not using OperatorSDK CLI >= 1.0.0 because projects migrated to versions >= 1.0.0 are scaffolded by default using the latest K8S API versions. However, by using the latest scaffold you can create and update your bundle via $ make bundle. Also, see FAQ-Can I customize the projects initialized with operator-sdk?.

If your project uses Webhooks:

Add the makers sideEffects=none and admissionReviewVersions to your webhook (e.g. memcached-operator/api/v1alpha1/memcached_webhook.go):

Example:

//+kubebuilder:webhook:path=/mutate-cache-example-com-v1alpha1-memcached,mutating=true,failurePolicy=fail,sideEffects=None,groups=cache.example.com,resources=memcacheds,verbs=create;update,versions=v1alpha1,name=mmemcached.kb.io,admissionReviewVersions={v1,v1beta1}

Then, run the command:

 controller-gen crd:trivialVersions=true,preserveUnknownFields=false rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=deploy/crds/

👌 Testing your changes

After you have packaged a new version, test out your Operator on K8s cluster 1.22+

By using OperatorSDK (latest versions)

---

Testing bundles

To test and deploy the Operator with OLM, make sure the bundle image is present in a registry, operator-sdk run bundle can create a pod to serve that bundle to OLM via a Subscription, along with other OLM objects, ephemerally, e.g.:

$ operator-sdk run bundle <some-registry>/memcached-operator-bundle:v0.0.1

Note: If testing a bundle whose image will be hosted in a registry that is private and/or has a custom CA, these configuration steps must be complete.

Upgrading a bundle to a newer version

You can also use the operator-sdk run bundle-upgrade command with your newer version of the bundle image to upgrade an existing operator bundle deployed on the cluster. For example, to upgrade the previously deployed memcached-operator bundle from version 0.0.1 to 0.0.2, run the following:

$ operator-sdk run bundle-upgrade <some-registry>/memcached-operator-bundle:v0.0.2

---

Upgrading a bundle that was installed traditionally using OLM

If you want to test your Operator bundle upgrade even if it was originally deployed using OLM without using the run bundle command.

You can create a CatalogSource and deploy an Operator bundle traditionally using OLM and then upgrade the Operator bundle to a newer version. See more details.

Let your Operator users know that your upgrade versions do not work on K8s versions < 1.16

See that v1 API for CRD version was introduced in 1.16 K8s version. That means, that any bundle/package version upgraded to work on 1.22+ will no longer work in the k8s version < 1.16. In this way, please let your Operators users know it by updating the CSV spec with:

minKubeVersion: 1.16.0

[camilamacedo86]

All CSVs which will be updated automatically can be checked in: https://github.com/k8s-operatorhub/community-operators/blob/e51b9e2e3a8895fd056e6c922a6bbb5b961b6e4e/scripts/add-annotation/output.yml

@mvalarh @J0zi