test(compatibility-versions): adds test script and periodic prow job for testing kubernetes feature compatibility versions #33534

aaron-prindle · 2024-09-26T04:36:18Z

This PR is a part of KEP-4330: Compatibility Versions in Kubernetes and adds the necessary components for a kubernetes compatibility versions test e2e test. The goal of this E2E test currently is to have nightly test run that uses compatibility versions (via the --emulated-versions= flag) to:

build the relevant kubernetes binaries @ HEAD (kube-apiserver, kube-controller-manager, and kube-scheduler)
set the --emulated-versions=n-1..3 (where n is the k8s version @ HEAD) for the relevant k8s components and create a cluster
pull down the n-1..3 release branch of k8s and run the n-1..3 E2E tests against the cluster using the k8s binaries @ HEAD w/ the –emulated-versions=n-1..3

This way we will have a smoke test around the integration and specifically able to validate that compatibility versions API enablement is as expected. Feature gate enablement correctness testing is WIP and will be part of a future PR.

NOTE: compatibility versions is only support since k8s release v1.31 and this PR reflects this opting for 1.31 hardcoding of the EMULATED_VERSION parameter and not doing any validations on that value in this first PR related to E2E testing this.

The files here are adapted from:

Prow Config Questions:

Currently this is an early version of the test that I am hoping to iterate on. Should I have test-grid annotations on this?

krzyzacy

I'll leave the bash part to @aojea ... or, can this be written in go?

krzyzacy · 2024-09-26T06:06:31Z

config/jobs/kubernetes/sig-testing/periodic-kubernetes-compatibility-version-e2e.yaml

+periodics:
+- interval: 24h
+  cluster: k8s-infra-prow-build
+  name: pull-kubernetes-e2e-kind-emulated-version


name it ci-kubernetes-e2e-* for periodic jobs, pull-kubernetes-e2e are for presubmits in general

Done, renamed to:

name: ci-kubernetes-e2e-kind-compatibility-versions

krzyzacy · 2024-09-26T06:07:13Z

config/jobs/kubernetes/sig-testing/periodic-kubernetes-compatibility-version-e2e.yaml

+    testgrid-dashboards: sig-testing-kind
+    testgrid-tab-name: compatibility-version-test
+    description: Uses kubetest & kind to run e2e tests from older (n-1..3) kubernetes releases against a latest kubernetes master components w/ --emulated-version=<n-1..3> set.
+    testgrid-alert-email: aprindle@google.com


can we route to a rotation? (maybe ok for now, add a TODO?)

Good point, added a TODO comment for this now

In some places we have a SIG alerts list, I don't know if api machinery or architecture have one

it can route to multiple emails anyhow, so if you're personally working on this and want to be alerted to start that's common and fine

I am personally working on this and would like to be alerted to start so I think just having aprindle@google.com there for now should be ok in that case.

aaron-prindle · 2024-09-27T17:00:55Z

/cc @BenTheElder

BenTheElder · 2024-10-02T22:32:41Z

config/jobs/kubernetes/sig-testing/compatibility-versions-e2e.yaml

+      - curl -sSL https://kind.sigs.k8s.io/dl/latest/linux-amd64.tgz | tar xvfz - -C "${PATH%%:*}/" && ${GOPATH}/src/k8s.io/test-infra/experiment/compatibility-versions/e2e-k8s-compatibility-versions.sh
+      env:
+      - name: EMULATED_VERSION
+        value: "1.31"  # TODO(aaron-prindle) FIXME - hardcoded for now


we might be able to handle this with the config-forker, it only needs to change when we branch the release anyhow

https://github.com/kubernetes/test-infra/tree/master/releng/config-forker <-- run when we introduce a new release branch

Ah, I was not aware of the config-forker. I have created #33573 to track properly parameterizing the branch/branches to test against. Currently this won't be an issue as only testing 1.31 is all that is needed for the next ~2 months and I am hoping to get something in to kick the tires on the test. Lmk if that sg, thanks!

Yeah the forker is pretty niche, I think it may be cleaner than alternatives to solve this long term though. Tracking in an issue SGTM

BenTheElder · 2024-10-02T22:34:44Z

config/jobs/kubernetes/sig-testing/compatibility-versions-e2e.yaml

+      - wrapper.sh
+      - bash
+      - -c
+      - curl -sSL https://kind.sigs.k8s.io/dl/latest/linux-amd64.tgz | tar xvfz - -C "${PATH%%:*}/" && ${GOPATH}/src/k8s.io/test-infra/experiment/compatibility-versions/e2e-k8s-compatibility-versions.sh


Suggested change

- curl -sSL https://kind.sigs.k8s.io/dl/latest/linux-amd64.tgz | tar xvfz - -C "${PATH%%:*}/" && ${GOPATH}/src/k8s.io/test-infra/experiment/compatibility-versions/e2e-k8s-compatibility-versions.sh

- curl -sSL https://kind.sigs.k8s.io/dl/latest/linux-amd64.tgz | tar xvfz - -C "${PATH%%:*}/" && "$(go env GOPATH)"/src/k8s.io/test-infra/experiment/compatibility-versions/e2e-k8s-compatibility-versions.sh

or simpler with precedent:

Suggested change

- curl -sSL https://kind.sigs.k8s.io/dl/latest/linux-amd64.tgz | tar xvfz - -C "${PATH%%:*}/" && ${GOPATH}/src/k8s.io/test-infra/experiment/compatibility-versions/e2e-k8s-compatibility-versions.sh

- curl -sSL https://kind.sigs.k8s.io/dl/latest/linux-amd64.tgz | tar xvfz - -C "${PATH%%:*}/" && ./../test-infra/experiment/compatibility-versions/e2e-k8s-compatibility-versions.sh

Did you have a preference from the above options here? I used ${GOPATH} initially as it was in config/jobs/containerd/containerd/containerd-presubmit-jobs.yaml - https://github.com/kubernetes/test-infra/blob/master/config/jobs/containerd/containerd/containerd-presubmit-jobs.yaml#L261. I've changed this to use the first recommendation "$(go env GOPATH)" for now

Sorry for the late reply: I prefer the second because locally the user may not have a GOPATH these days but they're very likely to have these clones side by side (or they don't have them at all, and this won't work regardless) but it doesn't matter much.

BenTheElder

cc @aojea

LGTM, I think, any comments antonio?

config/jobs/kubernetes/sig-testing/compatibility-versions-e2e.yaml

aojea · 2024-10-04T16:50:46Z

experiment/compatibility-versions/e2e-k8s-compatibility-versions.sh

+    feature_gates='{"AllAlpha":false,"AllBeta":false}'
+    runtime_config='{"api/alpha":"false", "api/beta":"false"}'


do you want to run with all betas enable?

Good point, thanks for catching this. I would like to run with all betas enable. The script there is essentially https://github.com/kubernetes-sigs/kind/blob/main/hack/ci/e2e-k8s.sh with the only addition being the --emulated-version flag plumbed through. ~~I have updated the script directly to now have true for both Beta feature and API enablement~~:

feature_gates='{"AllAlpha":false,"AllBeta":true}' runtime_config='{"api/alpha":"false", "api/beta":"true"}'

EDIT: See below, I have changed the approach of how to do this

I realize now the script is parameterized to allow for the following. I undid my direct script changes (see above comment) and updated the prow job to pass in the params to the script to enable betas:

- name: FEATURE_GATES value: '{"AllBeta":true}' - name: RUNTIME_CONFIG value: '{"api/beta":"true", "api/ga":"true"}'

Quick question: Why are we running with Beta APIs and feature gates enabled? Shouldn't we use the default feature gate flag at a particular version?

…for testing kubernetes feature compatibility versions

aojea · 2024-10-07T17:14:59Z

/lgtm

Think about having a presubmit optional and disabled by default, use to be helpfor for debugging or reviewing code related to your feature

aojea · 2024-10-07T18:23:23Z

/approve

k8s-ci-robot · 2024-10-07T18:23:30Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aaron-prindle, aojea

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~config/OWNERS~~ [aojea]
~~experiment/OWNERS~~ [aojea]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2024-10-07T18:36:26Z

@aaron-prindle: Updated the job-config configmap in namespace default at cluster test-infra-trusted using the following files:

key compatibility-versions-e2e.yaml using file config/jobs/kubernetes/sig-testing/compatibility-versions-e2e.yaml

In response to this:

This PR is a part of KEP-4330: Compatibility Versions in Kubernetes and adds the necessary components for a kubernetes compatibility versions test e2e test. The goal of this E2E test currently is to have nightly test run that uses compatibility versions (via the --emulated-versions= flag) to:

build the relevant kubernetes binaries @ HEAD (kube-apiserver, kube-controller-manager, and kube-scheduler)

set the --emulated-versions=n-1..3 (where n is the k8s version @ HEAD) for the relevant k8s components and create a cluster

pull down the n-1..3 release branch of k8s and run the n-1..3 E2E tests against the cluster using the k8s binaries @ HEAD w/ the –emulated-versions=n-1..3

This way we will have a smoke test around the integration and specifically able to validate that compatibility versions API enablement is as expected. Feature gate enablement correctness testing is WIP and will be part of a future PR.

NOTE: compatibility versions is only support since k8s release v1.31 and this PR reflects this opting for 1.31 hardcoding of the EMULATED_VERSION parameter and not doing any validations on that value in this first PR related to E2E testing this.

The files here are adapted from:

kubernetes-sigs/kind - hack/ci/e2e-k8s.sh

kubernetes test-infra - config/jobs/kubernetes/sig-testing/kubernetes-kind.yaml#ci-kubernetes-e2e-kind-alpha-features

Prow Config Questions:

Currently this is an early version of the test that I am hoping to iterate on. Should I have test-grid annotations on this?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

BenTheElder · 2024-10-07T19:28:31Z

Think about having a presubmit optional and disabled by default, use to be helpfor for debugging or reviewing code related to your feature

we're going with iterating towards a release-informing => blocking job for now.

BenTheElder · 2024-10-07T19:29:24Z

config/jobs/kubernetes/sig-testing/compatibility-versions-e2e.yaml

+  decorate: true
+  decoration_config:
+    timeout: 60m
+  extra_refs:


oh, because this is interval based, we need to explicitly add cloning kubernetes as well (first which will make it default for being the working directory)

(or switch to have the script use kubernetes from a CI build instead of building itself)

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Sep 26, 2024

k8s-ci-robot requested a review from aojea September 26, 2024 04:36

k8s-ci-robot added the area/config Issues or PRs related to code in /config label Sep 26, 2024

k8s-ci-robot requested a review from dims September 26, 2024 04:36

k8s-ci-robot added area/jobs sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 26, 2024

aaron-prindle force-pushed the compatibility-versions-test branch 3 times, most recently from ef74464 to b858224 Compare September 26, 2024 05:22

krzyzacy reviewed Sep 26, 2024

View reviewed changes

aaron-prindle force-pushed the compatibility-versions-test branch 4 times, most recently from cd19a98 to fb0fdcc Compare September 26, 2024 06:13

k8s-ci-robot requested a review from BenTheElder September 27, 2024 17:00

BenTheElder reviewed Oct 2, 2024

View reviewed changes

aaron-prindle force-pushed the compatibility-versions-test branch from fb0fdcc to 4b8fb5c Compare October 3, 2024 18:17

aaron-prindle mentioned this pull request Oct 3, 2024

Removed hardcoded 1.31 version for compatibility-versions-e2e.yaml test and properly parameterize the n-1..3 release branch #33573

Open

aojea reviewed Oct 4, 2024

View reviewed changes

config/jobs/kubernetes/sig-testing/compatibility-versions-e2e.yaml Outdated Show resolved Hide resolved

aojea reviewed Oct 4, 2024

View reviewed changes

config/jobs/kubernetes/sig-testing/compatibility-versions-e2e.yaml Outdated Show resolved Hide resolved

aaron-prindle force-pushed the compatibility-versions-test branch from 4b8fb5c to cfa2e60 Compare October 4, 2024 16:47

aojea requested changes Oct 4, 2024

View reviewed changes

config/jobs/kubernetes/sig-testing/compatibility-versions-e2e.yaml Outdated Show resolved Hide resolved

k8s-ci-robot assigned aojea Oct 4, 2024

aojea reviewed Oct 4, 2024

View reviewed changes

aaron-prindle force-pushed the compatibility-versions-test branch from 3303cc2 to b27ac00 Compare October 4, 2024 16:53

aaron-prindle force-pushed the compatibility-versions-test branch 2 times, most recently from 21c3949 to abdbf78 Compare October 4, 2024 17:50

test(compatibility-versions): adds test script and periodic prow job …

e04333e

…for testing kubernetes feature compatibility versions

aaron-prindle force-pushed the compatibility-versions-test branch from abdbf78 to e04333e Compare October 4, 2024 17:51

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 7, 2024

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 7, 2024

k8s-ci-robot merged commit f3ec85a into kubernetes:master Oct 7, 2024
7 checks passed

aaron-prindle mentioned this pull request Oct 7, 2024

test ci-kubernetes-e2e-kind-compatibility-versions failing w/ could not find Kubernetes source under current working directory or GOPATH #33591

Closed

BenTheElder reviewed Oct 7, 2024

View reviewed changes

aaron-prindle mentioned this pull request Oct 7, 2024

fix: resolve issue with ci-kubernetes-e2e-kind-compatibility-versions where expected kubernetes was not present + change config to not use GOPATH #33592

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test(compatibility-versions): adds test script and periodic prow job for testing kubernetes feature compatibility versions #33534

test(compatibility-versions): adds test script and periodic prow job for testing kubernetes feature compatibility versions #33534

aaron-prindle commented Sep 26, 2024 •

edited

Loading

krzyzacy left a comment

krzyzacy Sep 26, 2024

aaron-prindle Sep 26, 2024

krzyzacy Sep 26, 2024

aaron-prindle Sep 26, 2024

BenTheElder Oct 2, 2024

aaron-prindle Oct 2, 2024

aaron-prindle commented Sep 27, 2024

BenTheElder Oct 2, 2024

aaron-prindle Oct 3, 2024

BenTheElder Oct 7, 2024

BenTheElder Oct 2, 2024 •

edited

Loading

aaron-prindle Oct 3, 2024 •

edited

Loading

BenTheElder Oct 7, 2024

BenTheElder left a comment

aojea Oct 4, 2024

aaron-prindle Oct 4, 2024 •

edited

Loading

aaron-prindle Oct 4, 2024 •

edited

Loading

Jefftree Oct 10, 2024

aojea commented Oct 7, 2024

aojea commented Oct 7, 2024

k8s-ci-robot commented Oct 7, 2024

k8s-ci-robot commented Oct 7, 2024

BenTheElder commented Oct 7, 2024

BenTheElder Oct 7, 2024 •

edited

Loading

	- curl -sSL https://kind.sigs.k8s.io/dl/latest/linux-amd64.tgz \| tar xvfz - -C "${PATH%%:*}/" && ${GOPATH}/src/k8s.io/test-infra/experiment/compatibility-versions/e2e-k8s-compatibility-versions.sh
	- curl -sSL https://kind.sigs.k8s.io/dl/latest/linux-amd64.tgz \| tar xvfz - -C "${PATH%%:*}/" && "$(go env GOPATH)"/src/k8s.io/test-infra/experiment/compatibility-versions/e2e-k8s-compatibility-versions.sh

		feature_gates='{"AllAlpha":false,"AllBeta":false}'
		runtime_config='{"api/alpha":"false", "api/beta":"false"}'

test(compatibility-versions): adds test script and periodic prow job for testing kubernetes feature compatibility versions #33534

test(compatibility-versions): adds test script and periodic prow job for testing kubernetes feature compatibility versions #33534

Conversation

aaron-prindle commented Sep 26, 2024 • edited Loading

krzyzacy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aaron-prindle commented Sep 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenTheElder Oct 2, 2024 • edited Loading

Choose a reason for hiding this comment

aaron-prindle Oct 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenTheElder left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aaron-prindle Oct 4, 2024 • edited Loading

Choose a reason for hiding this comment

aaron-prindle Oct 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aojea commented Oct 7, 2024

aojea commented Oct 7, 2024

k8s-ci-robot commented Oct 7, 2024

k8s-ci-robot commented Oct 7, 2024

BenTheElder commented Oct 7, 2024

BenTheElder Oct 7, 2024 • edited Loading

Choose a reason for hiding this comment

aaron-prindle commented Sep 26, 2024 •

edited

Loading

BenTheElder Oct 2, 2024 •

edited

Loading

aaron-prindle Oct 3, 2024 •

edited

Loading

aaron-prindle Oct 4, 2024 •

edited

Loading

aaron-prindle Oct 4, 2024 •

edited

Loading

BenTheElder Oct 7, 2024 •

edited

Loading