add PodReplacementPolicy for Deployments: terminating pods #128546

atiratree · 2024-11-04T19:39:27Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

A new status field .spec.terminatingReplicas is added to Deployments and ReplicaSets to allow tracking of terminating pods.

This is a part 1 - please see special notes.

Which issue(s) this PR fixes:

tracking issue: kubernetes/enhancements#3973

Special notes for your reviewer:

This is a prerequisite for add PodReplacementPolicy for Deployments #123430.
Additional tests also test this field in add PodReplacementPolicy for Deployments #123430 and [WIP] add e2e tests for DeploymentPodReplacementPolicy #126176

Does this PR introduce a user-facing change?

A new status field `.spec.terminatingReplicas` is added to Deployments and ReplicaSets to allow tracking of terminating pods.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-triage-robot · 2024-11-04T20:14:58Z

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

atiratree · 2024-11-05T10:40:02Z

/test pull-kubernetes-e2e-gce

soltysh

/lgtm
/approve
from sig-apps pov

k8s-ci-robot · 2024-11-05T19:08:31Z

LGTM label has been added.

Git tree hash: 55eabbc1e4eb3dd523e255d1fdaa13a8b09cb7c8

soltysh · 2024-11-05T19:09:38Z

/triage accepted
/priority important-longterm
/label api-review

k8s-ci-robot · 2024-11-07T16:01:53Z

New changes are detected. LGTM label has been removed.

staging/src/k8s.io/api/apps/v1/types.go

staging/src/k8s.io/api/testdata/v1.31.0/apps.v1.Deployment.after_roundtrip.pb

pkg/apis/apps/types.go

pkg/apis/apps/validation/validation.go

pkg/controller/deployment/sync.go

staging/src/k8s.io/api/apps/v1beta1/types.go

staging/src/k8s.io/api/apps/v1beta2/types.go

staging/src/k8s.io/api/extensions/v1beta1/types.go

atiratree · 2024-11-07T20:53:58Z

seems like the #123946 flake is still present
/test pull-kubernetes-integration

liggitt · 2024-11-08T16:13:41Z

pkg/controller/deployment/util/deployment_util.go

+	terminatingReplicas := int32(0)
+	for _, rs := range replicaSets {
+		if rs != nil {
+			terminatingReplicas += ptr.Deref(rs.Status.TerminatingReplicas, 0)


Defaulting a nil here to 0 unconditionally is not correct.

If this replicaset has been synced by the replicaset controller at all (status.observedGeneration != 0), and TerminatingReplicas is nil, we cannot reason about the sum of terminatingReplicas across the replicasets and have to return nil from GetTerminatingReplicaCountForReplicaSets

Suggested change

terminatingReplicas += ptr.Deref(rs.Status.TerminatingReplicas, 0)

if rs.Status.ObservedGeneration != 0 && rs.Status.TerminatingReplicas == nil {

// we cannot sum TerminatingReplicas from replicasets that have been synced by the replicaset controller but do not have TerminatingReplicas set

return nil

}

// sum TerminatingReplicas reported by the replicaset controller (or 0 for replicasets not yet synced by the controller)

terminatingReplicas += ptr.Deref(rs.Status.TerminatingReplicas, 0)

This seems reasonable at first glance. I will try to update the code handling for this in both PRs and test it.

I have updated both PRs to handle the pointer fields. It seems to work fine when we check the ObservedGeneration as we will get non nil fields during normal operations.

If we are unable to asses the number of terminating pods, the deployment will stop progressing until we are able to make that decision again. As mentioned in #128546 (comment), this should almost never happen.

liggitt · 2024-11-08T16:14:22Z

pkg/controller/deployment/util/deployment_util.go

@@ -714,6 +714,17 @@ func GetAvailableReplicaCountForReplicaSets(replicaSets []*apps.ReplicaSet) int3
 	return totalAvailableReplicas
 }

+// GetTerminatingReplicaCountForReplicaSets returns the number of terminating pods for all replica sets.
+func GetTerminatingReplicaCountForReplicaSets(replicaSets []*apps.ReplicaSet) int32 {


this has to return *int32 to distinguish between scenarios where we can sum across replicasets and scenarios where we cannot. We can special-case handling of replicasets never synced by the controller at all and treat terminatingReplicas as 0 in that case, see the comment below.

liggitt · 2024-11-08T16:17:29Z

pkg/controller/replicaset/replica_set.go

 	if err != nil {
 		return err
 	}

+	if utilfeature.DefaultFeatureGate.Enabled(features.DeploymentPodReplacementPolicy) {
+		terminatingPods = controller.FilterTerminatingPods(allPods)
+		terminatingPods, err = rsc.claimPods(ctx, rs, selector, terminatingPods)


just counting terminating pods already managed by the replicaset is a much smaller change to reason about than starting to claim terminating pods as well... (it's not adding any new API calls, it's just tweaking the data we're already reporting upward into ReplicaSet and Deployment)

liggitt · 2024-11-08T16:20:37Z

pkg/controller/replicaset/replica_set_utils.go

+				fmt.Sprintf("fullyLabeledReplicas %d->%d, ", rs.Status.FullyLabeledReplicas, newStatus.FullyLabeledReplicas) +
+				fmt.Sprintf("readyReplicas %d->%d, ", rs.Status.ReadyReplicas, newStatus.ReadyReplicas) +
+				fmt.Sprintf("availableReplicas %d->%d, ", rs.Status.AvailableReplicas, newStatus.AvailableReplicas) +
+				fmt.Sprintf("terminatingReplicas %d->%d, ", ptr.Deref(rs.Status.TerminatingReplicas, 0), ptr.Deref(newStatus.TerminatingReplicas, 0)) +


don't mask the difference between nil and 0 here... we should be able to see nil→0 or 5→nil in the log if that's what's happening

it would also be easier to see the diff to conditionally construct the terminatingReplicas bit into a string and just append the string

terminatingReplicasInfo := "" if utilfeature.DefaultFeatureGate.Enabled(features.DeploymentPodReplacementPolicy) { terminatingReplicasInfo = ... } logger.V(4).Info(fmt.Sprintf("Updating status for %v: %s/%s, ", rs.Kind, rs.Namespace, rs.Name) + fmt.Sprintf("replicas %d->%d (need %d), ", rs.Status.Replicas, newStatus.Replicas, *(rs.Spec.Replicas)) + fmt.Sprintf("replicas %d->%d (need %d), ", rs.Status.Replicas, newStatus.Replicas, *(rs.Spec.Replicas)) + fmt.Sprintf("fullyLabeledReplicas %d->%d, ", rs.Status.FullyLabeledReplicas, newStatus.FullyLabeledReplicas) + fmt.Sprintf("fullyLabeledReplicas %d->%d, ", rs.Status.FullyLabeledReplicas, newStatus.FullyLabeledReplicas) + fmt.Sprintf("readyReplicas %d->%d, ", rs.Status.ReadyReplicas, newStatus.ReadyReplicas) + fmt.Sprintf("readyReplicas %d->%d, ", rs.Status.ReadyReplicas, newStatus.ReadyReplicas) + fmt.Sprintf("availableReplicas %d->%d, ", rs.Status.AvailableReplicas, newStatus.AvailableReplicas) + fmt.Sprintf("availableReplicas %d->%d, ", rs.Status.AvailableReplicas, newStatus.AvailableReplicas) + terminatingReplicasInfo + fmt.Sprintf("sequence No: %v->%v", rs.Status.ObservedGeneration, newStatus.ObservedGeneration)) fmt.Sprintf("sequence No: %v->%v", rs.Status.ObservedGeneration, newStatus.ObservedGeneration))

Ack, reporting also a nil transitions now.

- update internal ReplicaSet and Deployment type documentation to match with versioned API - made Replicaset and Deployment type documentation more consistent

k8s-ci-robot · 2024-11-22T15:54:10Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: atiratree, soltysh
Once this PR has been reviewed and has the lgtm label, please ask for approval from liggitt. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

api/OWNERS
pkg/apis/OWNERS
~~pkg/controller/OWNERS~~ [atiratree,soltysh]
~~pkg/features/OWNERS~~ [soltysh]
pkg/generated/openapi/OWNERS
~~pkg/registry/apps/OWNERS~~ [atiratree,soltysh]
staging/src/k8s.io/api/OWNERS
staging/src/k8s.io/client-go/applyconfigurations/OWNERS
~~test/OWNERS~~ [soltysh]
~~test/featuregates_linter/test_data/OWNERS~~ [soltysh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot requested review from aojea and caesarxuchao November 4, 2024 19:39

atiratree force-pushed the pod-replacement-policy-terminating-pods branch from 804f453 to f4a8480 Compare November 4, 2024 20:29

This was referenced Nov 4, 2024

[WIP] add e2e tests for DeploymentPodReplacementPolicy #126176

Open

add PodReplacementPolicy for Deployments #123430

Open

atiratree force-pushed the pod-replacement-policy-terminating-pods branch from f4a8480 to 46560dc Compare November 4, 2024 22:26

soltysh approved these changes Nov 5, 2024

View reviewed changes

k8s-ci-robot assigned soltysh Nov 5, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 5, 2024

k8s-ci-robot removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Nov 5, 2024

atiratree force-pushed the pod-replacement-policy-terminating-pods branch from 46560dc to ca34534 Compare November 7, 2024 16:01

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 7, 2024

k8s-ci-robot requested a review from soltysh November 7, 2024 16:01

liggitt reviewed Nov 7, 2024

View reviewed changes

staging/src/k8s.io/api/apps/v1/types.go Outdated Show resolved Hide resolved

liggitt reviewed Nov 7, 2024

View reviewed changes

staging/src/k8s.io/api/testdata/v1.31.0/apps.v1.Deployment.after_roundtrip.pb Outdated Show resolved Hide resolved

liggitt reviewed Nov 7, 2024

View reviewed changes

liggitt self-assigned this Nov 7, 2024

atiratree force-pushed the pod-replacement-policy-terminating-pods branch 3 times, most recently from b9df89f to 642842f Compare November 7, 2024 19:37

atiratree mentioned this pull request Nov 7, 2024

[Flaking Test] ci-kubernetes-integration-master TestSelectableFields #123946

Open

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 8, 2024

atiratree force-pushed the pod-replacement-policy-terminating-pods branch from 642842f to 444ba7b Compare November 8, 2024 16:19

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 8, 2024

liggitt reviewed Nov 8, 2024

View reviewed changes

atiratree mentioned this pull request Nov 8, 2024

Consider Terminating Pods in Deployments kubernetes/enhancements#3973

Open

4 tasks

atiratree added 5 commits November 22, 2024 15:54

api: add terminatingReplicas field to ReplicaSet and Deployment statuses

27a61b8

- update internal ReplicaSet and Deployment type documentation to match with versioned API - made Replicaset and Deployment type documentation more consistent

add validation, field disablement and tests

013b7e7

update generated files and feature gates

0d17a21

ReplicaSet controller should count terminating pods in the status

a68cf79

Deployment controller should count terminating pods in the status

b0e9ae4

atiratree force-pushed the pod-replacement-policy-terminating-pods branch from 444ba7b to b0e9ae4 Compare November 22, 2024 15:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add PodReplacementPolicy for Deployments: terminating pods #128546

add PodReplacementPolicy for Deployments: terminating pods #128546

atiratree commented Nov 4, 2024

k8s-triage-robot commented Nov 4, 2024

atiratree commented Nov 5, 2024

soltysh left a comment

k8s-ci-robot commented Nov 5, 2024

soltysh commented Nov 5, 2024

k8s-ci-robot commented Nov 7, 2024

atiratree commented Nov 7, 2024

liggitt Nov 8, 2024

atiratree Nov 8, 2024

atiratree Nov 22, 2024

liggitt Nov 8, 2024

liggitt Nov 8, 2024

liggitt Nov 8, 2024

liggitt Nov 8, 2024

atiratree Nov 22, 2024

k8s-ci-robot commented Nov 22, 2024

-			terminatingReplicas += ptr.Deref(rs.Status.TerminatingReplicas, 0)
+			if rs.Status.ObservedGeneration != 0 && rs.Status.TerminatingReplicas == nil {
+				// we cannot sum TerminatingReplicas from replicasets that have been synced by the replicaset controller but do not have TerminatingReplicas set
+				return nil
+			}
+			// sum TerminatingReplicas reported by the replicaset controller (or 0 for replicasets not yet synced by the controller)
+			terminatingReplicas += ptr.Deref(rs.Status.TerminatingReplicas, 0)

add PodReplacementPolicy for Deployments: terminating pods #128546

Are you sure you want to change the base?

add PodReplacementPolicy for Deployments: terminating pods #128546

Conversation

atiratree commented Nov 4, 2024

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-triage-robot commented Nov 4, 2024

atiratree commented Nov 5, 2024

soltysh left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Nov 5, 2024

soltysh commented Nov 5, 2024

k8s-ci-robot commented Nov 7, 2024

atiratree commented Nov 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented Nov 22, 2024