KEP 1645: add labels and annotations export

[MrFreezeex]

• One-line PR description: add labels and annotations export

• Issue link: Multi-Cluster Services API #1645

• Other comments: Add new fields in the ServiceExport resources to sync labels and annotations. This approach is inspired by the GatewayAPI (https://gateway-api.sigs.k8s.io/geps/gep-1762/#labels-and-annotations) one to have dedicated fields for that and there's some explanation inline in the KEP why we might favor that.

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: MrFreezeex
Once this PR has been reviewed and has the lgtm label, please assign skitt for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• keps/sig-multicluster/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tpantelis]

My understanding of the motivation for this is to provide a way to specify which labels/annotations should be synced/transferred from the exported Service to the derived Service. Therefore, shouldn't these fields specify only the names of the corresponding labels/annotations on the exported Service and the implementation would extract the values from exported Service instance (and also handle subsequent updates to the exported Service's values)?

[MrFreezeex]

You mean like that we would have some filters instead of the actual values? If so yes that might be one alternative approach that would work too, let's see what others think!

[skitt]

If the ServiceExport stores the values, we need to describe how updates to those values (in the Service) are handled. I don’t think we’d ever want different values on a Service and its ServiceExport, so having only the names of the labels/annotations is sufficient IMO and avoids some complexity.

[MrFreezeex]

Hmm maybe it's not clear but what I wrote in the KEP is not supposed to look at the corresponding Service at all, like all the labels/annotations exported would be in the ServiceExport spec a bit like what's done in gateway API (which makes me realize that I haven't posted the link to gateway API doc in the description: https://gateway-api.sigs.k8s.io/geps/gep-1762/#labels-and-annotations)

[MrFreezeex]

Although I was originally in favor of not adding any fields to the ServiceExport object at all, but with @mikemorris suggestion in the meeting I was a bit more inclined to do it a bit like GatewayAPI and tried to advocate for the reasons we would like to do that in this PR but a different approach is definitely doable!

[sftim]

• Specify the exact keys and values to copy in the ServiceExport spec.

• Specify the keys that are copied from the Service instance.

How about a CEL expression that takes a key and returns true or false? It's expensive to evaluate but the set of keys is typically stable so the filter shouldn't be recomputed often.

Potentially two expressions: one for labels, one for annotations

Why is this good?

• flexible
• lets you do things like wildcards but with an exception
• very configurable handling for unrecognized labels / annotations
• allows for rules like "export either both of these labels or neither of them" or "export label A or B, but never both"
• extensible (I think) to filtering on values as well

[sftim]

AIUI server-side apply lets us go further and decide whether to become a field manager for a particular key, and if so to:

• take over management
• jointly manage the key
• set / update the value but only if there is no other declared manager
• skip this key and not copy it to the target

A CEL expression could return which of those four options to pick, per label / annotation key.

[MrFreezeex]

AIUI server-side apply lets us go further and decide whether to become a field manager for a particular key

I don't think this is a case where server-side apply would be relevant here because this is not a conflict problem within the same cluster between different actors.

The context is that we need to figure out a set of labels/annotations that are getting exported from each "local Services" in a way that would reasonably not result in a "cross-cluster conflict" even if third party tools are used that might add extra labels/annotations (kubectl with the last applied annotation, argocd with the application labels are example of that).

How about a CEL expression that takes a key and returns true or false? It's expensive to evaluate but the set of keys is typically stable so the filter shouldn't be recomputed often.

This is quite an interesting idea to apply CEL here. I see it as a "copy-filter" approach but in a much more flexible way. But it's also way more complex I am not sure if as a user of an MCS-API implementation, I would want to write some CEL expressions to filter my labels/annotations and also probably making sure that this expression would be the same on all similar ServiceExport across clusters...

So IMO this has the same benefits of "copy-filter" approach (for instance a list of prefix) which is consistency with the rest of the API: the source of truth is on the Service itself with more flexibility but also more complex for users in respect to other "copy-filter" solutions.

I am not entirely sure if this is really better than the way this KEP is currently proposing which is trying to skip this copy-filter problem entirely by adding dedicated fields in the spec that won't be touched by third party tooling which may result in a conflict when reconciling those cross clusters.

[sftim]

@MrFreezeex I'd like the CEL expression idea to appear in the list of alternatives; it doesn't have to get listed as the winning choice.

What I disagreed with, lightly, was the implication that we had to choose between exactly 2 design options here. Actually there are more.

[MrFreezeex]

Well we never considered CEL but AFAICT it's still is fundamentally a "copy-filter" approach which we discussed here and with some points against it in Exporting labels/annotations from the Service/ServiceExport objects. The only thing that might not be relevant for the CEL proposal is the last point about flexibility which is traded by more complexity. Will try to rephrase that last point to include CEL expressions.

[sftim]

• Specify the exact keys and values to copy in the ServiceExport spec.

• Specify the keys that are copied from the Service instance.

How about a CEL expression that takes a key and returns true or false? It's expensive to evaluate but the set of keys is typically stable so the filter shouldn't be recomputed often.

Potentially two expressions: one for labels, one for annotations

Why is this good?

• flexible
• lets you do things like wildcards but with an exception
• very configurable handling for unrecognized labels / annotations
• allows for rules like "export either both of these labels or neither of them" or "export label A or B, but never both"
• extensible (I think) to filtering on values as well

[sftim]

AIUI server-side apply lets us go further and decide whether to become a field manager for a particular key, and if so to:

• take over management
• jointly manage the key
• set / update the value but only if there is no other declared manager
• skip this key and not copy it to the target

A CEL expression could return which of those four options to pick, per label / annotation key.