feat(kuma-cp): implement delta xDS for envoy config exchange

[lukidzi]

Motivation

Envoy supports incremental xDS, which sends only changes rather than the entire state.

Implementation information

• Run a Kuma 2-cluster multizone setup with incremental xDS. While fix(delta): force push EDS once CDS sent for Ads go-control-plane#4 is still needed, it’s not required for Envoy 1.32+ since it supports the Endpoints cache when initial_fetch_timeout is set.
• Extracted common logic in each callback to support both communication types.
• Each callback now uses a separate map to store connection information, depending on mode, preventing stream conflicts.
• Added a global and dataplane-only flag to toggle incremental xDS for all DPPs or a single DPP.

Supporting documentation

https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol#incremental-xds

Fix #XX

[Icarus9913]

Is it possible with this case?
2.9.0(CP|DP) --> 2.10.0(CP|DP) --> 2.9.0(CP), 2.10.0(DP)

We just allow hight version CP wth low version DP right?

[Icarus9913]

If the CP disabled UseDeltaXds and the DP request DeltaXDSMode, should we return the DP an error?

[lukidzi]

No, we should allow individual DPPs to run with Delta. This approach enables a progressive migration. UseDeltaXds enforces Delta mode for all DPPs but still allows individual DPPs to revert to the previous mode if needed.

[Icarus9913]

OK, I understand it right now. With the file pkg/xds/server/v3/components.go we enable the deltaServer by default. I think maybe we shold rename the variable Experimental.UseDeltaXds because we may thought it was a swicth to enable or disable the whole delta XDS feature.

So we got these configurations:

1. run the delta server by default
2. CP Experimental.UseDeltaXds force enable deltaXDS for all DPPs
3. DP ENV KUMA_EXPERIMENTAL_USE_DELTA_XDS to choose use stow or delta

[lukidzi]

I think the DPP environment variable is clear, and it shouldn't be a boolean. If it were, we wouldn’t be able to distinguish between using SOTW and "Not Defined." For the control plane configuration, you're right—I need to find a better name for it

[lahabana]

The docs talk about XDSTransportProtocolVariant: https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol#variants-of-the-xds-transport-protocol

[lahabana]

Or Just APIType:

// APIs may be fetched via either REST or gRPC.
type ApiConfigSource_ApiType int32

const (
	// Ideally this would be 'reserved 0' but one can't reserve the default
	// value. Instead we throw an exception if this is ever used.
	//
	// Deprecated: Marked as deprecated in envoy/config/core/v3/config_source.proto.
	ApiConfigSource_DEPRECATED_AND_UNAVAILABLE_DO_NOT_USE ApiConfigSource_ApiType = 0
	// REST-JSON v2 API. The `canonical JSON encoding
	// <https://developers.google.com/protocol-buffers/docs/proto3#json>`_ for
	// the v2 protos is used.
	ApiConfigSource_REST ApiConfigSource_ApiType = 1
	// SotW gRPC service.
	ApiConfigSource_GRPC ApiConfigSource_ApiType = 2
	// Using the delta xDS gRPC service, i.e. DeltaDiscovery{Request,Response}
	// rather than Discovery{Request,Response}. Rather than sending Envoy the entire state
	// with every update, the xDS server only sends what has changed since the last update.
	ApiConfigSource_DELTA_GRPC ApiConfigSource_ApiType = 3
	// SotW xDS gRPC with ADS. All resources which resolve to this configuration source will be
	// multiplexed on a single connection to an ADS endpoint.
	// [#not-implemented-hide:]
	ApiConfigSource_AGGREGATED_GRPC ApiConfigSource_ApiType = 5
	// Delta xDS gRPC with ADS. All resources which resolve to this configuration source will be
	// multiplexed on a single connection to an ADS endpoint.
	// [#not-implemented-hide:]
	ApiConfigSource_AGGREGATED_DELTA_GRPC ApiConfigSource_ApiType = 6
)

In which I'd use the values: DELTA_GRPC and GRPC btw how come we're not using AGGREGATED_GRPC ?

[lukidzi]

It's because we set it here : https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/config_source.proto#envoy-v3-api-msg-config-core-v3-apiconfigsource

	AdsConfig: &envoy_core_v3.ApiConfigSource{
		ApiType:                   configType,
		TransportApiVersion:       envoy_core_v3.ApiVersion_V3,
		SetNodeOnFirstMessageOnly: true,
		GrpcServices: []*envoy_core_v3.GrpcService{
			buildGrpcService(parameters, enableReloadableTokens),
		},
	},

And ApiType can be API type (gRPC, REST, delta gRPC)

[lukidzi]

This is looking interesting! One thing that isn't clear to me is how you do the opt-in on Kubernetes. How do I set the dataplane env var to just opt-in 1 deployment?

It was my mistake not to validate a single Kubernetes resource with Delta. I had to introduce a new struct to the Dataplane object, which allows setting this configuration. I fixed it, and now it’s possible to enable Delta using an annotation on the Kubernetes resource.

[lahabana]

Didn't look at implem only the API.

This is looking interesting! One thing that isn't clear to me is how you do the opt-in on Kubernetes. How do I set the dataplane env var to just opt-in 1 deployment?

[lahabana]

The docs talk about XDSTransportProtocolVariant: https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol#variants-of-the-xds-transport-protocol

[lahabana]

Or Just APIType:

// APIs may be fetched via either REST or gRPC.
type ApiConfigSource_ApiType int32

const (
	// Ideally this would be 'reserved 0' but one can't reserve the default
	// value. Instead we throw an exception if this is ever used.
	//
	// Deprecated: Marked as deprecated in envoy/config/core/v3/config_source.proto.
	ApiConfigSource_DEPRECATED_AND_UNAVAILABLE_DO_NOT_USE ApiConfigSource_ApiType = 0
	// REST-JSON v2 API. The `canonical JSON encoding
	// <https://developers.google.com/protocol-buffers/docs/proto3#json>`_ for
	// the v2 protos is used.
	ApiConfigSource_REST ApiConfigSource_ApiType = 1
	// SotW gRPC service.
	ApiConfigSource_GRPC ApiConfigSource_ApiType = 2
	// Using the delta xDS gRPC service, i.e. DeltaDiscovery{Request,Response}
	// rather than Discovery{Request,Response}. Rather than sending Envoy the entire state
	// with every update, the xDS server only sends what has changed since the last update.
	ApiConfigSource_DELTA_GRPC ApiConfigSource_ApiType = 3
	// SotW xDS gRPC with ADS. All resources which resolve to this configuration source will be
	// multiplexed on a single connection to an ADS endpoint.
	// [#not-implemented-hide:]
	ApiConfigSource_AGGREGATED_GRPC ApiConfigSource_ApiType = 5
	// Delta xDS gRPC with ADS. All resources which resolve to this configuration source will be
	// multiplexed on a single connection to an ADS endpoint.
	// [#not-implemented-hide:]
	ApiConfigSource_AGGREGATED_DELTA_GRPC ApiConfigSource_ApiType = 6
)

In which I'd use the values: DELTA_GRPC and GRPC btw how come we're not using AGGREGATED_GRPC ?

[slonka]

A couple of Qs but really happy that we're going to be shipping this 🚢 . Also does go-control-plane handle the stuff like hashing individual resources instead of whole arrays?

[slonka]

Why do we need that in zone ingress? (I might answer that question myself later)

[lukidzi]

It's because Egress and Ingress also can use delta for configuration exchange

[slonka]

do we need a builder for this? it's getting quite packed with parameters

[lukidzi]

We could change it, but probably in a separate PR

[slonka]

maybe instead of this constant if/else you could have a generic function that takes either dpStreams or dpDeltaStreams? this applies to all functions with isDelta

[lukidzi]

I've changed to a function as a parameter, You can take a look

[lukidzi]

A couple of Qs but really happy that we're going to be shipping this 🚢 . Also does go-control-plane handle the stuff like hashing individual resources instead of whole arrays?

Yes, it should verify individual resources so only one endpoint will be sent. I don't know if that is the question.

[lukidzi]

I think there is still some edge case causing flake, need to investigate