feat: add CustomResourceMonitor CRD (WIP)

[CatherineF-dev]

What this PR does / why we need it:
What: A more user-friendly UX to collect custom resource metrics.
Why:

1. Using CustomResourceState monitoring feature is not user-friendly. Current ways on configuring CustomResourceState monitoring are:

• --custom-resource-state-config "inline yaml (see example)" or
• --custom-resource-state-config-file /path/to/config.yaml

2. Current CustomResourceState monitoring feature doesn't support multiple configuration files.
   For example, for a company with 10 teams, each team wants to collect CustomResourceState metrics for their owned Custom Resources.

See more in design doc.

How does this change affect the cardinality of KSM: (increases, decreases or does not change cardinality) no

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #2050 #1948

Design doc is ready to review while codes are proof-of-concepts.
Tested: 1. apply examples/cr.yaml 2. apply examples/cr2.yaml 3. delete two yamls.

[CatherineF-dev]

fyi: @chrischdi

[rexagod]

Hey, thank you for working on this. Just a question right off the top of my mind, would this be better suited to be in a KSM operator? I remember a discussion started my @mrueg in the room highlighting the need for this feature a while back but we eventually held off on adding it for the sole reason of setting up an operator first, IIRC.

[chrischdi]

Hey, thank you for working on this. Just a question right off the top of my mind, would this be better suited to be in a KSM operator? I remember a discussion started my @mrueg in the room highlighting the need for this feature a while back but we eventually held off on adding it for the sole reason of setting up an operator first, IIRC.

Let's move that discussion over to the issue itself #1948, happy to collect the facts and opinions there to properly document a decision :-)

Thanks @CatherineF-dev , I'll try to figure out time next week after kubecon to take a look at it 👍 🥳

[CatherineF-dev]

Hey, thank you for working on this. Just a question right off the top of my mind, would this be better suited to be in a KSM operator? I remember a discussion started my @mrueg in the room highlighting the need for this feature a while back but we eventually held off on adding it for the sole reason of setting up an operator first, IIRC.

1. The reason why for this PR is that coupled custom resources metrics targets and kube-state-metrics deployment causes managing custom resource configuration not easy. And then I found a CRD can handle this issue and also solve Read custom resource metrics configuration from a Custom Resource Definition #1948.

Currently, one of these flags is added into kube-state-metrics deployment.

• --custom-resource-state-config "inline yaml (see example)" or
• --custom-resource-state-config-file /path/to/config.yaml

For example, for a company with two teams (monitoring platform team and application team), application team needs to change kube-state-metrics deployment managed by monitoring platform team.

2. This isn't make KSM as KSM operator fully. It won't add CRD for non custom resource metrics. I agree that CRD makes control KSM memory harder, since new metrics might be added overtime. A flag --custom_resources_ksm_cr_watched_labels is added to handle this case.

3. KSM already watches pods/endpoints and reconciles around autosharding. This PR only adds one additional watch on CustomResourceMonitor CR and update custom metrics store dynamically.

[CatherineF-dev]

I find *.k8s.io needs additional approvals. Planned to use customresource.ksm.k8s.io before.

[rexagod]

I'd prefer *.k8s.io group for the CR given we exist under the same organization here.

[CatherineF-dev]

Hi @dgrisonnet, could you help review this design doc when you're available? Thanks!

Related issues #2050 #1948

[dgrisonnet]

Sure, I'll try to catch up on the various discussion asap

[logicalhan]

/assign

[dgrisonnet]

/triage accepted
/assign

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle stale
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[rexagod]

@CatherineF-dev Are you working on this? I see how this could be a large-enough issue entailing multiple cases to warrant more hands, and if so, feel free to bring this up in the next SIG call and we can have more folks contributing to your fork, if that sounds good!

[CatherineF-dev]

Yes, from previous discussion, I remember we plan to split KSM and KSM Custom Metrics into 2 binaries first. Then add CRD feature.

So I haven't resolved conflicts in this PR. Waiting 2 binaries.

(This prototype was working in my k8s cluster)

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle rotten
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[rexagod]

I believe we can get this in regardless, and include this change whenever the split happens. Getting this one in would resolve a lot of issues, in addition to deserving its own minor release.

I'm +1 for making this happen pre-split, since we don't plan on doing that in the immediate future.

[rexagod]

/remove-lifecycle rotten

[k8s-ci-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle stale
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle rotten
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[tyxeron]

@CatherineF-dev
Whats the status on this issue?

This PR would be very helpful