Check templating

[moshloop]

Use Cases:

1. Running a http check against all endpoints of:
   • Ingress
   • Istio Service
   • DNS records (i.e. for external SRV or internal headless services e.g. Http Check - Check all records returned by DNS #1780
2. Check the validity of all TLS certificates stored in secrets.
3. Extend / replace Helm / Flux health checks

Depending on how we implement this, it could enable other features including #2130 #1959

I see a couple of options:

1. Check sequencing

Currently checks are entirely independent of other checks in the same canary, the canary acts as logical group, and a place to suspend/resume/schedule/annotate multiple checks in the same place.

If we add sequencing so that the checks run in order, we could have the first check being a kubernetesResource lookup, followed by a http check that templates itself based on that lookup.

kind: Canary
metadata:
   name: "sequence"
spec: 
  - name: lookup
     kubernetesResource: ...
  - name: check
     forEach: $(.lookup.resources)
     http:
       url: $(.item.spec.hosts.0)

Pros: High flexibility, but would also enable other use cases Cons: Breaking change and/or a new CRD version with migration required.

2. Check dependencies

kind: Canary
metadata:
   name: "lookup"
spec: 
  - name: ingress
     kubernetesResource: ...
---
kind: Canary
metadata:
   name: "dependencies"
spec: 
  schedule: "@every 30s"
  forEach: lookupCanary("lookup").ingress.resources
  http:
     - name: "ingress-$(.item.metadata.name)"
       url: $(.item.spec.hosts.0)

Pros: Reduces load on k8s-server Cons: Increased database storage in check_status table, scripting is not ideal for a primary, very common use case.

3. Add a kubernetes lookup directly to the http check:

spec: 
  schedule: "@every 30s"
   http:
     - name: "ingress-$(.item.metadata.name)"
        kubernetesResources:
           - kind: Ingress
        url: $(.item.spec.hosts.0)

Pros: Straightforward to implement, Cons: Limited to http checks and would have impact on k8s api server (unless we cache kubernetesResources)

4. Use a topology

Currently this is undocumented, and implementation would add some complexity and overhead for end-users

Pros: Works today, Cons: currently undocumented, and would force users to learn a new CRD and concepts, + overhead for components that may not be necessary.

5. Rely on 3rd party tools like kyverno

e.g. https://kyverno.io/policies/?policytypes=generate