Improve usage of resources in Kubernetes

[worldtiki]

Describe the feature

The documentation shows an example of how to use Mittens with Kubernetes. The example describes how to setup Mittens as a sidecar, which will send requests to the primary container on the same pod.
This setup will indeed run the warmup process, but it will leave the sidecar running after the warmup completes. There is no actual cpu usage from that point onwards, but the container will still have some cpu and memory allocated to it which could otherwise be used by other pods.

There's been new features added to Kubernetes that may allow us to have a more optimized setup and stop the Mittens container at the end. The goal of this ticket is to find someone willing to spend some time to investigate the different Kube mechanisms (init containers, warmup probes, etc) and propose a better setup 😄

Rational

Less wastage of resources.

[sumitanvekar]

Init containers run to completion before the app containers start, so they cannot be used for warm up. https://kubernetes.io/docs/concepts/workloads/pods/init-containers/. If you want to try it out we need to pass -file-probe-enabled=false and -exit-after-warmup=true and not have readinessProbe and livenessProbe sections on the initContainer spec since those are not supported by init containers.

I thought of couple more options:

• postStart hook - not feasible since we will need to have mittens installed within the app containers which is not desirable.
• reduce resource request - We can reduce resource request to minimum which keeping limits as today. That way containers will use less resource while it's idle. But it can lead to unexpected edge cases where mittens might not get any resource to run