WIP: alerts: Alert when a node evicts pods for any reason #273
Conversation
Evictions are unusual and may be a sign of resource pressure. While it is not a strong error signal, it is important to know when these events occur as they may be symptomatic of workload, node, or cluster problems. In a healthy cluster, eviction should be rare. The kubelet_evictions metric was added in Kubernetes 1.16.
smarterclayton
force-pushed
the
track_evictions
branch
from
4d59222
to
cd7901f
Compare
|
Bumped but still testing this (it's hard to trigger evictions on nodes surprisingly) |
|
This might just be our systems, but evictions are actually a pretty common occurrence in many of our clusters. We purposefully have some pods around to provide a buffer that can be quickly evicted and the capacity used while waiting for the cluster autoscaler to add capacity. To get some evictions, you could create a PriorityClass with value -1 and deploy some pods using it. Then any other deployments can easily evict one of those. |
|
That would be preemption (which we should also alert on), but eviction is more like "You use too much local filesystem". Priority class impacts eviction, but less so than preemption. |
|
This PR has been automatically marked as stale because it has not The next time this stale check runs, the stale label will be Thank you for your contributions! |
Evictions are unusual and may be a sign of resource pressure. While
it is not a strong error signal, it is important to know when these
events occur as they may be symptomatic of workload, node, or cluster
problems. In a healthy cluster, eviction should be rare.
The kubelet_evictions metric was added in Kubernetes 1.16.
Still testing this alert