Add kube tolerations to limit which nodes a test pod can be launched on

[techcobweb]

Story

As a kubernetes admin I want galasa to only schedule test run pods on nodes which are a subset of the nodes available, so that some other nodes will not suffer when lots of galasa tests are running.

Background

Tolerations in Kube are a thing which can dictate where a pod gets scheduled.

Adding them to our helm charts, values files and config map can indicate which nodes tests should execute on.

Externals:

As an ecosystem admin, you may need to configure where which hardware nodes your Galasa test pods are run on. As an example, you may wish to dedicate certain k8s nodes to exlusively to running Galasa tests to avoid resource conflicts with other workloads on the cluster. This ensures hardware isolation and gives a better guarantee of the performance of your Galasa Ecosystem.

Two components are required to achieve this:

• A k8s node affinity allows you to designate a preffered node for your Galasa tests to be run on
• A k8s node taint prevents other workloads from scheduling on that node. To allow you Galasa test pods to be scheduled on the tainted node, you must define a node tolerance

These two settings can be configured via the Galasa Ecosystem configmap, for example:

galasa_node_preferred_affinity: galasa-engines

galasa_node_tolerations: galasa-engines=Exists:NoSchedule
The node affinity defines a label that can be used to identify nodes that your Galasa Ecosystem should schedule pods on.

The Galasa Node Tolerations param defines a comma-separated list of toleration conditions of the format "nodelabel=Operator:Condition". In the above example, nodes with the label "galasa-engines" that have a "NoSchedule" taint should be tolerated, allowing the Galasa test run pods to be scheduled on that node.

Tasks

• Code in engine manager which launches tests, now reads a config map to see which tolerances to apply. See PR: Adding in Tolerations to the Galasa TestPodScheduler galasa#45
• Documentation of properties in the values file and helm chart which sets up the config map.
  See PR Add node affinity and toleration conditions and doc helm#56