Working with taints and tolerations

Have you ever needed to resize your nodes for a specific need of an application? Or have you ever had problems with an app which have trouble and impact all the other applications ?
We already saw how to add taints on your AWS nodes with Terraform. You may want to know now why it is useful to separate your pod by group on node groups. We will redefine, in this article, what is taints and toleration. And we will understand when it is useful to add it to your nodes and pods and how you can do it with kubectl.

How taints and toleration works?

First, let’s check what is exactly taints and toleration in Kubernetes. 


Taint

Taints are property of nodes that push pods away if they don't tolerate this taint. Like Labels, one or more Taints can be applied to a node. This means that the node must not accept any pod that does not tolerate all of these taints.

Toleration 

Tolerations are applied to pods, and allow the pods to schedule onto nodes with matching taints.

Taint and toleration

Taints and tolerations works together to ensure that pods are not scheduled onto inappropriate nodes. One or more taints are applied to a node, this marks that the node should not accept any pods that do not tolerate the taints.

Taint is like: key=value:Effect

You can assign 3 different values to “effect:

  • NoSchedule: if this taint is applied to a node which contains already some pod that doesn’t tolerate this taint, they are not excluded from this node. But no more pods are scheduled on this node if it doesn’t match all the taints of this node. This is a strong constraint.

  • PreferNoSchedule: Like the previous one, this taint may not allow pods to be scheduled on the node. But this time if the pod tolerates one taint, it can be scheduled. This is a soft constraint. 

  • NoExecute: This taint applies to a node excluding all actual running pods on it and doesn’t allow scheduling if new pods don’t tolerate all taint. This is a strong constraint. 

 

The default value for operator is “Equal. But it can also be “Exist

  • Equal : key=value:Effect
  • Exist : keyExist:Effect

Which means that if you change the operator by “exist you don’t have to give a value. If the taint isn’t present on the node, “effect is applied to the pod. 

Let's illustrate with schemas. First, when there is no taint, pod can be affected to any node (here Pod1 is attach to Node1 but it could also be on Node2).

cluster-1

<schema-without-taint>

 

Now, add a taint on the node:
taint=test:NoSchedule

This means if the pod doesn't match this taint, it can’t be scheduled on this node. You can see on the schema below, “new pod doesn’t have toleration so it has been affected on Node2. But Pod2, which has toleration that match the taint, can be affected on Node1. 

cluster-2
<schema-with-noschedule-taint>

 

Now, if we add a taint on the node:
taint=test:NoExecute

It will exclude all pods which have toleration that doesn’t match the taint. 

 

cluster-3

<schema-with-noexecute-taint>

Deploy node with a taint

To add a taint to a existing node, you can run the following command:

$ kubectl taint nodes node-name key=value:effect

Example: $ kubectl taint nodes node-main taint=test:PrefereNoSchedule

 

To show taint of your node you can run this command: 

$ kubectl describe node

 

Name:  node-main

Roles:  <none>

Labels:  [...]

Annotations:   [...]

CreationTimestamp:  Sat, 17 Apr 2021 04:38:19 +0200

Taints:  taint=test:PreferNoSchedule

[...]

Deploy a pod with a specific node

Now that your nodes are tainted, here is an example to add tolerations to your pod: 

tolerations:

key: "key1"

operator: "Equal"

value: "value1"

Exemple :


Taint and toleration are useful if you want to work with dedicated nodes. With dedicated nodes you can create a node pool with very specific parameters (for exemple high CPU) and use these nodes only for specific applications. You also can separate your applicative pods from those of your secondary applications.

Pay attention to the rules defined in the applied taints. Indeed, if a strong taint is applied to all the nodes and a pod does not have any tolerance, it will not be able to be executed anywhere. Another example: if several taints are applied to a node, the strongest wins.

Kimelyne Servais

Kimelyne Servais

Kimelyne is a Site Reliability Engineer (SRE) at Padok. She works with DevOps technologies such as Terraform, Docker, Gitlab CI, Kubernetes, and AWS.

What do you think? Leave your comments here !