Posted on 8 July 2020, updated on 6 August 2021.
You are using EKS, your Gitlab runners, monitoring, logging tools and your core API are deployed on the same node? They don’t use the same resources and you would like to assign pods to a specific node depending on their need? We can do that in 5 min.
A bit of context
Before assigning pods to nodes, let’s understand what we are doing. Kubernetes has several features you can use to customize how pods are scheduled: labels and node selector, affinity and anti-affinity, taint and toleration.
Labels and node selector is the most intuitive feature: you put a label on your node and then you tell your pod (using a node selector) to be deployed only on nodes with a specific label. If no node with the specified label is available, the pod will stay pending.
Affinity and anti-affinity is more complex: you can force your pods to be deployed on specific nodes but it also allows you to define different behaviours for the scheduler. For example, the preferredDuringSchedulingIgnoredDuringExecution policy is interpreted as “schedule the pod on a node with label XXX, if not possible run it elsewhere”. Affinity and anti-affinity can also be used to add behaviour between pods, see the doc for more information.
We saw how to target nodes with pods but not how to deny pods from scheduling/executing on your nodes: taint and toleration are used for that. If you put a taint on a node, no pods nor daemonset will be scheduled on it unless they have the matching toleration.
To resume, if you want to assign pods on a specific node and be sure that only these pods are deployed on the node you need to use taint, toleration and either node selector or affinity. Below is an example of taint, toleration and node selector on EKS.
Deploy an EKS node with taint and label
Firstly, you need to deploy a node with taint and a label. I assume you already have an EKS cluster running, and we will add a new node to it. To set up a taint and a label on your EKS node, you have to add the --kubelet-extra-args parameter in the node userdata. Here is an example:
If you start an EKS node with this userdata you should see the taint dedicated=app:NoSchedule and the label kube/nodetype=app when you describe it. The label still has no effect, but the taint should prevent pods from deploying on your new node. Note that NoSchedule is a keyword, it has to be that or NoExecute. Here is more information about the meaning of these keywords.
The next step is to deploy your pods on this EKS node.
Add a toleration and a nodeselector to your pod
Now that your EKS node is running, you have to add a toleration on your pod to match the node taint. More than that, you will add a nodeSelector attribute to prevent your pod from deploying on other nodes. If you are using Helm charts don’t worry, every helm chart has nodeSelector and toleration implemented, you just need to complete your value file. This pod is an example which will be deployed on your new node:
Your pod is now deployed on your specific EKS node, and he is the only which can do so! You can now create groups of EKS nodes depending on your needs, but don’t overuse this feature if you want your EKS cluster to stay easy to manage!