Pod Placement on TechBlog about OpenShift/Ansible/Satellite and much more

Working with Environments

Wed, 12 Jan 2022 00:00:00 +0000

Imagine you have one OpenShift cluster and you would like to create 2 or more environments inside this cluster, but also separate them and force the environments to specific nodes, or use specific inbound routers. All this can be achieved using labels, IngressControllers and so on. The following article will guide you to set up dedicated compute nodes for infrastructure, development and test environments as well as the creation of IngressController which are bound to the appropriate nodes.

Prerequisites

Before we start we need an OpenShift cluster of course. In this example we have a cluster with typical 3 control plane nodes (labelled as master) and 7 compute nodes (labelled as worker)

oc get nodes
NAME STATUS ROLES AGE VERSION
ip-10-0-138-104.us-east-2.compute.internal Ready master 13h v1.21.1+6438632
ip-10-0-149-168.us-east-2.compute.internal Ready worker 13h v1.21.1+6438632 # <-- will become infra
ip-10-0-154-244.us-east-2.compute.internal Ready worker 16m v1.21.1+6438632 # <-- will become infra
ip-10-0-158-44.us-east-2.compute.internal Ready worker 15m v1.21.1+6438632 # <-- will become infra
ip-10-0-160-91.us-east-2.compute.internal Ready master 13h v1.21.1+6438632
ip-10-0-188-198.us-east-2.compute.internal Ready worker 13h v1.21.1+6438632 # <-- will become worker-test
ip-10-0-191-9.us-east-2.compute.internal Ready worker 16m v1.21.1+6438632 # <-- will become worker-test
ip-10-0-192-174.us-east-2.compute.internal Ready master 13h v1.21.1+6438632
ip-10-0-195-201.us-east-2.compute.internal Ready worker 16m v1.21.1+6438632 # <-- will become worker-dev
ip-10-0-199-235.us-east-2.compute.internal Ready worker 16m v1.21.1+6438632 # <-- will become worker-dev

We will use the 7 nodes to create the environments for:

Infrastructure Services (3 nodes) - will be labelled as infra
Development Environment (2 nodes) - will be labelled as worker-dev
Test Environment (2 nodes) - will be labelled as worker-test

To do this, we will label the nodes and create dedicated roles for them.

Create Nodes Labels and MachineConfigPools

Let’s create a maschine config pool for the different environments.

The pool inherits the configuration from worker nodes by default, which means that any new update on the worker configuration will also update custom labeled nodes. Or in other words: it is possible to remove the worker label from the custom pools.

Create the following objects:

# Infrastructure
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
name: infra
spec:
machineConfigSelector:
matchExpressions:
- {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,infra]} (1)
maxUnavailable: 1 (2)
nodeSelector:
matchLabels:
node-role.kubernetes.io/infra: "" (3)
paused: false
---
# Worker-DEV
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
name: worker-dev
spec:
machineConfigSelector:
matchExpressions:
- {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,worker-dev]}
nodeSelector:
matchLabels:
node-role.kubernetes.io/worker-dev: ""
---
# Worker-TEST
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
name: worker-test
spec:
machineConfigSelector:
matchExpressions:
- {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,worker-test]}
nodeSelector:
matchLabels:
node-role.kubernetes.io/worker-test: ""

1	User worker and infra so that the default worker configuration gets applied during upgrades
2	Whenever an update happens, do only 1 at a time
3	This pool is valid for nodes which are labelled as `infra`

Now let’s label our nodes. First add the new, additional label:

# Add the label "infra"
oc label node ip-10-0-149-168.us-east-2.compute.internal ip-10-0-154-244.us-east-2.compute.internal ip-10-0-158-44.us-east-2.compute.internal node-role.kubernetes.io/infra=
# Add the label "worker-dev"
oc label node ip-10-0-195-201.us-east-2.compute.internal ip-10-0-199-235.us-east-2.compute.internal node-role.kubernetes.io/worker-dev=
# Add the label "worker-test"
oc label node ip-10-0-188-198.us-east-2.compute.internal ip-10-0-191-9.us-east-2.compute.internal node-role.kubernetes.io/worker-test=

This will result in:

NAME STATUS ROLES AGE VERSION
ip-10-0-138-104.us-east-2.compute.internal Ready master 13h v1.21.1+6438632
ip-10-0-149-168.us-east-2.compute.internal Ready infra,worker 13h v1.21.1+6438632
ip-10-0-154-244.us-east-2.compute.internal Ready infra,worker 20m v1.21.1+6438632
ip-10-0-158-44.us-east-2.compute.internal Ready infra,worker 19m v1.21.1+6438632
ip-10-0-160-91.us-east-2.compute.internal Ready master 13h v1.21.1+6438632
ip-10-0-188-198.us-east-2.compute.internal Ready worker,worker-test 13h v1.21.1+6438632
ip-10-0-191-9.us-east-2.compute.internal Ready worker,worker-test 20m v1.21.1+6438632
ip-10-0-192-174.us-east-2.compute.internal Ready master 13h v1.21.1+6438632
ip-10-0-195-201.us-east-2.compute.internal Ready worker,worker-dev 20m v1.21.1+6438632
ip-10-0-199-235.us-east-2.compute.internal Ready worker,worker-dev 20m v1.21.1+6438632

We can remove the worker label from the worker-dev and worker-test nodes now.

Keep the label "worker" for the infra nodes as this is the default worker label which is used when no nodeselector is in use. You can use any other node, just keep in mind that per default new applications will be started on nodes with the labels "worker". As an alternative, you can also define a cluster-wide default node selector.

oc label node ip-10-0-195-201.us-east-2.compute.internal ip-10-0-199-235.us-east-2.compute.internal node-role.kubernetes.io/worker-
oc label node ip-10-0-188-198.us-east-2.compute.internal ip-10-0-191-9.us-east-2.compute.internal node-role.kubernetes.io/worker-

The final node labels will look like the following:

NAME STATUS ROLES AGE VERSION
ip-10-0-138-104.us-east-2.compute.internal Ready master 13h v1.21.1+6438632
ip-10-0-149-168.us-east-2.compute.internal Ready infra,worker 13h v1.21.1+6438632
ip-10-0-154-244.us-east-2.compute.internal Ready infra,worker 22m v1.21.1+6438632
ip-10-0-158-44.us-east-2.compute.internal Ready infra,worker 21m v1.21.1+6438632
ip-10-0-160-91.us-east-2.compute.internal Ready master 13h v1.21.1+6438632
ip-10-0-188-198.us-east-2.compute.internal Ready worker-test 13h v1.21.1+6438632
ip-10-0-191-9.us-east-2.compute.internal Ready worker-test 21m v1.21.1+6438632
ip-10-0-192-174.us-east-2.compute.internal Ready master 13h v1.21.1+6438632
ip-10-0-195-201.us-east-2.compute.internal Ready worker-dev 21m v1.21.1+6438632
ip-10-0-199-235.us-east-2.compute.internal Ready worker-dev 22m v1.21.1+6438632

Since the custom pools (infra, worker-test and worker-dev) inherit their configuration from the default worker pool, no changes on the files on the nodes themselves are triggered at this point.

Create Custom Configuration

Let’s test our setup by deploying a configuration on specific nodes. The following MaschineConfig objects will create a file at /etc/myfile on the nodes labelled either infra, worker-dev or worker_test. Dependent on the node role the content of the file will vary.

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
labels:
machineconfiguration.openshift.io/role: infra (1)
name: 55-infra
spec:
config:
ignition:
version: 2.2.0
storage:
files:
- contents:
source: data:,infra (2)
filesystem: root
mode: 0644
path: /etc/myfile (3)
---
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
labels:
machineconfiguration.openshift.io/role: worker-dev
name: 55-worker-dev
spec:
config:
ignition:
version: 2.2.0
storage:
files:
- contents:
source: data:,worker-dev
filesystem: root
mode: 0644
path: /etc/myfile
---
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
labels:
machineconfiguration.openshift.io/role: worker-test
name: 55-worker-test
spec:
config:
ignition:
version: 2.2.0
storage:
files:
- contents:
source: data:,worker-test
filesystem: root
mode: 0644
path: /etc/myfile

1	Valid for node with the role xyz
2	Content of the file
3	File to be created

Since this is a new configuration, all nodes will get reconfigured.

NAME STATUS ROLES AGE VERSION
ip-10-0-138-104.us-east-2.compute.internal Ready master 13h v1.21.1+6438632
ip-10-0-149-168.us-east-2.compute.internal Ready,SchedulingDisabled infra,worker 13h v1.21.1+6438632
ip-10-0-154-244.us-east-2.compute.internal Ready infra,worker 28m v1.21.1+6438632
ip-10-0-158-44.us-east-2.compute.internal Ready infra,worker 27m v1.21.1+6438632
ip-10-0-160-91.us-east-2.compute.internal Ready master 13h v1.21.1+6438632
ip-10-0-188-198.us-east-2.compute.internal Ready worker-test 13h v1.21.1+6438632
ip-10-0-191-9.us-east-2.compute.internal NotReady,SchedulingDisabled worker-test 27m v1.21.1+6438632
ip-10-0-192-174.us-east-2.compute.internal Ready master 13h v1.21.1+6438632
ip-10-0-195-201.us-east-2.compute.internal Ready worker-dev 27m v1.21.1+6438632
ip-10-0-199-235.us-east-2.compute.internal NotReady,SchedulingDisabled worker-dev 28m v1.21.1+6438632

Wait until all nodes are ready and test the configuration by verifying the content of /etc/myfile:

###
# infra nodes:
###
oc get pods -n openshift-machine-config-operator -l k8s-app=machine-config-daemon --field-selector "spec.nodeName=ip-10-0-149-168.us-east-2.compute.internal"
NAME READY STATUS RESTARTS AGE
machine-config-daemon-f85kd 2/2 Running 6 16h
# Get file content
oc rsh -n openshift-machine-config-operator machine-config-daemon-f85kd chroot /rootfs cat /etc/myfile
Defaulted container "machine-config-daemon" out of: machine-config-daemon, oauth-proxy
infra
###
#worker-dev:
###
oc get pods -n openshift-machine-config-operator -l k8s-app=machine-config-daemon --field-selector "spec.nodeName=ip-10-0-195-201.us-east-2.compute.internal"
NAME READY STATUS RESTARTS AGE
machine-config-daemon-s6rr5 2/2 Running 4 3h5m
# Get file content
oc rsh -n openshift-machine-config-operator machine-config-daemon-s6rr5 chroot /rootfs cat /etc/myfile
Defaulted container "machine-config-daemon" out of: machine-config-daemon, oauth-proxy
worker-dev
###
# worker-test:
###
oc get pods -n openshift-machine-config-operator -l k8s-app=machine-config-daemon --field-selector "spec.nodeName=ip-10-0-188-198.us-east-2.compute.internal"
NAME READY STATUS RESTARTS AGE
machine-config-daemon-m22rf 2/2 Running 6 16h
# Get file content
oc rsh -n openshift-machine-config-operator machine-config-daemon-m22rf chroot /rootfs cat /etc/myfile
Defaulted container "machine-config-daemon" out of: machine-config-daemon, oauth-proxy
worker-test

The file /etc/myfile exists on all nodes and depending on their role the files have a different content.

Bind an Application to a Specific Environment

The following will label the nodes with a specific environment and will deploy an example application, which should only be executed on the appropriate nodes.

Let’s label the nodes with environment=worker-dev and environment=worker-test:

oc label node ip-10-0-195-201.us-east-2.compute.internal ip-10-0-199-235.us-east-2.compute.internal environment=worker-dev
oc label node ip-10-0-188-198.us-east-2.compute.internal ip-10-0-191-9.us-east-2.compute.internal environment=worker-test

Create a namespace for the example application
```
oc new-project bookinfo
```
Create an annotation and a label for the namespace. The annotation will make sure that the application will only be started on nodes with the same label. The label will be later used for the IngressController setup.
```
oc annotate namespace bookinfo environment=worker-dev
oc annotate namespace bookinfo openshift.io/node-selector: environment=worker-test
oc label namespace bookinfo environment=worker-dev
```

Deploy the example application. In this article the sample application of Istio was used:

oc apply -f https://raw.githubusercontent.com/istio/istio/release-1.11/samples/bookinfo/platform/kube/bookinfo.yaml

This will start the application on worker-dev` nodes only, because the annotation in the namespace was created accordingly.

oc get pods -n bookinfo -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
details-v1-86dfdc4b95-v8zfv 1/1 Running 0 9m19s 10.130.2.17 ip-10-0-199-235.us-east-2.compute.internal <none> <none>
productpage-v1-658849bb5-8gcl7 1/1 Running 0 7m17s 10.128.4.21 ip-10-0-195-201.us-east-2.compute.internal <none> <none>
ratings-v1-76b8c9cbf9-cc4js 1/1 Running 0 9m19s 10.130.2.19 ip-10-0-199-235.us-east-2.compute.internal <none> <none>
reviews-v1-58b8568645-mbgth 1/1 Running 0 7m44s 10.128.4.20 ip-10-0-195-201.us-east-2.compute.internal <none> <none>
reviews-v2-5d8f8b6775-qkdmz 1/1 Running 0 9m19s 10.130.2.21 ip-10-0-199-235.us-east-2.compute.internal <none> <none>
reviews-v3-666b89cfdf-8zv8w 1/1 Running 0 9m18s 10.130.2.22 ip-10-0-199-235.us-east-2.compute.internal <none> <none>

Create Dedicated IngressController

IngressController are responsible to bring the traffic into the cluster. OpenShift comes with one default controller, but it is possible to create more in order to use different domains and separate the incoming traffic to different nodes.

Bind the default ingress controller to the infra labeled nodes, so we can be sure that the default router pods are executed only on these nodes:

oc patch ingresscontroller default -n openshift-ingress-operator --type=merge --patch='{"spec":{"nodePlacement":{"nodeSelector": {"matchLabels":{"node-role.kubernetes.io/infra":""}}}}}'

The pods will get restarted, to be sure they are running on infra:

oc get pods -n openshift-ingress -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
router-default-78f8dd6f69-dbtbv 0/1 ContainerCreating 0 2s <none> ip-10-0-149-168.us-east-2.compute.internal <none> <none>
router-default-78f8dd6f69-wwpgb 0/1 ContainerCreating 0 2s <none> ip-10-0-158-44.us-east-2.compute.internal <none> <none>
router-default-7bbbc8f9bd-vfh84 1/1 Running 0 22m 10.129.4.6 ip-10-0-158-44.us-east-2.compute.internal <none> <none>
router-default-7bbbc8f9bd-wggrx 1/1 Terminating 0 19m 10.128.2.8 ip-10-0-149-168.us-east-2.compute.internal <none> <none>

Create the following IngressController objects for worker-dev and worker-test. Replace with the domain of your choice

apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
name: ingress-worker-dev
namespace: openshift-ingress-operator
spec:
domain: worker-dev.<yourdomain> (1)
endpointPublishingStrategy:
type: HostNetwork
httpErrorCodePages:
name: ''
namespaceSelector:
matchLabels:
environment: worker-dev (2)
nodePlacement:
nodeSelector:
matchLabels:
node-role.kubernetes.io/worker-dev: '' (3)
replicas: 3
tuningOptions: {}
unsupportedConfigOverrides: null
---
apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
name: ingress-worker-test
namespace: openshift-ingress-operator
spec:
domain: worker-test.<yourdomain>
endpointPublishingStrategy:
type: HostNetwork
httpErrorCodePages:
name: ''
namespaceSelector:
matchLabels:
environment: worker-test
nodePlacement:
nodeSelector:
matchLabels:
node-role.kubernetes.io/worker-test: ''
replicas: 3
tuningOptions: {}
unsupportedConfigOverrides: null

1	Domainname which is used by this Controller
2	Namespace selector … namespaces with such label will be handled by this IngressController
3	Node Placement … This Controller should run on nodes with this label/role

This will spin up additional router pods on the collect labelled nodes:

oc get pods -n openshift-ingress -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
router-default-78f8dd6f69-dbtbv 1/1 Running 0 8m7s 10.128.2.11 ip-10-0-149-168.us-east-2.compute.internal <none> <none>
router-default-78f8dd6f69-wwpgb 1/1 Running 0 8m7s 10.129.4.10 ip-10-0-158-44.us-east-2.compute.internal <none> <none>
router-ingress-worker-dev-76b65cf558-mspvb 1/1 Running 0 113s 10.130.2.13 ip-10-0-199-235.us-east-2.compute.internal <none> <none>
router-ingress-worker-dev-76b65cf558-p2jpg 1/1 Running 0 113s 10.128.4.12 ip-10-0-195-201.us-east-2.compute.internal <none> <none>
router-ingress-worker-test-6bbf9967f-4whfs 1/1 Running 0 113s 10.131.2.13 ip-10-0-191-9.us-east-2.compute.internal <none> <none>
router-ingress-worker-test-6bbf9967f-jht4w 1/1 Running 0 113s 10.131.0.8 ip-10-0-188-198.us-east-2.compute.internal <none> <none>

Verify Ingress Configuration

To test our new ingress router lets create a route object for our example application:

kind: Route
apiVersion: route.openshift.io/v1
metadata:
name: productpage
namespace: bookinfo
spec:
host: productpage-bookinfo.worker-dev.<yourdomain>
to:
kind: Service
name: productpage
weight: 100
port:
targetPort: http
wildcardPolicy: None
---
kind: Route
apiVersion: route.openshift.io/v1
metadata:
name: productpage-worker-test
namespace: bookinfo
spec:
host: productpage-bookinfo.worker-test.<yourdomain>
to:
kind: Service
name: productpage
weight: 100
port:
targetPort: http
wildcardPolicy: None

Be sure that the name is resolvable and a load balancer is configured accordingly

Verify that the router pod has the correct configuration in the file haproxy.config :

oc get pods -n openshift-ingress
NAME READY STATUS RESTARTS AGE
router-default-78f8dd6f69-dbtbv 1/1 Running 0 95m
router-default-78f8dd6f69-wwpgb 1/1 Running 0 95m
router-ingress-worker-dev-76b65cf558-mspvb 1/1 Running 0 88m
router-ingress-worker-dev-76b65cf558-p2jpg 1/1 Running 0 88m
router-ingress-worker-test-6bbf9967f-4whfs 1/1 Running 0 88m
router-ingress-worker-test-6bbf9967f-jht4w 1/1 Running 0 88m

Verify the content of the haproxy configuration for one of the worker-dev router

oc rsh -n openshift-ingress router-ingress-worker-dev-76b65cf558-mspvb cat haproxy.config | grep productpage
backend be_http:bookinfo:productpage
server pod:productpage-v1-658849bb5-8gcl7:productpage:http:10.128.4.21:9080 10.128.4.21:9080 cookie 3758caf21badd7e4f729209173eece08 weight 256

Compare with worker-test router

oc rsh -n openshift-ingress router-ingress-worker-test-6bbf9967f-jht4w cat haproxy.config | grep productpage
--> Empty result, this router is not configured with that route.

Compare with default router:

backend be_http:bookinfo:productpage
server pod:productpage-v1-658849bb5-8gcl7:productpage:http:10.128.4.21:9080 10.128.4.21:9080 cookie 3758caf21badd7e4f729209173eece08 weight 256

Why does this happen? Why are the default router and the router for worker-dev configured? This happens because it is the default router and we must explicitly tell it to ignore certain labels.

Modify the default IngressController

oc edit ingresscontroller.operator default -n openshift-ingress-operator

Add the following

 namespaceSelector:
matchExpressions:
- key: environment
operator: NotIn
values:
- worker-dev
- worker-test

This will tell the default IngressController to ignore selectors on worker-dev and worker-test

Wait a few seconds until the route pods have been restarted:

oc get pods -n openshift-ingress
NAME READY STATUS RESTARTS AGE
router-default-744998df46-8lh4t 1/1 Running 0 2m32s
router-default-744998df46-hztgf 1/1 Running 0 2m31s
router-ingress-worker-dev-76b65cf558-mspvb 1/1 Running 0 96m
router-ingress-worker-dev-76b65cf558-p2jpg 1/1 Running 0 96m
router-ingress-worker-test-6bbf9967f-4whfs 1/1 Running 0 96m
router-ingress-worker-test-6bbf9967f-jht4w 1/1 Running 0 96m

And test again

oc rsh -n openshift-ingress router-default-744998df46-8lh4t cat haproxy.config | grep productpage
--> empty result

At this point the new router feels responsible. Be sure to have a load balancer configured correctly.

Appendix

Bind other infra-workload to infrastructure nodes:

Internal Registry

oc patch configs.imageregistry.operator.openshift.io/cluster -n openshift-image-registry --type=merge --patch '{"spec":{"nodeSelector":{"node-role.kubernetes.io/infra":""}}}'

OpenShift Monitoring Workload

Create the following file and apply it.

cat <<'EOF' > cluster-monitoring-config-cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: cluster-monitoring-config
namespace: openshift-monitoring
data:
config.yaml: |+
alertmanagerMain:
nodeSelector:
node-role.kubernetes.io/infra: ""
prometheusK8s:
nodeSelector:
node-role.kubernetes.io/infra: ""
prometheusOperator:
nodeSelector:
node-role.kubernetes.io/infra: ""
grafana:
nodeSelector:
node-role.kubernetes.io/infra: ""
k8sPrometheusAdapter:
nodeSelector:
node-role.kubernetes.io/infra: ""
kubeStateMetrics:
nodeSelector:
node-role.kubernetes.io/infra: ""
telemeterClient:
nodeSelector:
node-role.kubernetes.io/infra: ""
EOF

oc create -f cluster-monitoring-config-cm.yaml

Introduction

Thu, 26 Aug 2021 00:00:00 +0000

Pod scheduling is an internal process that determines placement of new pods onto nodes within the cluster. It is probably one of the most important tasks for a Day-2 scenario and should be considered at a very early stage for a new cluster. OpenShift/Kubernetes is already shipped with a default scheduler which schedules pods as they get created accross the cluster, without any manual steps.

However, there are scenarios where a more advanced approach is required, like for example using a specifc group of nodes for dedicated workload or make sure that certain applications do not run on the same nodes. Kubernetes provides different options:

Controlling placement with node selectors
Controlling placement with pod/node affinity/anti-affinity rules
Controlling placement with taints and tolerations
Controlling placement with topology spread constraints

This series will try to go into the detail of the different options and explains in simple examples how to work with pod placement rules. It is not a replacement for any official documentation, so always check out Kubernetes and or OpenShift documentations.

Pod Placement Series

Prerequisites

The following prerequisites are used for all examples.

Let’s image that our cluster (OpenShift 4) has 4 compute nodes

oc get node --selector='node-role.kubernetes.io/worker'
NAME STATUS ROLES AGE VERSION
compute-0 Ready worker 7h1m v1.19.0+d59ce34
compute-1 Ready worker 7h1m v1.19.0+d59ce34
compute-2 Ready worker 7h1m v1.19.0+d59ce34
compute-3 Ready worker 7h1m v1.19.0+d59ce34

An example application (from the catalog Django + Postgres) has been deployed in the namespace podtesting. It contains by default 1 pod for a PostGresql database and one pod for a frontend web application.

oc get pods -n podtesting -o wide | grep Running
django-psql-example-1-h6kst 1/1 Running 0 20m 10.130.2.97 compute-2 <none> <none>
postgresql-1-4pcm4 1/1 Running 0 21m 10.131.0.51 compute-3 <none> <none>

Without any configuration the OpenShift scheduler will try to spread the pods evenly accross the cluster.

Let’s increase the replica of the web frontend:

oc scale --replicas=4 dc/django-psql-example -n podtesting

Eventually 4 additional pods will be started accross the compute nodes of the cluster:

oc get pods -n podtesting -o wide | grep Running
django-psql-example-1-842fl 1/1 Running 0 2m7s 10.131.0.65 compute-3 <none> <none>
django-psql-example-1-h6kst 1/1 Running 0 24m 10.130.2.97 compute-2 <none> <none>
django-psql-example-1-pxhlv 1/1 Running 0 2m7s 10.128.2.13 compute-0 <none> <none>
django-psql-example-1-xms7x 1/1 Running 0 2m7s 10.129.2.10 compute-1 <none> <none>
postgresql-1-4pcm4 1/1 Running 0 26m 10.131.0.51 compute-3 <none> <none>

As you can see, the scheduler already tries to spread the pods evenly, so that every worker node will host one frontend pod.

However, let’s try to apply a more advanced configuration for the pod placement, starting with the Pod Placement - NodeSelector

Node Affinity

Thu, 26 Aug 2021 00:00:00 +0000

Node Affinity allows to place a pod to a specific group of nodes. For example, it is possible to run a pod only on nodes with a specific CPU or disktype. The disktype was used as an example for the nodeSelector and yes … Node Affinity is conceptually similar to nodeSelector but allows a more granular configuration.

Pod Placement Series

Please check out other ways of pod placements:

Expand me...

Using Node Affinity

Currently two types of affinity settings are known:

requiredDuringSchedulingIgnoreDuringExecuption (short required) - a hard requirement which must be met before a pod can be scheduled
preferredDuringSchedulingIgnoredDuringExecution (short preferred) - a soft requirement the scheduler tries to meet, but does not guarantee it

Both types can be specified. In such case the node must first meet the required rule and then attempt to meet the preferred rule.

Preparing node labels

Remember the prerequisites explained in the . Pod Placement - Introduction. We have 4 compute nodes and an example web application up and running.

Before we start with affinity rules we need to label all nodes. Let’s create 2 zones (east and west) for our compute nodes.

You can skip this, if these labels are still set.

Figure 1. Node Zones

oc label nodes compute-0 compute-1 topology.kubernetes.io/zone=east
oc label nodes compute-2 compute-3 topology.kubernetes.io/zone=west

Configure node affinity rule

Like pod affinity the node affinity is defined on the pod specification:

kind: DeploymentConfig
apiVersion: apps.openshift.io/v1
metadata:
name: django-psql-example
namespace: podtesting
[...]
spec:
[...]
template:
[...]
spec:
[...]
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: topology.kubernetes.io/zone
operator: In
values:
- west

In this example the pods are started only on nodes of the zone "West". Since the value is an array, multiple zones can be defined letting the web application be executed on West and East for example. With this setup you can control on which node a specific application shall be executed. For example: you have a group of nodes which provide a GPU and your GPU application must be started only on this group of nodes.

Like with pod affinity you can combine required and preferred settings.

What happened to Node Anti-Affinity?

Unlike Pod Anti-Affinity, there is no concept to define a node Anti-Affinity. Instead you can use the NotIn and DoesNotExist operators to achieve this bahaviour.

Cleanup

As cleanup simply remove the affinity specification from the DeploymentConf. The node labels can stay as they are since they do not hurt.

Summary

This concludes the quick overview of the node affinity. Further information can be found at Node Affinity

NodeSelector

Thu, 26 Aug 2021 00:00:00 +0000

One of the easiest ways to tell your Kubernetes cluster where to put certain pods is to use a nodeSelector specification. A nodeSelector defines a key-value pair and are defined inside the specification of the pods and as a label on one or multiple nodes (or machine set or machine config). Only if selector matches the node label, the pod is allowed to be scheduled on that node.

Kubernetes distingushes between 2 types of selectors:

cluster-wide node selectors: defined by the cluster administrators and valid for the whole cluster
project node selectors: to place new pods inside projects into specific nodes.

Pod Placement Series

Please check out other ways of pod placements:

Expand me...

Using nodeSelector

As previously described, we have a cluster with an example application scheduled accross the worker nodes evenly by the scheduler.

oc get pods -n podtesting -o wide | grep Running
django-psql-example-1-842fl 1/1 Running 0 2m7s 10.131.0.65 compute-3 <none> <none>
django-psql-example-1-h6kst 1/1 Running 0 24m 10.130.2.97 compute-2 <none> <none>
django-psql-example-1-pxhlv 1/1 Running 0 2m7s 10.128.2.13 compute-0 <none> <none>
django-psql-example-1-xms7x 1/1 Running 0 2m7s 10.129.2.10 compute-1 <none> <none>
postgresql-1-4pcm4 1/1 Running 0 26m 10.131.0.51 compute-3 <none> <none>

However, our 4 compute nodes are assembled with different hardware specification and are using different harddisks (sdd vs hdd).

Figure 1. Nodes with Different Specifications

Since our web application must run on fast disks must configure the cluster to schedule the pods on nodes with SSD only.

To start using nodeSelectors we first label our nodes accordingly:

compute-0 and compute-1 are faster nodes with an SSD attached.
compute-2 and compute-2 have a HDD attached.

oc label nodes compute-0 compute-1 disktype=ssd (1)
oc label nodes compute-2 compute-3 disktype=hdd

1	as key we are using disktype

As crosscheck we can list nodes with a specific label:

oc get nodes -l disktype=ssd
NAME STATUS ROLES AGE VERSION
compute-0 Ready worker 7h32m v1.19.0+d59ce34
compute-1 Ready worker 7h31m v1.19.0+d59ce34
oc get nodes -l disktype=hdd
NAME STATUS ROLES AGE VERSION
compute-2 Ready worker 7h32m v1.19.0+d59ce34
compute-3 Ready worker 7h32m v1.19.0+d59ce34

If no matching label is found, the pod cannot be scheduled. Therefore, always label the nodes first.

The 2nd step is to add the node selector to the specification of the pod. In our example we are using a DeploymentConfig, so let’s add it there:

oc patch dc django-psql-example -n podtesting --patch '{"spec":{"template":{"spec":{"nodeSelector":{"disktype":"ssd"}}}}}'

This adds the nodeSelector into: spec/template/spec

 nodeSelector:
disktype: ssd

Kubernetes will now trigger a restart of the pods on the supposed nodes.

oc get pods -n podtesting -o wide | grep Running
django-psql-example-3-4j92k 1/1 Running 0 42s 10.129.2.7 compute-1 <none> <none>
django-psql-example-3-d7hsd 1/1 Running 0 42s 10.129.2.8 compute-1 <none> <none>
django-psql-example-3-fkbfm 1/1 Running 0 14m 10.128.2.18 compute-0 <none> <none>
django-psql-example-3-psskb 1/1 Running 0 14m 10.128.2.17 compute-0 <none> <none>

As you can see, only nodes with a SSD (compute-0 and compute-1) are being used.

Controlling pod placement with project-wide selector

Adding a nodeSelector to a deployment seems fine… until somebody forgets to add it. Then the pods would be started anywhere the scheduler finds suitable. Therefore, it might make sense to use a project-wide node selector, which will automatically be applied on all pods on that project. The project selector is added by the cluster administrator to the Namespace object (no matter what the OpenShift documentation says in it’s example) as openshift.io/node-selector parameter.

Let’s remove our previous configuration and add the setting to our namespace podtesting:

Cleanup

Remove the nodeSelector from the deployment configuration and wait until all pods have been reshuffeld

oc patch dc django-psql-example -n podtesting --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/nodeSelector", "value": "disktype=ssd" }]'

Add the label to the project

oc annotate ns/podtesting openshift.io/node-selector="disktype=ssd"

The OpenShift scheduler will now spread the Pods accross compute-0 or compute-1 again, but not on compute-2 or 3.

We can prove that by stressing our cluster (and nodes) a little bit and scale our frontend application to 10:

oc get pods -n podtesting -o wide | grep Running
django-psql-example-4-2jn2l 1/1 Running 0 27s 10.128.2.8 compute-0 <none> <none>
django-psql-example-4-6g7ks 1/1 Running 0 7m47s 10.129.2.23 compute-1 <none> <none>
django-psql-example-4-752nm 1/1 Running 0 7m47s 10.128.2.7 compute-0 <none> <none>
django-psql-example-4-c5jvm 1/1 Running 0 27s 10.129.2.4 compute-1 <none> <none>
django-psql-example-4-f5kwg 1/1 Running 0 27s 10.129.2.5 compute-1 <none> <none>
django-psql-example-4-g7bcs 1/1 Running 0 7m47s 10.129.2.24 compute-1 <none> <none>
django-psql-example-4-h5tgb 1/1 Running 0 27s 10.129.2.6 compute-1 <none> <none>
django-psql-example-4-spvpp 1/1 Running 0 28s 10.128.2.5 compute-0 <none> <none>
django-psql-example-4-v9qwj 1/1 Running 0 7m48s 10.129.2.22 compute-1 <none> <none>
django-psql-example-4-zgwcv 1/1 Running 0 27s 10.128.2.6 compute-0 <none> <none>

As you can see compute-0 and compute-1 are the only nodes which are used.

Well-Known Labels

nodeSelector is one of the easiest ways to control where an application shall be started. Working with labels is therefore very important as soon as workload shall be added to the cluster. Kubernetes reserves some labels which can be leveraged and some are already predefined on the nodes, for example:

beta.kubernetes.io/arch=amd64
kubernetes.io/hostname=compute-0
kubernetes.io/os=linux
node-role.kubernetes.io/worker=
node.openshift.io/os_id=rhcos

A list of all known can be found at: [1]

Two of them I would like to mention here, since they might become very important when designing the placement of pods:

topology.kubernetes.io/zone
topology.kubernetes.io/region

With these two labels you can create availability zones for your cluster. A zone can be seen a logical failure domain and a cluster is typically spanned across multiple zones. This could be a rack in a data center for example, hardware which is sharing the same switch or simply different data centers. Zones are seen as independent to each other.

A region is made up of one or more zones. A cluster is usually not spanned across multiple region.

Kubernetes makes a few assumptions about the structure of zones and regions:

regions and zones are hierarchical: zones are strict subsets of regions and no zone can be in 2 regions
zone names are unique across regions; for example region "africa-east-1" might be comprised of zones "africa-east-1a" and "africa-east-1b"

Cleanup

This concludes the chapter about nodeSelectors. For the next chapter of the Pod Placement Series (Pod Affinity and Anti Affinity) we need to cleanup our configuration.

Scale the frontend down to 2

oc scale --replicas=2 dc/django-psql-example -n podtesting

Remove the label from the namespace
```
oc annotate ns/podtesting openshift.io/node-selector- (1)
```
1 The minus at the end defines that this annotation shall be removed

And, just to be sure if you have not done this before, remove the nodeSelector from the DeploymentConfig

oc patch dc django-psql-example -n podtesting --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/nodeSelector", "value": "disktype=ssd" }]'

Sources

[1]: Well-Known Labels, Annotations and Taints

Pod Affinity/Anti-Affinity

Thu, 26 Aug 2021 00:00:00 +0000

While noteSelector provides a very easy way to control where a pod shall be scheduled, the affinity/anti-affinity feature, expands this configuration with more expressive rules like logical AND operators, constraints against labels on other pods or soft rules instead of hard requirements.

The feature comes with two types:

pod affinity/anti-affinity - allows constrains against other pod labels rather than node labels.
node affinity - allows pods to specify a group of nodes they can be placed on

Pod Placement Series

Please check out other ways of pod placements:

Expand me...

Pod Affinity and Anti-Affinity

Affinity and Anti-Affinity controls the nodes on which a pod should (or should not) be scheduled based on labels on Pods that are already scheduled on the node. This is a different approach than nodeSelector, since it does not directly take the node labels into account. That said, one example for such setup would be: You have dedicated nodes for developement and production workload and you have to be sure that pods of dev or prod applications do not run on the same node.

Affinity and Anti-Affinity are shortly defined as:

Pod affinity - tells scheduler to put a new pod onto the same node as other pods (selection is done using label selectors)

For example:
- I want to run where this other labelled pod is already running (Pods from same service shall be running on same node.)
Pod Anti-Affinity - prevents the scheduler to place a new pod onto the same nodes with pods with the same labels

For example:
- I definitely do not want to start a pod where this other pod with the defined label is running (to prevent that all Pods of same service are running in the same availability zone.)

As described in the official Kubernetes documention two things should be considered:

The affinity feature requires processing which can slow down the cluster. Therefore, it is not recommended for cluster with more than 100 nodes.
The feature requires that all nodes are consistently labelled (topologyKey). Unintended behaviour might happen if labels are missing.

Using Affinity/Anti-Affinity

Currently two types of pod affinity/anti-affinity are known:

requiredDuringSchedulingIgnoreDuringExecuption (short required) - a hard requirement which must be met before a pod can be scheduled
preferredDuringSchedulingIgnoredDuringExecution (short preferred) - a soft requirement the scheduler tries to meet, but does not guarantee it

Both types can be defined in the same specification. In such case the node must first meet the required rule and then attempt based on best effort to meet the preferred rule.

Preparing node labels

Remember the prerequisites explained in the . Pod Placement - Introduction. We have 4 compute nodes and an example web application up and running.

Before we start with affinity rules we need to label all nodes. Let’s create 2 zones (east and west) for our compute nodes using the well-known label topology.kubernetes.io/zone

Figure 1. Node Zones

oc label nodes compute-0 compute-1 topology.kubernetes.io/zone=east
oc label nodes compute-2 compute-3 topology.kubernetes.io/zone=west

Configure pod affinity rule

In our example we have one database pod and multiple web application pods. Let’s image we would like to always run these pods in the same zone.

The pod affinity defines that a pod can be scheduled onto a node ONLY if that node is in the same zone as at least one already-running pod with a certain label.

This means we must first label the postgres pod accordingly. Let’s labels the pod with security=zone1

oc patch dc postgresql -n podtesting --type='json' -p='[{"op": "add", "path": "/spec/template/metadata/labels/security", "value": "zone1" }]'

After a while the postgres pod is restarted on one of the 4 compute nodes (since we did not specify in which zone this single pod shall be started) - here compute-1 (zone == east):

oc get pods -n podtesting -o wide | grep Running
postgresql-5-6v5h6 1/1 Running 0 119s 10.129.2.24 compute-1 <none> <none>

As a second step, the deployment configuration of the web application must be modified. Remember, we want to run web application pods only on nodes located in the same zone as the postgres pods. In our example this would be either compute-0 or compute-1.

Modify the config accordingly:

kind: DeploymentConfig
apiVersion: apps.openshift.io/v1
[...]
namespace: podtesting
spec:
[...]
template:
[...]
spec:
[...]
affinity:
podAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: security (1)
operator: In (2)
values:
- zone1 (3)
topologyKey: topology.kubernetes.io/zone (4)

1	The key of the label of a pod which is already running on that node is "security"
2	As operator "In" is used the postgres pod must have a matching key (security) containing the value (zone1). Other options like "NotIn", "DoesNotExist" or "Exact" are available as well
3	The value must be "zone1"
4	As topology the topology.kubernetes.io/zone is used. The application can be deployed on nodes with the same label

Setting this (and maybe scaling the replicas up a little bit) will start all frontend pods either on compute-0 or on compute-1.

In other words: On nodes of the same zone, where the postgres pod with the label security=zone1 is running.

oc get pods -n podtesting -o wide | grep Running
django-psql-example-13-4w6qd 1/1 Running 0 67s 10.128.2.58 compute-0 <none> <none>
django-psql-example-13-655dj 1/1 Running 0 67s 10.129.2.28 compute-1 <none> <none>
django-psql-example-13-9d4pj 1/1 Running 0 67s 10.129.2.27 compute-1 <none> <none>
django-psql-example-13-bdwhb 1/1 Running 0 67s 10.128.2.61 compute-0 <none> <none>
django-psql-example-13-d4jrw 1/1 Running 0 67s 10.128.2.57 compute-0 <none> <none>
django-psql-example-13-dm9qk 1/1 Running 0 67s 10.128.2.60 compute-0 <none> <none>
django-psql-example-13-ktmfm 1/1 Running 0 67s 10.129.2.25 compute-1 <none> <none>
django-psql-example-13-ldm56 1/1 Running 0 77s 10.128.2.55 compute-0 <none> <none>
django-psql-example-13-mh2f5 1/1 Running 0 67s 10.129.2.29 compute-1 <none> <none>
django-psql-example-13-qfkhq 1/1 Running 0 67s 10.129.2.26 compute-1 <none> <none>
django-psql-example-13-v88qv 1/1 Running 0 67s 10.128.2.56 compute-0 <none> <none>
django-psql-example-13-vfgf4 1/1 Running 0 67s 10.128.2.59 compute-0 <none> <none>
postgresql-5-6v5h6 1/1 Running 0 3m18s 10.129.2.24 compute-1 <none> <none>

Configure pod anti-affinity rule

For now the database pod and the web application pod are running on nodes of the same zone. However, somebody is asking us to configure it vice versa: the web application should not run in the same zone as postgresql.

Here we can use the Anti-Affinity feature.

As an alternative, it would also be possible to change the operator in the affinity rule from "In" to "NotIn"

kind: DeploymentConfig
apiVersion: apps.openshift.io/v1
[...]
namespace: podtesting
spec:
[...]
template:
[...]
spec:
[...]
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: security (1)
operator: In (2)
values:
- zone1 (3)
topologyKey: topology.kubernetes.io/zone (4)

This will force the web application pods to run only on "west" zone nodes.

django-psql-example-16-4n9h5 1/1 Running 0 40s 10.131.1.53 compute-3 <none> <none>
django-psql-example-16-blf8b 1/1 Running 0 29s 10.130.2.63 compute-2 <none> <none>
django-psql-example-16-f9plb 1/1 Running 0 29s 10.130.2.64 compute-2 <none> <none>
django-psql-example-16-tm5rm 1/1 Running 0 28s 10.131.1.55 compute-3 <none> <none>
django-psql-example-16-x8lbh 1/1 Running 0 29s 10.131.1.54 compute-3 <none> <none>
django-psql-example-16-zb5fg 1/1 Running 0 28s 10.130.2.65 compute-2 <none> <none>
postgresql-5-6v5h6 1/1 Running 0 18m 10.129.2.24 compute-1 <none> <none>

Combining required and preferred affinities

It is possible to combine requiredDuringSchedulingIgnoredDuringExecution and preferredDuringSchedulingIgnoredDuringExecution. In such case the required affinity MUST be met, while the preferred affinity is tried to be met. The following examples combines these two types in an affinity and anti-affinity specification.

The podAffinity block defines the same as above: schedule the pod on a node of the same zone, where a pod with the label security=zone1 is running. The podAntiAffinity defines that the pod should not be started on a node if that node has a pod running with the label security=zone2. However, the scheduler might decide to do so as long the podAffinity rule is met.

spec:
affinity:
podAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: security
operator: In
values:
- zone1
topologyKey: topology.kubernetes.io/zone
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100 (1)
podAffinityTerm:
labelSelector:
matchExpressions:
- key: security
operator: In
values:
- zone2
topologyKey: topology.kubernetes.io/zone

1	The `weight` field is used by the scheduler to create a scoring. The higher the scoring the more preferred is that node.

topologyKey

It is important to understand the topologyKey setting. This is the key for the node label. If an affinity rule is met, Kubernetes will try to find suitable nodes which are labelled with the topologyKey. All nodes must be labelled consistently, otherwise unintended behaviour might occur.

As described in the Kubernetes documentation at Pod Affinity and Anti-Affinity, the topologyKey has some constraints:

Quote Kubernetes:

For pod affinity, empty topologyKey is not allowed in both requiredDuringSchedulingIgnoredDuringExecution and preferredDuringSchedulingIgnoredDuringExecution.
For pod anti-affinity, empty topologyKey is also not allowed in both requiredDuringSchedulingIgnoredDuringExecution and preferredDuringSchedulingIgnoredDuringExecution.
For requiredDuringSchedulingIgnoredDuringExecution pod anti-affinity, the admission controller LimitPodHardAntiAffinityTopology was introduced to limit topologyKey to kubernetes.io/hostname. If you want to make it available for custom topologies, you may modify the admission controller, or disable it.
Except for the above cases, the topologyKey can be any legally label-key.

End of quote

Cleanup

As cleanup simply remove the affinity specification from the DeploymentConf. The node labels can stay as they are since they do not hurt.

Summary

This concludes the quick overview of the pod affinity. The next chapter will discuss Node Affinity rules, which allows affinity based on node specifications.

Taints and Tolerations

Thu, 26 Aug 2021 00:00:00 +0000

While Node Affinity is a property of pods that attracts them to a set of nodes, taints are the exact opposite. Nodes can be configured with one or more taints, which mark the node in a way to only accept pods that do tolerate the taints. The tolerations themselves are applied to pods, telling the scheduler to accept a taints and start the workload on a tainted node.

A common use case would be to mark certain nodes as infrastructure nodes, where only specific pods are allowed to be executed or to taint nodes with a special hardware (i.e. GPU).

Pod Placement Series

Please check out other ways of pod placements:

Expand me...

Understanding Taints and Tolerations

Matching taints and tolerations is defined by a key/value pair and a taint effect.

For example, a node can be tainted as:

spec:
[...]
template:
[...]
spec:
taints:
- effect: NoExecute
key: key1
value: value1
[...]

And a pod can have the matching toleration:

spec:
[...]
template:
[...]
spec:
tolerations:
- key: "key1"
operator: "Equal"
value: "value1"
effect: "NoExecute"
tolerationSeconds: 3600
[...]

This means that the pod is tolerating the taint of the node with the pair key1=value1.

Parameter: effect

Above example is using as effect NoExecute. The following effects are possible:

NoSchedule - new pods are not scheduled, existing pods remain
PreferNoSchedule - new pods are not preferred but can still be scheduled but the scheduler tries to avoid that, existing pods remain
NoExecute - new pods are not scheduled, existing pods (without matching toleration) are removed! The setting tolerationSeconds is used to define a maximum time until a pod is allowed to stay.

Parameter: operator

For the operator two options are possible:

Exists - simply checks if a key exists and key and effect matches. No value should be configured in this case.
Equal (default) - with this option key, value and effect must match exactly.

Configuring Taints and Tolerations

Our compute-0 node is a special node with a GPU installed. We would like that our web application is running on this node and ONLY our web application is running on that node. To achieve this, we will taint the node accordingly with the key/value gpu=enabled and the effect NoExecute and configure a toleration to the DeploymentConfig of our web fronted.

Figure 1. Tainting Node

Always configure tolerations before you taint a node. Otherwise the scheduler might not be able to start a pod until it has been configured to tolerate the taints.

First we will set the toleration to the DeploymentConfig:

oc edit dc/django-psql-example -n podtesting

And add a toleration for the key/value pair, the effect NoExecute and a tolerationSeconds of X seconds.

spec:
[...]
template:
[...]
spec:
tolerations:
- key: "gpu"
value: "enabled"
operator: "Equal"
effect: "NoExecute"
tolerationSeconds: 30 (1)

1	I am setting the tolerationSeconds to 30 seconds to get it done quicker.

Before we taint our node, let’s check which pods are currently running there:

oc get pods -A -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,STATE:.status.phase | grep compute-0 | grep Running
tuned-hrvl4 compute-0 Running
dns-default-ln9s4 compute-0 Running
node-resolver-qk24n compute-0 Running
node-ca-lxrvf compute-0 Running
ingress-canary-fv5w6 compute-0 Running
machine-config-daemon-558v4 compute-0 Running
grafana-78ccdb8c9d-8rqsp compute-0 Running
node-exporter-rn8h4 compute-0 Running
prometheus-adapter-66976bf759-fxdcd compute-0 Running
multus-additional-cni-plugins-29qgq compute-0 Running
multus-mkd87 compute-0 Running
network-metrics-daemon-64hrw compute-0 Running
network-check-target-l6l7n compute-0 Running
ovnkube-node-hrtln compute-0 Running
django-psql-example-24-c599b compute-0 Running
django-psql-example-24-l7znv compute-0 Running
django-psql-example-24-zpg77 compute-0 Running

Multiple different pods are running here, as well as two of our web frontend (django-psql-example)

The other django-psql-example pods are started accross the cluster:

oc get pods -n podtesting -o wide
oc get pods -n podtesting -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
django-psql-example-25-8ppxc 1/1 Running 0 21s 10.128.0.61 master-2 <none> <none>
django-psql-example-25-8rv76 1/1 Running 0 21s 10.130.2.92 compute-2 <none> <none>
django-psql-example-25-9m8k2 1/1 Running 0 21s 10.129.0.67 master-1 <none> <none>
django-psql-example-25-fhvxg 1/1 Running 0 31s 10.128.2.126 compute-0 <none> <none>
django-psql-example-25-kqgwz 0/1 Running 0 21s 10.130.0.71 master-0 <none> <none>
django-psql-example-25-m66nn 1/1 Running 0 21s 10.130.0.70 master-0 <none> <none>
django-psql-example-25-ntjqb 1/1 Running 0 21s 10.128.0.60 master-2 <none> <none>
django-psql-example-25-nxxqh 1/1 Running 0 21s 10.130.2.93 compute-2 <none> <none>
django-psql-example-25-p8nbz 1/1 Running 0 21s 10.131.1.183 compute-3 <none> <none>
django-psql-example-25-ttr9g 1/1 Running 0 21s 10.129.2.57 compute-1 <none> <none>
django-psql-example-25-xn4fp 1/1 Running 0 21s 10.129.2.56 compute-1 <none> <none>
django-psql-example-25-xpqf4 1/1 Running 0 21s 10.128.2.127 compute-0 <none> <none>
django-psql-example-25-xwmwv 1/1 Running 0 21s 10.131.1.184 compute-3 <none> <none>

To taint the node we can simply execute the following command. Be sure to use the same values as in the toleration:

oc adm taint nodes compute-0 gpu=enabled:NoExecute

This will create the following specification in the node object:

spec:
[...]
taints:
- effect: NoExecute
key: gpu
value: enabled

OpenShift will allow pods, which are not tolerating the taints, to keep on running for 30 seconds. After that, these pods will be evicted and started elsewhere.

When we check after the tolerationSeconds time has passed which pods are running on the node compute-0, we will see that most pods have disappeared, except the pods for the webapplication and pods which are part of DaemonSets. (DaemonSets are defined to run on all or specific nodes and are not evicted. The cluster-DaemonSets are tolerating everything)

oc get pods -A -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,STATE:.status.phase | grep compute-0 | grep Running
tuned-hrvl4 compute-0 Running
node-resolver-qk24n compute-0 Running
node-ca-lxrvf compute-0 Running
machine-config-daemon-558v4 compute-0 Running
node-exporter-rn8h4 compute-0 Running
multus-additional-cni-plugins-29qgq compute-0 Running
multus-mkd87 compute-0 Running
network-metrics-daemon-64hrw compute-0 Running
network-check-target-l6l7n compute-0 Running
ovnkube-node-hrtln compute-0 Running
django-psql-example-25-8dzqh compute-0 Running
django-psql-example-25-9p4sb compute-0 Running
django-psql-example-25-pqbvn compute-0 Running
django-psql-example-25-sb6hr compute-0 Running

Removing taints

Taints can be simply removed with the oc command added a trailing -

oc adm taint nodes compute-0 gpu=enabled:NoExecute-

Built-in Taints - Taint Nodes by Condition

Several taints are built into OpenShift and are set during certain events (aka Taint Nodes by Condition) and cleared when the condition is resolved.

The following list is quoted from the OpenShift documentation and provides a list of taints which are automatically set. For example, when a node becomes unavailable:

node.kubernetes.io/not-ready: The node is not ready. This corresponds to the node condition Ready=False.
node.kubernetes.io/unreachable: The node is unreachable from the node controller. This corresponds to the node condition Ready=Unknown.
node.kubernetes.io/out-of-disk: The node has insufficient free space on the node for adding new pods. This corresponds to the node condition OutOfDisk=True.
node.kubernetes.io/memory-pressure: The node has memory pressure issues. This corresponds to the node condition MemoryPressure=True.
node.kubernetes.io/disk-pressure: The node has disk pressure issues. This corresponds to the node condition DiskPressure=True.
node.kubernetes.io/network-unavailable: The node network is unavailable.
node.kubernetes.io/unschedulable: The node is unschedulable.
node.cloudprovider.kubernetes.io/uninitialized: When the node controller is started with an external cloud provider, this taint is set on a node to mark it as unusable. After a controller from the cloud-controller-manager initializes this node, the kubelet removes this taint.

Depending on the condition, the node will either have the effect NoSchedule, which means no new pods will be started there (unless a toleration is configured) or the effect NoExecute, which will evict pods with no tolerations from the node. Typical examples for NoSchedule condition would be: memory-pressure or disk-pressure. If a node is unreachable or not-ready, then it will be automatically tainted with the effect NoExecute. tolerationSeconds (default 300) will be respected.

Tolerating all taints

Tolerations can be configured to tolerate all possible taints. In such case no value or key is configured and operator: "Exists" is used:

spec:
[...]
template:
[...]
spec:
tolerations:
- operator: “Exists”

Some cluster daemonsets (i.e. tuned) are configured this way.

Cleanup

Remove the taint from the node:

oc adm taint nodes compute-0 gpu=enabled:NoExecute-

And remove the toleration specification from the DeploymentConfig

spec:
[...]
template:
[...]
spec:
tolerations:
- key: "gpu"
value: "enabled"
operator: "Equal"
effect: "NoExecute"
tolerationSeconds: 30 (1)

Topology Spread Constraints

Thu, 26 Aug 2021 00:00:00 +0000

Topology spread constraints is a new feature since Kubernetes 1.19 (OpenShift 4.6) and another way to control where pods shall be started. It allows to use failure-domains, like zones or regions or to define custom topology domains. It heavily relies on configured node labels, which are used to define topology domains. This feature is a more granular approach than affinity, allowing to achieve higher availability.

Pod Placement Series

Please check out other ways of pod placements:

Expand me...

Understanding Topology Spread Constraints

Imagine, like for pod affinity, we have two zones (east and west) in our 4 compute node cluster.

Figure 1. Node Zones

These zones are defined by node labels, which can be created with the following commands:

oc label nodes compute-0 compute-1 topology.kubernetes.io/zone=east
oc label nodes compute-2 compute-3 topology.kubernetes.io/zone=west

Now let’s scale our web application to 3 pods. The OpenShift scheduler will try to evenly spread the pods accross the cluster:

oc get pods -n podtesting -o wide
django-psql-example-31-jrhsr 0/1 Running 0 8s 10.130.2.112 compute-2 <none> <none>
django-psql-example-31-q66hk 0/1 Running 0 8s 10.131.1.219 compute-3 <none> <none>
django-psql-example-31-xv7jc 0/1 Running 0 8s 10.128.3.115 compute-0 <none> <none>

Figure 2. Running Pods

If a new pod is started the scheduler may try to start it on compute-1. However, this is done based on best effort. The scheduler does not guarantee that and may try to start it on one of the nodes in zone "West". With the configuration topologySpreadConstraints this can be controlled and incoming pods can only be scheduled on a node of zone "East" (either on compute-0 or compute-1).

Configure TopologySpreadContraint

Let’s configure our topology by adding the following into the DeploymentConfig:

spec:
[...]
template:
[...]
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
name: django-psql-example

This defines the following parameter:

maxSkew - defines the degree to which pods may be unevenly distributed (must be greater than zero). Depending on the whenUnsatisfiable parameter the bahaviour differs:
- whenUnsatisfiable == DoNotSchedule: would not schedule if the maxSkew is is not met.
- whenUnsatisfiable == ScheduleAnyway: the scheduler will still schedule and gives the topology which would decrease the skew a better score
topologyKey - key of node lables
whenUnsatisfiable - defines what to do with a pod if the spread constraint is not met. Can be either DoNotSchedule (default) or ScheduleAnyway
lableSelector - used to find matching pods. Found pods are considered to be part of the topology domain. In our example we simply use a default label of the application: name: django-psql-example

If now a 4th pod shall be started the topologySpreadConstraints will allow this pod on one of the nodes of zone=east (either compute-0 or compute-1). It cannot be scheduled on nodes in zone=west since it would violate the maxSkew: 1

Pod distribution would be 1 in zone east and 3 on zone west.
--> the skew would then be 3 - 1 = 2 which is not equal to 1

Configure multiple TopologySpreadContraint

It is possible to configure multiple TopologySpreadConstraints. In such a case all contrains must meet the requirement (logical AND). For example:

spec:
[...]
template:
[...]
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
name: django-psql-example
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
name: django-psql-example

The first part is identical to our first example and would allow the schedule to start a pod in topology.kubernetes.io/zone: east (compute-0 or compute-1). The second configuration defines not a zone but a node hostname. Now the scheduler can only deploy into zone=east AND onto node=compute-1

Conflicts with multiple TopologySpreadConstraints

If you use multiple constraints conflicts are possible. For example, you have a 3 node cluster and the 2 zones east and west. In such cases the maxSkew might be increased or the whenUnsatisfiable might be set to ScheduleAnyway

See https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/#example-multiple-topologyspreadconstraints for further information.

Cleanup

Remove the topologySpreadConstraint

spec:
[...]
template:
[...]
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
name: django-psql-example

Using Descheduler

Thu, 26 Aug 2021 00:00:00 +0000

Descheduler is a new feature which is GA since OpenShift 4.7. It can be used to evict pods from nodes based on specific strategies. The evicted pod is then scheduled on another node (by the Scheduler) which is more suitable.

This feature can be used when:

nodes are under/over-utilized
pod or node affinity, taints or labels have changed and are no longer valid for a running pod
node failures
pods have been restarted too many times

Pod Placement Series

Please check out other ways of pod placements:

Expand me...

Descheduler Profiles

The following descheduler profiles are known and one or multiple can be configured for the Descheduler. Each profiles enables certain strategies which the Descheduler is leveraging. Since the strategies names are more or less self explaining, I did not add their full description here. Instead detailed information can be found at: Descheduler Profiles

AffinityAndTaints - removes pods that violates affinity and anti-affinity rules or taints
- RemovePodsViolatingInterPodAntiAffinity
- RemovePodsViolatingNodeAffinity
- RemovePodsViolatingNodeTaints
TopologyAndDuplicates - evicts pods which are not evenly spreaded or which are violating the topology domain
- RemovePodsViolatingTopologySpreadConstraint
- RemoveDuplicates
LifecycleAndUtilization - evicts long-running pods to balance resource usage of nodes
- RemovePodsHavingTooManyRestarts - Pods that are restarted more than 100 times
- LowNodeUtilization - removes pods from overutilized nodes.
  - A node is considered underutilized if its usage is below 20% for all thresholds (CPU, memory, and number of pods).
  - A node is considered overutilized if its usage is above 50% for any of the thresholds (CPU, memory, and number of pods).
- PodLifeTime - evicts pods that are too old

Descheduler mechanism

The following rules are followed by the Descheduler to ensure that eviction of pods does not go wild. Therefore the following pods will never be evicted:

pods in openshift-* or kube-system namespaces
pods with priorityClassName equal to system-cluster-critical or system-node-critical
pods which cannot be recreated, for example: static or stand-alone pods/jobs or pods without a replication controller or replica set
pods of a daemon set
pods with local storage
pods which are violating the pod disruption budget

Installing the Descheduler

The Descheduler is not installed by default and must be installed after the cluster has been initiated. This is done by installed the Kube Descheduler Operator.

First we create a separate namespace for our operator, including a label:

oc adm new-project openshift-kube-descheduler-operator
oc label ns/openshift-kube-descheduler-operator openshift.io/cluster-monitoring=true

Then we search for the Kube Descheduler operator and install it, using the newly created namespace:

Figure 1. Install Descheduler Operator

After a few moments the operator will be installed.

You can now create a Descheduler instance either via UI (wizard) or by using the following specification:

apiVersion: operator.openshift.io/v1
kind: KubeDescheduler
metadata:
name: cluster
namespace: openshift-kube-descheduler-operator
spec:
deschedulingIntervalSeconds: 3600 (1)
logLevel: Normal (2)
managementState: Managed
operatorLogLevel: Normal (3)
profiles: (4)
- AffinityAndTaints
- TopologyAndDuplicates
- LifecycleAndUtilization

1	Defines the time interval the descheduler is running. Default is 3600 seconds
2	Defines logging for overall component. Can be Normal, Debug, Trace or TraceAll
3	Defines logging for the operator itself. Can be Normal, Debug, Trace or TraceAll
4	Enables on or multiple profiles the Descheduler should consider