Using Descheduler

OpenShift Pod Placement

- Thomas Jungbauer Thomas Jungbauer ( Lastmod: 2024-05-08 ) - 2 min read

Descheduler is a new feature which is GA since OpenShift 4.7. It can be used to evict pods from nodes based on specific strategies. The evicted pod is then scheduled on another node (by the Scheduler) which is more suitable.

This feature can be used when:

  • nodes are under/over-utilized

  • pod or node affinity, taints or labels have changed and are no longer valid for a running pod

  • node failures

  • pods have been restarted too many times

Pod Placement Series

Please check out other ways of pod placements:

Expand me...

Descheduler Profiles

The following descheduler profiles are known and one or multiple can be configured for the Descheduler. Each profiles enables certain strategies which the Descheduler is leveraging. Since the strategies names are more or less self explaining, I did not add their full description here. Instead detailed information can be found at: Descheduler Profiles

  • AffinityAndTaints - removes pods that violates affinity and anti-affinity rules or taints

    • RemovePodsViolatingInterPodAntiAffinity

    • RemovePodsViolatingNodeAffinity

    • RemovePodsViolatingNodeTaints

  • TopologyAndDuplicates - evicts pods which are not evenly spreaded or which are violating the topology domain

    • RemovePodsViolatingTopologySpreadConstraint

    • RemoveDuplicates

  • LifecycleAndUtilization - evicts long-running pods to balance resource usage of nodes

    • RemovePodsHavingTooManyRestarts - Pods that are restarted more than 100 times

    • LowNodeUtilization - removes pods from overutilized nodes.

      • A node is considered underutilized if its usage is below 20% for all thresholds (CPU, memory, and number of pods).

      • A node is considered overutilized if its usage is above 50% for any of the thresholds (CPU, memory, and number of pods).

    • PodLifeTime - evicts pods that are too old

Descheduler mechanism

The following rules are followed by the Descheduler to ensure that eviction of pods does not go wild. Therefore the following pods will never be evicted:

  • pods in openshift-* or kube-system namespaces

  • pods with priorityClassName equal to system-cluster-critical or system-node-critical

  • pods which cannot be recreated, for example: static or stand-alone pods/jobs or pods without a replication controller or replica set

  • pods of a daemon set

  • pods with local storage

  • pods which are violating the pod disruption budget

Installing the Descheduler

The Descheduler is not installed by default and must be installed after the cluster has been initiated. This is done by installed the Kube Descheduler Operator.

First we create a separate namespace for our operator, including a label:

oc adm new-project openshift-kube-descheduler-operator

oc label ns/openshift-kube-descheduler-operator openshift.io/cluster-monitoring=true

Then we search for the Kube Descheduler operator and install it, using the newly created namespace:

Install Descheduler Operator
Figure 1. Install Descheduler Operator

After a few moments the operator will be installed.

You can now create a Descheduler instance either via UI (wizard) or by using the following specification:

apiVersion: operator.openshift.io/v1
kind: KubeDescheduler
metadata:
  name: cluster
  namespace: openshift-kube-descheduler-operator
spec:
  deschedulingIntervalSeconds: 3600 (1)
  logLevel: Normal (2)
  managementState: Managed
  operatorLogLevel: Normal (3)
  profiles: (4)
  - AffinityAndTaints
  - TopologyAndDuplicates
  - LifecycleAndUtilization
1Defines the time interval the descheduler is running. Default is 3600 seconds
2Defines logging for overall component. Can be Normal, Debug, Trace or TraceAll
3Defines logging for the operator itself. Can be Normal, Debug, Trace or TraceAll
4Enables on or multiple profiles the Descheduler should consider

See Also:

  • Single log out from Keycloak and OpenShift –

    The following 1-minute article is a follow-up to my previous article about how to use Keycloak as an authentication provider for OpenShift. In this article, I will show you how to configure Keycloak and OpenShift for Single Log Out (SLO). This means that when you log out from Keycloak, you will also be logged out from OpenShift automatically. This requires some additional configuration in Keycloak and OpenShift, but it is not too complicated.

  • Step by Step - Using Keycloak Authentication in OpenShift –

    I was recently asked about how to use Keycloak as an authentication provider for OpenShift. How to install Keycloak using the Operator and how to configure Keycloak and OpenShift so that users can log in to OpenShift using OpenID. I have to admit that the exact steps are not easy to find, so I decided to write a blog post about it, describing each step in detail. This time I will not use GitOps, but the OpenShift and Keycloak Web Console to show the steps, because before we put it into GitOps, we need to understand what is actually happening. This article tries to explain every step required so that a user can authenticate to OpenShift using Keycloak as an Identity Provider (IDP) and that Groups from Keycloak are imported into OpenShift. This article does not cover a production grade installation of Keycloak, but only a test installation, so you can see how it works. For production, you might want to consider a proper database (maybe external, but at least with a backup), high availability, etc.).

  • Using ApplicationSet with Matrix Generator and define individual Namespaces –

    During my day-to-day business, I am discussing the following setup with many customers: Configure App-of-Apps. Here I try to explain how I use an ApplicationSet that watches over a folder in Git and automatically adds a new Argo CD Application whenever a new folder is found. This works great, but there is a catch: The ApplicationSet uses the same Namespace default for all Applications. This is not always desired, especially when you have different teams working on different Applications. Recently I was asked by the customer if this can be fixed and if it is possible to define different Namespaces for each Application. The answer is yes, and I would like to show you how to do this.

  • Introducing AdminNetworkPolicies –

    Classic Kubernetes/OpenShift offer a feature called NetworkPolicy that allows users to control the traffic to and from their assigned Namespace. NetworkPolicies are designed to give project owners or tenants the ability to protect their own namespace. Sometimes, however, I worked with customers where the cluster administrators or a dedicated (network) team need to enforce these policies. Since the NetworkPolicy API is namespace-scoped, it is not possible to enforce policies across namespaces. The only solution was to create custom (project) admin and edit roles, and remove the ability of creating, modifying or deleting NetworkPolicy objects. Technically, this is possible and easily done. But shifts the whole network security to cluster administrators. Luckily, this is where AdminNetworkPolicy (ANP) and BaselineAdminNetworkPolicy (BANP) comes into play.

  • Using Kustomize to post render a Helm Chart –

    Lately I came across several issues where a given Helm Chart must be modified after it has been rendered by Argo CD. Argo CD does a helm template to render a Chart. Sometimes, especially when you work with Subcharts or when a specific setting is not yet supported by the Chart, you need to modify it later …​ you need to post-render the Chart. In this very short article, I would like to demonstrate this on a real-live example I had to do. I would like to inject annotations to a Route objects, so that the certificate can be injected. This is done by the cert-utils operator. For the post-rendering the Argo CD repo pod will be extended with a sidecar container, that is watching for the repos and patches them if required.

Copyright © 2020 - 2025 Toni Schmidbauer & Thomas Jungbauer