Update Cluster Version using GitOps approach

OpenShift GitOps

- Thomas Jungbauer Thomas Jungbauer ( Lastmod: 2024-06-08 ) - 4 min read

During a GitOps journey at one point, the question arises, how to update a cluster? Nowadays it is very easy to update a cluster using CLI or WebUI, so why bother with GitOps in that case? The reason is simple: Using GitOps you can be sure that all clusters are updated to the correct, required version and the version of each cluster is also managed in Git.

All you need is the channel you want to use and the desired cluster version. Optionally, you can define the exact image SHA. This might be required when you are operating in a restricted environment.

Overview

Since OpenShift 4 it is very easy to update a cluster. It can either be done using the Web UI Administration → Cluster Settings or via the command line using the command oc adm upgrade.

There are 2 main configurations for an upgrade:

  1. channel: This declares the update strategy tied to versions of OpenShift.

  2. The desired version: This is the target version the cluster should be updated to.

To get the current clusterversion you can use the following command:

❯ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.14.1    True        False         4m38s   Cluster version is 4.14.1

Here we can see that the version is currently 4.14.1 and no upgrade is currently progressing.

Now we want to update to the latest possible version.

Where to find the available updates?

Before you start the update, you will need to fetch the possible available updates.

This information can be gathered with the command oc adm upgrade or oc get clusterversion/version -o yaml.

The output will look like the following:

Cluster version is 4.14.1
Upstream is unset, so the cluster will use an appropriate default.
Channel: stable-4.14 (available channels: candidate-4.14, eus-4.14, fast-4.14, stable-4.14, candidate-4.15, fast-4.15, stable-4.15) (1)
Recommended updates: (2)
  VERSION     IMAGE
  4.14.27     quay.io/openshift-release-dev/ocp-release@sha256:4d30b359aa6600a89ed49ce6a9a5fdab54092bcb821a25480fdfbc47e66af9ec
  4.14.26     quay.io/openshift-release-dev/ocp-release@sha256:4fe7d4ccf4d967a309f83118f1a380a656a733d7fcee1dbaf4d51752a6372890
  [...]
1Current Channel and available channels
2List of available updates

The command describes the current channel (stable-4.14) and a list of possible updates. (The list is much longer).

The latest available version is 4.14.27 and the list of possible channels is all the 4.14 channels, but also 4.15 channels. This means we could change the channel here as well.

It is your responsibility to find the correct and supported upgrade paths. Not all upgrades to any version are supported. Also, do not rely on consecutive path numbers, some versions were never available.

Channels

Channels help users to define the timing and level of support for their environment. The following channels typically exist:

  1. fast: This channel is fully supported but not yet fully tested and can be used to quickly update to the latest GA release. For example, when a certain bug is triggered in the current version.

  2. stable: This is the latest stable version. Versions here are added after enough data points have been collected and therefore have some delay.

  3. candidate: This channel offers unsupported early access to specific versions.

  4. eus: All even-numbered versions offer an Externed Update Support (EUS) channel. They allow EUS-to-EUS updates and have a longer support phase.

Helm Chart - update-clusterversion

Now we want to update the version to the latest of the current channel 4.14.27 and change the channel to stable-4.15. The Helm Chart update-clusterversion has been created to help with a cluster update.

It will modify the clusterversion resource.

The required values are quite simple:

channel: stable-4.15
desiredVersion: 4.14.27
image: ''
Be sure that you choose the correct and available updates.

This is everything we need. As soon as this is synchronized into the cluster, the update process will be started by the cluster.

GitOps Synchronization

When we verify the current version in the OpenShift UI, we will see the possibility of an upgrade:

Cluster before the update process starts
Figure 1. Cluster before the update process starts

In OpenShift GitOps we have the Application to start the update process:

Syncing new version
Figure 2. Syncing new version

After a few seconds OpenShift will start with the update:

Progressing Cluster Update
Figure 3. Progressing Cluster Update

This can also be verified via the command line

❯ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.14.1    True        True          95s     Working towards 4.14.27: 116 of 860 done (13% complete), waiting on etcd, kube-apiserver

Eventually, the cluster update process finishes successfully. We are now on version 4.14.27 and using the channel stable-4.15.

We can see that the next upgrade with be to version 4.15.15. This means we can directly upgrade to this version. This is possible because we switched the channel to stable-4.15.

Other channels, like candidate-4.15 might offer different, but not recommended, versions.

❯ oc adm upgrade
Cluster version is 4.14.27

Upstream is unset, so the cluster will use an appropriate default.
Channel: stable-4.15 (available channels: candidate-4.14, candidate-4.15, eus-4.14, fast-4.14, fast-4.15, stable-4.14, stable-4.15)

Recommended updates:

  VERSION     IMAGE
  4.15.15     quay.io/openshift-release-dev/ocp-release@sha256:bb1182cd9001d6811dea8c5823235c17b9a316cce3bb13c51325250c14b46787

What about the SHA for the image?

The SHA for the image field in the ClusterVersion resource is required in certain scenarios to provide a precise reference to the container image that represents the OpenShift version you want to update to.

Such scenarios could be for example:

  1. Offline or Restricted Networks In environments where clusters are running in offline or restricted network conditions, specifying the exact image SHA ensures that the cluster updates to a specific, known image that has been pre-pulled and is available within the network.

  2. Precise Version Control Using the SHA ensures that the exact image version is used for the update, providing a higher level of precision and control.

  3. Custom or Private Registries If you are using custom or private container registries, specifying the image SHA can help avoid ambiguities and ensure that the correct image is pulled from the correct registry.

  4. Specific Compliance or Security Requirements Some regulations might require precise specification of container images, including SHAs, to ensure traceability and verifiability of the software components being deployed.

Conclusion

With this very simple method, it is easy to manage the version of multiple clusters via GitOps. Especially, where there is a bigger cluster fleet it will become essential to ensure which cluster has which version. Disconnected environments can also use the image setting to specify the exact SHA of an available update.

The current Helm Chart is very small and limited. It was created to quickly show the main use case: a simple and straightforward cluster upgrade.

Other possible options that might be required in the future. If you find anything missing, please let me know.

Please note, to always verify which version and channel is available and always consult the official documentation before an upgrade to find the latest release notes.

  1. References: Updating Clusters

See Also:

  • Introducing AdminNetworkPolicies –

    Classic Kubernetes/OpenShift offer a feature called NetworkPolicy that allows users to control the traffic to and from their assigned Namespace. NetworkPolicies are designed to give project owners or tenants the ability to protect their own namespace. Sometimes, however, I worked with customers where the cluster administrators or a dedicated (network) team need to enforce these policies. Since the NetworkPolicy API is namespace-scoped, it is not possible to enforce policies across namespaces. The only solution was to create custom (project) admin and edit roles, and remove the ability of creating, modifying or deleting NetworkPolicy objects. Technically, this is possible and easily done. But shifts the whole network security to cluster administrators. Luckily, this is where AdminNetworkPolicy (ANP) and BaselineAdminNetworkPolicy (BANP) comes into play.

  • Using Kustomize to post render a Helm Chart –

    Lately I came across several issues where a given Helm Chart must be modified after it has been rendered by Argo CD. Argo CD does a helm template to render a Chart. Sometimes, especially when you work with Subcharts or when a specific setting is not yet supported by the Chart, you need to modify it later …​ you need to post-render the Chart. In this very short article, I would like to demonstrate this on a real-live example I had to do. I would like to inject annotations to a Route objects, so that the certificate can be injected. This is done by the cert-utils operator. For the post-rendering the Argo CD repo pod will be extended with a sidecar container, that is watching for the repos and patches them if required.

  • Managing Certificates using GitOps approach –

    The article SSL Certificate Management for OpenShift on AWS explains how to use the Cert-Manager Operator to request and install a new SSL Certificate. This time, I would like to leverage the GitOps approach using the Helm Chart cert-manager I have prepared to deploy the Operator and order new Certificates. I will use an ACME Letsencrypt issuer with a DNS challenge. My domain is hosted at AWS Route 53. However, any other integration can be easily used.

  • Multiple Sources for Applications in Argo CD –

    Argo CD or OpenShift GitOps uses Applications or ApplicationSets to define the relationship between a source (Git) and a cluster. Typically, this is a 1:1 link, which means one Application is using one source to compare the cluster status. This can be a limitation. For example, if you are working with Helm Charts and a Helm repository, you do not want to re-build (or re-release) the whole chart just because you made a small change in the values file that is packaged into the repository. You want to separate the configuration of the chart with the Helm package. The most common scenarios for multiple sources are (see: Argo CD documentation): Your organization wants to use an external/public Helm chart You want to override the Helm values with your own local values You don’t want to clone the Helm chart locally as well because that would lead to duplication and you would need to monitor it manually for upstream changes. This small article describes three different ways with a working example and tries to cover the advantages and disadvantages of each of them. They might be opinionated but some of them proved to be easier to use and manage.

  • Installing OpenShift Logging using GitOps –

    OpenShift Logging is one of the more complex things to install and configure on an OpenShift cluster. Not because the service or Operators are so complex to understand, but because of the dependencies logging has. Besides the logging operator itself, the Loki operator is required, the Loki operator requires access to an object storage, that might be configured or is already available. In this article, I would like to demonstrate the configuration of the full stack using an object storage from OpenShift Data Foundation. This means: Installing the logging operator into the namespace openshift-logging Installing the Loki operator into the namespace openshift-operators-redhat Creating a new BackingStore and BucketClass Generating the Secret for Loki to authenticate against the object storage Configuring the LokiStack resource Configuring the ClusterLogging resource All steps will be done automatically. In case you have S3 storage available, or you are not using OpenShift Data Foundation, the setup will be a bit different. For example, you do not need to create a BackingStore or the Loki authentication Secret.

Copyright © 2020 - 2024 Toni Schmidbauer & Thomas Jungbauer