Update Cluster Version using GitOps approach

OpenShift GitOps

- Thomas Jungbauer Thomas Jungbauer ( Lastmod: 2024-06-08 ) - 4 min read

During a GitOps journey at one point, the question arises, how to update a cluster? Nowadays it is very easy to update a cluster using CLI or WebUI, so why bother with GitOps in that case? The reason is simple: Using GitOps you can be sure that all clusters are updated to the correct, required version and the version of each cluster is also managed in Git.

All you need is the channel you want to use and the desired cluster version. Optionally, you can define the exact image SHA. This might be required when you are operating in a restricted environment.

Overview

Since OpenShift 4 it is very easy to update a cluster. It can either be done using the Web UI Administration → Cluster Settings or via the command line using the command oc adm upgrade.

There are 2 main configurations for an upgrade:

  1. channel: This declares the update strategy tied to versions of OpenShift.

  2. The desired version: This is the target version the cluster should be updated to.

To get the current clusterversion you can use the following command:

❯ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.14.1    True        False         4m38s   Cluster version is 4.14.1

Here we can see that the version is currently 4.14.1 and no upgrade is currently progressing.

Now we want to update to the latest possible version.

Where to find the available updates?

Before you start the update, you will need to fetch the possible available updates.

This information can be gathered with the command oc adm upgrade or oc get clusterversion/version -o yaml.

The output will look like the following:

Cluster version is 4.14.1
Upstream is unset, so the cluster will use an appropriate default.
Channel: stable-4.14 (available channels: candidate-4.14, eus-4.14, fast-4.14, stable-4.14, candidate-4.15, fast-4.15, stable-4.15) (1)
Recommended updates: (2)
  VERSION     IMAGE
  4.14.27     quay.io/openshift-release-dev/ocp-release@sha256:4d30b359aa6600a89ed49ce6a9a5fdab54092bcb821a25480fdfbc47e66af9ec
  4.14.26     quay.io/openshift-release-dev/ocp-release@sha256:4fe7d4ccf4d967a309f83118f1a380a656a733d7fcee1dbaf4d51752a6372890
  [...]
1Current Channel and available channels
2List of available updates

The command describes the current channel (stable-4.14) and a list of possible updates. (The list is much longer).

The latest available version is 4.14.27 and the list of possible channels is all the 4.14 channels, but also 4.15 channels. This means we could change the channel here as well.

It is your responsibility to find the correct and supported upgrade paths. Not all upgrades to any version are supported. Also, do not rely on consecutive path numbers, some versions were never available.

Channels

Channels help users to define the timing and level of support for their environment. The following channels typically exist:

  1. fast: This channel is fully supported but not yet fully tested and can be used to quickly update to the latest GA release. For example, when a certain bug is triggered in the current version.

  2. stable: This is the latest stable version. Versions here are added after enough data points have been collected and therefore have some delay.

  3. candidate: This channel offers unsupported early access to specific versions.

  4. eus: All even-numbered versions offer an Externed Update Support (EUS) channel. They allow EUS-to-EUS updates and have a longer support phase.

Helm Chart - update-clusterversion

Now we want to update the version to the latest of the current channel 4.14.27 and change the channel to stable-4.15. The Helm Chart update-clusterversion has been created to help with a cluster update.

It will modify the clusterversion resource.

The required values are quite simple:

channel: stable-4.15
desiredVersion: 4.14.27
image: ''
Be sure that you choose the correct and available updates.

This is everything we need. As soon as this is synchronized into the cluster, the update process will be started by the cluster.

GitOps Synchronization

When we verify the current version in the OpenShift UI, we will see the possibility of an upgrade:

Cluster before the update process starts
Figure 1. Cluster before the update process starts

In OpenShift GitOps we have the Application to start the update process:

Syncing new version
Figure 2. Syncing new version

After a few seconds OpenShift will start with the update:

Progressing Cluster Update
Figure 3. Progressing Cluster Update

This can also be verified via the command line

❯ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.14.1    True        True          95s     Working towards 4.14.27: 116 of 860 done (13% complete), waiting on etcd, kube-apiserver

Eventually, the cluster update process finishes successfully. We are now on version 4.14.27 and using the channel stable-4.15.

We can see that the next upgrade with be to version 4.15.15. This means we can directly upgrade to this version. This is possible because we switched the channel to stable-4.15.

Other channels, like candidate-4.15 might offer different, but not recommended, versions.

❯ oc adm upgrade
Cluster version is 4.14.27

Upstream is unset, so the cluster will use an appropriate default.
Channel: stable-4.15 (available channels: candidate-4.14, candidate-4.15, eus-4.14, fast-4.14, fast-4.15, stable-4.14, stable-4.15)

Recommended updates:

  VERSION     IMAGE
  4.15.15     quay.io/openshift-release-dev/ocp-release@sha256:bb1182cd9001d6811dea8c5823235c17b9a316cce3bb13c51325250c14b46787

What about the SHA for the image?

The SHA for the image field in the ClusterVersion resource is required in certain scenarios to provide a precise reference to the container image that represents the OpenShift version you want to update to.

Such scenarios could be for example:

  1. Offline or Restricted Networks In environments where clusters are running in offline or restricted network conditions, specifying the exact image SHA ensures that the cluster updates to a specific, known image that has been pre-pulled and is available within the network.

  2. Precise Version Control Using the SHA ensures that the exact image version is used for the update, providing a higher level of precision and control.

  3. Custom or Private Registries If you are using custom or private container registries, specifying the image SHA can help avoid ambiguities and ensure that the correct image is pulled from the correct registry.

  4. Specific Compliance or Security Requirements Some regulations might require precise specification of container images, including SHAs, to ensure traceability and verifiability of the software components being deployed.

Conclusion

With this very simple method, it is easy to manage the version of multiple clusters via GitOps. Especially, where there is a bigger cluster fleet it will become essential to ensure which cluster has which version. Disconnected environments can also use the image setting to specify the exact SHA of an available update.

The current Helm Chart is very small and limited. It was created to quickly show the main use case: a simple and straightforward cluster upgrade.

Other possible options that might be required in the future. If you find anything missing, please let me know.

Please note, to always verify which version and channel is available and always consult the official documentation before an upgrade to find the latest release notes.

  1. References: Updating Clusters

See Also:

  • Managing Certificates using GitOps approach –

    The article SSL Certificate Management for OpenShift on AWS explains how to use the Cert-Manager Operator to request and install a new SSL Certificate. This time, I would like to leverage the GitOps approach using the Helm Chart cert-manager I have prepared to deploy the Operator and order new Certificates. I will use an ACME Letsencrypt issuer with a DNS challenge. My domain is hosted at AWS Route 53. However, any other integration can be easily used.

  • Multiple Sources for Applications in Argo CD –

    Argo CD or OpenShift GitOps uses Applications or ApplicationSets to define the relationship between a source (Git) and a cluster. Typically, this is a 1:1 link, which means one Application is using one source to compare the cluster status. This can be a limitation. For example, if you are working with Helm Charts and a Helm repository, you do not want to re-build (or re-release) the whole chart just because you made a small change in the values file that is packaged into the repository. You want to separate the configuration of the chart with the Helm package. The most common scenarios for multiple sources are (see: Argo CD documentation): Your organization wants to use an external/public Helm chart You want to override the Helm values with your own local values You don’t want to clone the Helm chart locally as well because that would lead to duplication and you would need to monitor it manually for upstream changes. This small article describes three different ways with a working example and tries to cover the advantages and disadvantages of each of them. They might be opinionated but some of them proved to be easier to use and manage.

  • Installing OpenShift Logging using GitOps –

    OpenShift Logging is one of the more complex things to install and configure on an OpenShift cluster. Not because the service or Operators are so complex to understand, but because of the dependencies logging has. Besides the logging operator itself, the Loki operator is required, the Loki operator requires access to an object storage, that might be configured or is already available. In this article, I would like to demonstrate the configuration of the full stack using an object storage from OpenShift Data Foundation. This means: Installing the logging operator into the namespace openshift-logging Installing the Loki operator into the namespace openshift-operators-redhat Creating a new BackingStore and BucketClass Generating the Secret for Loki to authenticate against the object storage Configuring the LokiStack resource Configuring the ClusterLogging resource All steps will be done automatically. In case you have S3 storage available, or you are not using OpenShift Data Foundation, the setup will be a bit different. For example, you do not need to create a BackingStore or the Loki authentication Secret.

  • Configure Buckets in MinIO using GitOps –

    MinIO is a simple, S3-compatible object storage, built for high-performance and large-scale environments. It can be installed as an Operator to Openshift. In addition, to a command line tool, it provides a WebUI where all settings can be done, especially creating and configuring new buckets. Currently, this is not possible in a declarative GitOps-friendly way. Therefore, I created the Helm chart minio configurator, that will start a Kubernetes Job, which will take care of the configuration. Honestly, when I say I have created it, the truth is, that it is based on an existing MinIO Chart by Bitnami, that does much more than just set up a bucket. I took out the bucket configuration part, streamlined it a bit and added some new features, which I required. This article shall explain how to achieve this.

  • Setup & Configure Advanced Cluster Security using GitOps –

    Today I want to demonstrate the deployment and configuration of Advanced Cluster Security (ACS) using a GitOps approach. The required operator shall be installed, verified if it is running and then ACS shall be initialized. This initialization contains the deployment of several components: Central - as UI and as a main component of ACS SecuredClusters - installs a Scanner, Controller pods etc. Console link into OpenShift UI - to directly access the ACS Central UI Job to create an initialization bundle to install the Secured Cluster Job to configure authentication using OpenShift Let’s start …​

Copyright © 2020 - 2024 Toni Schmidbauer & Thomas Jungbauer