Summary: GitOps Episodes

In [Ep.18] I recommended ACM policies for the audited baseline of a cluster and Argo CD for everything that needs rendering. [Ep.19] showed how ACM feeds Argo CD with clusters and the different models that can be used in Argo CD. This part turns the integration around: Argo CD deploys ACM policies, and a policy bootstraps Argo CD on every new cluster.

The result is a setup where the policies live in Git, are reviewed in pull requests like any other change, and ACM does what it is good at: distribution, enforcement and compliance reporting.

In [Ep.18] I compared OpenShift GitOps and Advanced Cluster Management and ended with the statement that they are layers, not competitors. ACM knows the fleet and Argo CD delivers the content. This part is about the glue between the two: how ACM registers its clusters in Argo CD, and the three ways an Argo CD Application can reach a managed cluster: the push model, the pull model and the Argo CD Agent.

For each of them I will explain what it actually is, how the data flows, and how to set it up step by step.

At customer engagements where I support the implementation of a GitOps process, I often hear the question: "Should we use OpenShift GitOps (Argo CD) or Advanced Cluster Management (Open Cluster Management)?" The honest answer is: it is the wrong question. Both tools solve different problems, they overlap in one specific area, and in larger environments with numerous clusters they are designed to work together.

In this article I will try to explain what each tool is (and also what it is not), where they overlap, how they integrate, and I will walk through an example, so that you can decide for yourself which tool fits which job.

Argo CD is very good at one thing: making sure that whatever is stored in Git is what is running on the cluster. It compares both things and updates the cluster accordingly. It is not good at noticing that a CI pipeline has just published a new container image (into the registry). That is the gap between Continuous Integration and Continuous Delivery, and most of us have closed it with a small piece of glue: a pipeline task that clones the configuration repository, patches a tag and pushes the change. I did exactly that in Step 8 of the Secure Supply Chain series with a Tekton task.

With OpenShift GitOps 1.21 the Argo CD Image Updater became generally available. This controller aims to close the gap between CI and CD.

Argo CD is a powerhouse for GitOps. It is easy to get started with and helps you manage clusters and applications. As the number of clusters, applications, and repositories grows, performance can suffer: refreshes take longer, the UI may feel sluggish, and the experience turns frustrating.

This article is about performance tuning: which options exist, which scenarios matter, and which settings might help in your environment.

OpenShift GitOps based on Argo CD is a powerful tool to manage the infrastructure and applications on an OpenShift cluster. Initially, there were two ways of deployment: centralized and decentralized (or distributed). Both methods had their own advantages and disadvantages. The choice was mainly between scalability and centralization. With OpenShift GitOps v1.19, the Argo CD Agent was finally generally available. This agent tries to solve this problem by bringing the best of both worlds together. In this quite long article, I will show you how to install and configure the Argo CD Agent with OpenShift GitOps using hub and spoke architecture.

When working with Argo CD at scale, you often find yourself creating similar Application manifests repeatedly. Each application needs the same basic structure but with different configurations for source repositories, destinations, and sync policies. Additionally, managing namespace metadata becomes tricky when you need to conditionally control whether Argo CD should manage namespace metadata based on sync options.

In this article, I’ll walk you through a reusable Helm template that solves these challenges by providing a flexible, DRY (Don’t Repeat Yourself) approach to creating Argo CD Applications. This template is available in my public Helm Chart library and can easily be used by anyone.

During my day-to-day business, I am discussing the following setup with many customers: Configure App-of-Apps. Here I try to explain how I use an ApplicationSet that watches over a folder in Git and automatically adds a new Argo CD Application whenever a new folder is found. This works great, but there is a catch: The ApplicationSet uses the same Namespace default for all Applications. This is not always desired, especially when you have different teams working on different Applications.

Recently I was asked by the customer if this can be fixed and if it is possible to define different Namespaces for each Application. The answer is yes, and I would like to show you how to do this.

Lately I came across several issues where a given Helm Chart must be modified after it has been rendered by Argo CD. Argo CD does a helm template to render a Chart. Sometimes, especially when you work with Subcharts or when a specific setting is not yet supported by the Chart, you need to modify it later …​ you need to post-render the Chart.

In this very short article, I would like to demonstrate this on a real-live example I had to do. I would like to inject annotations to a Route objects, so that the certificate can be injected. This is done by the cert-utils operator. For the post-rendering the Argo CD repo pod will be extended with a sidecar container, that is watching for the repos and patches them if required.

The article SSL Certificate Management for OpenShift on AWS explains how to use the Cert-Manager Operator to request and install a new SSL Certificate. This time, I would like to leverage the GitOps approach using the Helm Chart cert-manager I have prepared to deploy the Operator and order new Certificates.

I will use an ACME Letsencrypt issuer with a DNS challenge. My domain is hosted at AWS Route 53.

However, any other integration can be easily used.

During a GitOps journey at one point, the question arises, how to update a cluster? Nowadays it is very easy to update a cluster using CLI or WebUI, so why bother with GitOps in that case? The reason is simple: Using GitOps you can be sure that all clusters are updated to the correct, required version and the version of each cluster is also managed in Git.

All you need is the channel you want to use and the desired cluster version. Optionally, you can define the exact image SHA. This might be required when you are operating in a restricted environment.

Argo CD or OpenShift GitOps uses Applications or ApplicationSets to define the relationship between a source (Git) and a cluster. Typically, this is a 1:1 link, which means one Application is using one source to compare the cluster status. This can be a limitation. For example, if you are working with Helm Charts and a Helm repository, you do not want to re-build (or re-release) the whole chart just because you made a small change in the values file that is packaged into the repository. You want to separate the configuration of the chart with the Helm package.

The most common scenarios for multiple sources are (see: Argo CD documentation):

This small article describes three different ways with a working example and tries to cover the advantages and disadvantages of each of them. They might be opinionated but some of them proved to be easier to use and manage.

OpenShift Logging is one of the more complex things to install and configure on an OpenShift cluster. Not because the service or Operators are so complex to understand, but because of the dependencies logging has. Besides the logging operator itself, the Loki operator is required, the Loki operator requires access to an object storage, that might be configured or is already available.

In this article, I would like to demonstrate the configuration of the full stack using an object storage from OpenShift Data Foundation. This means:

All steps will be done automatically. In case you have S3 storage available, or you are not using OpenShift Data Foundation, the setup will be a bit different. For example, you do not need to create a BackingStore or the Loki authentication Secret.

MinIO is a simple, S3-compatible object storage, built for high-performance and large-scale environments. It can be installed as an Operator to Openshift. In addition, to a command line tool, it provides a WebUI where all settings can be done, especially creating and configuring new buckets. Currently, this is not possible in a declarative GitOps-friendly way. Therefore, I created the Helm chart minio configurator, that will start a Kubernetes Job, which will take care of the configuration.

Honestly, when I say I have created it, the truth is, that it is based on an existing MinIO Chart by Bitnami, that does much more than just set up a bucket. I took out the bucket configuration part, streamlined it a bit and added some new features, which I required.

This article shall explain how to achieve this.

Today I want to demonstrate the deployment and configuration of Advanced Cluster Security (ACS) using a GitOps approach. The required operator shall be installed, verified if it is running and then ACS shall be initialized. This initialization contains the deployment of several components:

Let’s start …​

In the previous articles, we have discussed the Git repository folder structure and the configuration of the App-Of-Apps. Now it is time to deploy our first configuration. One of the first things I usually deploy is the Compliance Operator. This Operator is recommended for any cluster and can be deployed without any addition to the Subscription.

In this article, I will describe how it is installed and how the Helm Chart is configured.

In the article Install GitOps to the cluster OpenShift GitOps is deployed using a shell script. This should be the very first installation and the only deployment that is done manually on a cluster. This procedure automatically installs the so-called App-of-Apps named Argo CD Resources Manager which is responsible for all further Argo CD Applications and ApplicationSets. No other configuration should be done manually if possible.

This article will demonstrate how to configure the App-of-Apps in an easy and declarative way, using ApplicationSet mainly.

„If it is not in GitOps, it does not exist“ - is a mantra I hear quite often and also try to practice at customer engagements. The idea is to have Git as the only source of truth on what happens inside the environment. That said, Everything as Code is a practice that treats every aspect of the system as a code. Storing this code in Git provides a shared understanding, traceability and repeatability of changes.

While there are many articles about how to get GitOps into the deployment process of applications, this one rather sets the focus on the cluster configuration and tasks system administrators usually have to do.

One of the most popular questions asked before adopting the GitOps approach is how to deploy an application to different environments (Test, Dev, Production, etc.) in a safe and repeatable way.

Each organisation has different requirements, and the choice will depend on a multitude of factors that also include non-technical aspects.

Therefore, it is important to state: "There is no unique “right” way, there are common practices".

When managing one or more clusters, the question arises as to how cluster configurations and applications can be installed securely, regularly, and in the same way. This is where the so-called GitOps approach helps, according to the mantra: "If it is not in Git, it does not exist".

The idea is to have Git as the only source of truth on what happens inside the environment. While there are many articles about how to get GitOps into the deployment process of applications, this series of articles tries to set the focus on the cluster configuration and tasks system administrators usually have to do, for example: Setup an Operator.