Advanced Cluster Management

In [Ep.18] I recommended ACM policies for the audited baseline of a cluster and Argo CD for everything that needs rendering. [Ep.19] showed how ACM feeds Argo CD with clusters and the different models that can be used in Argo CD. This part turns the integration around: Argo CD deploys ACM policies, and a policy bootstraps Argo CD on every new cluster.

The result is a setup where the policies live in Git, are reviewed in pull requests like any other change, and ACM does what it is good at: distribution, enforcement and compliance reporting.

In [Ep.18] I compared OpenShift GitOps and Advanced Cluster Management and ended with the statement that they are layers, not competitors. ACM knows the fleet and Argo CD delivers the content. This part is about the glue between the two: how ACM registers its clusters in Argo CD, and the three ways an Argo CD Application can reach a managed cluster: the push model, the pull model and the Argo CD Agent.

For each of them I will explain what it actually is, how the data flows, and how to set it up step by step.

At customer engagements where I support the implementation of a GitOps process, I often hear the question: "Should we use OpenShift GitOps (Argo CD) or Advanced Cluster Management (Open Cluster Management)?" The honest answer is: it is the wrong question. Both tools solve different problems, they overlap in one specific area, and in larger environments with numerous clusters they are designed to work together.

In this article I will try to explain what each tool is (and also what it is not), where they overlap, how they integrate, and I will walk through an example, so that you can decide for yourself which tool fits which job.

In my last article about the Compliance Operator, Compliance Operator: Writing Your Own Checks with CEL and CustomRules, we discussed CustomRule and how they can enhance the Compliance Operator. CustomRules are good for defining your own policies, for example to check whether a namespace is created with the correct labels. But they cannot stop someone from changing something on the cluster.

The Compliance Operator can detect violations, but it cannot prevent them. This is where ValidatingAdmissionPolicy comes in. It is a native Kubernetes feature that runs directly inside the kube-apiserver. It uses the same language as the CustomRules, CEL, and it can be used to prevent violations from happening in the first place.

Five years ago I first tested and described how to achieve compliance for a Kubernetes cluster in my article Compliance Operator. Many years passed since then, but up to this day the Compliance Operator is still one of the most important tools I recommend to my customers and one of the first that is installed on a new cluster. Since the first release of the Operator, it has been constantly improved and extended, for example by adding your own guardrails to the cluster to make sure that the cluster is compliant with your organisation’s requirements.

Today I would like to show you how to write your own checks with the CEL scanner and the CustomRule resource.

Argo CD is very good at one thing: making sure that whatever is stored in Git is what is running on the cluster. It compares both things and updates the cluster accordingly. It is not good at noticing that a CI pipeline has just published a new container image (into the registry). That is the gap between Continuous Integration and Continuous Delivery, and most of us have closed it with a small piece of glue: a pipeline task that clones the configuration repository, patches a tag and pushes the change. I did exactly that in Step 8 of the Secure Supply Chain series with a Tekton task.

With OpenShift GitOps 1.21 the Argo CD Image Updater became generally available. This controller aims to close the gap between CI and CD.

With authentication and secrets engines configured, the next step is enabling applications to consume secrets. This article covers three integration patterns on OpenShift:

Answering the question "who can do what on this cluster?" usually means juggling oc commands, YAML dumps, and several console pages. RoleBindings live in namespaces, ClusterRoleBindings are cluster-scoped, OpenShift adds Users and Groups on top of Kubernetes subjects, and SecurityContextConstraints have their own authorisation model. For a quick audit before a change window, that is a lot of clicking and I always struggled to find a quick and reliable way to get and overview. For example, I always wanted to see immediately which users have cluster-admin rights or which service accounts are on the privileged SCC.

The RBAC Overview is a dynamic OpenShift Console plugin that pulls these views into one place. This article walks through what it does, how to install or test it, and how I use the tabs in practice. This open source plugin is available on GitHub

Ansible Automation Platform (AAP) comes with an elaborate credential management solution. All credentials within AAP are stored encrypted in a PostgreSQL database. But the keys used for encrypting database fields are currently stored on the machine(s) running AAP. In this blog post, we show we created an auditd rule to get at least some kind of monitoring if those keys are accessed.

We wanted to use renovate bot to automatically update collection dependencies for one of our Ansible playbook repositories. This blog post describes the basics of renovate, how we configured it and how we actually run renovate locally.

Argo CD is a powerhouse for GitOps. It is easy to get started with and helps you manage clusters and applications. As the number of clusters, applications, and repositories grows, performance can suffer: refreshes take longer, the UI may feel sluggish, and the experience turns frustrating.

This article is about performance tuning: which options exist, which scenarios matter, and which settings might help in your environment.

In the blog post Creating a RHEL 10 VM on macOS with bootc-image-builder we described how to quickly create a bootc (image mode) base image on macOS. Now we would like to create a customized standard RHEL image.

Yes, we have Apple machines in our lab because why not. So we needed a RHEL 10 VM to set up Ansible Automation Platform, which seems to support AARCH64 and Red Hat Enterprise Linux 10.

Secrets engines are one of the most important concepts in OpenBao. Part 8 covered the KV secrets engine; this article covers the PKI secrets engine and its integration with cert-manager on Kubernetes and OpenShift.