GitOps on TechBlog about OpenShift/Ansible/Satellite and much more

GitOps Catalog

Mon, 22 Dec 2025 00:00:00 +0000

The GitOps Catalog page provides an interactive visualization of all available ArgoCD applications from the openshift-clusterconfig-gitops repository. Check out the page GitOps Catalog for more details.

GitOps Catalog

Mon, 22 Dec 2025 00:00:00 +0000

This page provides an interactive visualization of all available ArgoCD applications from the openshift-clusterconfig-gitops repository.

The repository demonstrates the usage of OpenShift GitOps with (mainly) Helm Charts that I use for my own clusters. As easy Secret Management I have Sealed Secrets. It focuses on main cluster configuration using a GitOps approach. Some of the charts or configurations are discussied in my blog posts. Please refer to the different GitOps blog posts for more details and to understand why it is done this way.

How to Use

Browse the catalog below to discover available configurations
Filter by category using the buttons (Security, Observability, Storage, etc.)
Search for specific applications using the search box
Click on "Blog" where available to read detailed documentation or "Source" to view the code on GitHub

Quick Start

# Clone the repository
git clone https://github.com/tjungbauer/openshift-clusterconfig-gitops.git
cd openshift-clusterconfig-gitops
# Initialize GitOps (deploys OpenShift GitOps operator and App of Apps)
./init_GitOps.sh

Most Helm Charts used in this repository can be found at charts.stderr.at

OpenShift GitOps Configuration Catalog

Available ArgoCD applications for cluster configuration via openshift-clusterconfig-gitops

Last updated: 2026-02-02 40 Applications

Platform Configuration v1.0.0

deploy-base-operators

Deploy base operators WITHOUT any additional configuration. This simply adds the Subscription and leave further configuration to you or another Chart.

clusters/all/base-operators

Source

Platform Configuration v1.0.5

proj-onboarding

This Chart shall deploy namespaces and their depending resources, like NetworkPolicies or Quotas etc.

clusters/all/project-onboarding

Source

Platform Configuration v1.0.1

gitops-deployment

Deployment of a GitOps for Application Management

clusters/management-cluster/applications-gitops

Dependencies / Requirements

Helm Charts:

openshift-gitops

Source

Platform Configuration v1.0.1

clusterbranding

Deploys Cluster Branding such as custom Login Page or Logo etc.

clusters/management-cluster/branding

Dependencies / Requirements

Helm Charts:

generic-cluster-config

secure-supply-chain

Source

Platform Configuration v1.0.1

clusterbranding

Deploys Cluster Branding such as custom Login Page or Logo etc.

clusters/management-cluster/branding-bottom

Dependencies / Requirements

Helm Charts:

generic-cluster-config

secure-supply-chain

Source

Security & Compliance v2.0.1

cert-manager

Setup and configure the cert Manager operator

clusters/management-cluster/cert-manager

Dependencies / Requirements

Helm Charts:

cert-manager

compliance

Source

Security & Compliance v1.0.3

clusterconfig-apiserver

Enables ETCD encryption, customer certificate and audit-profile for APIServer

clusters/management-cluster/clusterconfig-apiserver

Dependencies / Requirements

Helm Charts:

generic-cluster-config cert-manager

security

Source

Platform Configuration v1.0.0

enable-etcd-backup

Create a CronJob that performs ETCD Backup and stores the backup to a PV.

clusters/management-cluster/etcd-backup

Dependencies / Requirements

Helm Charts:

etcd-backup

Source

Platform Configuration v1.0.1

clusterconfig

Installs cluster configuration which is usually valid for ALL clusters

clusters/management-cluster/generic-clusterconfig

Dependencies / Requirements

Helm Charts:

generic-cluster-config

generic

Source

Security & Compliance v1.0.0

cert-manager

Setup and configure the cert Manager operator

clusters/management-cluster/idp

Dependencies / Requirements

Helm Charts:

generic-cluster-config

compliance

Source

Platform Configuration v1.0.0

Ingress Controller

Configures the OpenShift IngressController object with replicas, nodeSelector, and tolerations for infrastructure node placement.

clusters/management-cluster/ingresscontroller

ingress routing infrastructure

Source

Platform Configuration v1.0.0

install-cyclonedx

Install Cyclonedx to store SBOMs

clusters/management-cluster/install-cyclonedx

Dependencies / Requirements

Helm Charts:

cyclonedx

secure-supply-chain

Source

Storage & Data v1.0.0

internal-registry

Configures the internal OpenShift registry

clusters/management-cluster/internal-registry

Source

Platform Configuration v1.0.0

node-labels

Manage the labelling of the nodes using openshift-gitops 1.6+ and Server Side Apply

clusters/management-cluster/node-configuration

Source

Security & Compliance v1.0.4

setup-acm

Deploys Advanced Cluster Managment (ACM) on target cluster.

clusters/management-cluster/setup-acm

Dependencies / Requirements

Helm Charts:

rhacm-setup

acs

Source

Security & Compliance v1.0.0

setup-acs

Deploys Advanced Cluster Security (ACS) on target cluster. If enabled Central will be deployed too.

clusters/management-cluster/setup-acs

Dependencies / Requirements

Helm Charts:

rhacs-setup

security

Source

Security & Compliance v1.0.0

setup-acs-backup

Deploys Backup for Advanced Cluster Security (ACS) on target cluster.

clusters/management-cluster/setup-acs-backup

Dependencies / Requirements

Helm Charts:

rhacs-backup

security

Source

Security & Compliance v1.0.0

setup-cert-utils-test

Test Deployment of Cert Utils Operator. This is used to test new operator deployment methods.

clusters/management-cluster/setup-cert-utils-test

sandbox

Source

Observability & Monitoring v1.0.0

setup-cluster-observability-operator

Installs the Cluster Observability Operator. Currently it simply installs the Operator and its namespace.

clusters/management-cluster/setup-cluster-observability-operator

observability

Source

Security & Compliance v1.0.2

setup-compliance-operator

Deploy and configure the Compliance Operator

clusters/management-cluster/setup-compliance-operator

Dependencies / Requirements

Helm Charts:

compliance-operator-full-stack

compliance

Source

Security & Compliance v1.0.6

setup-container-security-operator

Setup the Quay Container Security Operator.

clusters/management-cluster/setup-container-security-operator

acs

Source

Platform Configuration v1.0.0

setup-cost-management-operator

Setup Cost Management Operator

clusters/management-cluster/setup-cost-management-operator

Dependencies / Requirements

Helm Charts:

cost-management

Source

Storage & Data v1.0.0

setup-crunchy-postgres

Deploy and configure Crunchy Postgres Operator and Postgres clusters

clusters/management-cluster/setup-crunchy-postgres

crunchy

Source

Development Tools v1.0.1

setup-dev-spaces

Deploy Dev Spaces Operator

clusters/management-cluster/setup-dev-spaces

dev

Source

Security & Compliance v1.0.1

setup-file-integrity-operator

Setup File Integrity Operator

clusters/management-cluster/setup-file-integrity-operator

Dependencies / Requirements

Helm Charts:

file-integrity-operator

Source

Observability & Monitoring v1.0.0

setup-grafana

Deploy the Grafana Operator

clusters/management-cluster/setup-grafana

grafana

Source

Observability & Monitoring v1.0.1

setup-loki-operator

Installs Loki Operator. Configuration should be done by the appropriate service that wants to use Loki. For example openshift-logging

clusters/management-cluster/setup-loki-operator

loki

Source

Observability & Monitoring v1.0.0

setup-lokistack-otel-k8s-events

Create a LokiStack instance to prepare a storage to OpenTelemetry Collector for Kubernetes Events.

clusters/management-cluster/setup-lokistack-otel-k8s-events

opentelemetry

Source

Observability & Monitoring v1.0.1

setup-multicluster-observability

Enabled MutliClusterObservability once ACM has been installed

clusters/management-cluster/setup-multicluster-observability

Dependencies / Requirements

Requires:

setup-acm

Source

Observability & Monitoring v1.0.1

setup-network-observability

Installs and configures OpenShift Network Observability.

clusters/management-cluster/setup-network-observability

Source

Platform Configuration v1.0.2

clusterbranding

Deploys Cluster Branding such as custom Login Page or Logo etc.

clusters/management-cluster/setup-openshift-data-foundation

Dependencies / Requirements

Helm Charts:

openshift-data-foundation

Source

Observability & Monitoring v1.0.1

setup-openshift-logging

Installs and configures OpenShift Logging by deploying Logging and Loki Operator and configuring them accordingly. Example configuration is creating a Bucket using OpenShift Data Foundation.

clusters/management-cluster/setup-openshift-logging

Dependencies / Requirements

Helm Charts:

openshift-logging

Requires:

setup-loki-operator setup-openshift-data-foundation

logging

Source

Observability & Monitoring v1.0.0

setup-otel-operator

Installs and configures the OpenTelemetry Operator.

clusters/management-cluster/setup-otel-operator

Dependencies / Requirements

Helm Charts:

rh-build-of-opentelemetry

tracing

Source

Storage & Data v2.0.0

setup-quay

Setup Quay Registry

clusters/management-cluster/setup-quay

Dependencies / Requirements

Helm Charts:

quay-registry-setup

Requires:

setup-openshift-data-foundation

Source

Security & Compliance v1.0.2

setup-rh-build-of-keycloak

Deploy and configure the operator Red Hat Build of Keycloak

clusters/management-cluster/setup-rh-build-of-keycloak

Dependencies / Requirements

Helm Charts:

cert-manager rh-build-keycloak

keycloak

Source

Observability & Monitoring v1.0.0

setup-tempo-operator

Installs and configures the Tempo Operator (distributed tracing). This Operator will require a S3 storage, which can be installed and provided using ODF and available Helm Sub-Charts.

clusters/management-cluster/setup-tempo-operator

Dependencies / Requirements

Helm Charts:

tempo-tracing

tracing

Source

Platform Configuration v1.0.0

setup-trusted-profile-analyzer

Setup Trusted Profile Analyzer

clusters/management-cluster/trusted-artifact-signer

Source

Platform Configuration v1.0.1

setup-trusted-profile-analyzer

Setup Trusted Profile Analyzer

clusters/management-cluster/trusted-profile-analyzer

Dependencies / Requirements

Helm Charts:

redhat-trusted-profile-analyzer

Source

Platform Configuration v1.0.1

Cluster Version Update

Updates OpenShift ClusterVersion for controlled cluster upgrades via GitOps.

clusters/management-cluster/update-clusterversion

upgrade version clusterversion

Blog Source

Multi-Cluster Management v1.0.0

ACM Policy Management Waves

Manages ACM policy deployment waves using ArgoCD sync waves for ordered policy application.

clusters/management-cluster/wave-acm-policy-management

acm policy governance waves

Source

The Guide to OpenBao - Initialisation, Unsealing, and Auto-Unseal - Part 6

Thu, 05 Mar 2026 00:00:00 +0000

After deploying OpenBao via GitOps (Part 5), OpenBao must be initialised and then unsealed before it becomes functional. You usually do not want to do this unsealing manually, since this is not scalable especially in bigger, productive environments. This article explains how to handle initialisation and unsealing, and possible options to configure an auto-unseal process so that OpenBao unseals itself on every restart without manual key entry.

What is happening?

OpenBao starts in a sealed state. You must:

Initialise (once): run bao operator init to generate unseal keys (or recovery keys when using auto-unseal).
Unseal (after each restart): provide a threshold of unseal keys so each node can decrypt the root key. In the previous articles, we used a threshold of 3 (out of 5). This means that you need to run bao operator unseal three times with different keys on every node after every start before the service becomes available.

Unseal workflow (assuming a threshold of 3 and 5 nodes and initialisation already happened):

---
title: "Unseal Workflow"
config:
theme: 'dark'
---
flowchart LR
A["openbao-0 starts → 3× unseal using different keys"]
A --> B["openbao-1 starts → 3× unseal using different keys"]
B --> C["openbao-2 starts → 3× unseal using different keys"]
C --> D[OpenBao is ready]

This is a chicken-egg problem: you need to start somewhere, but you cannot start without the keys.

Manual unsealing does not scale very well and for production, auto-unseal is recommended.

This article covers:

Practical approaches to initialisation and unsealing (manual, Jobs, init containers).
Potential auto-unseal options: with AWS KMS, static key, and transit seal.

Handling Initialisation and Unsealing

The challenge with GitOps is that initialisation and unsealing are one-off or stateful operations. Below are several approaches. The problem is always the same: where do you start? You must have the unseal keys available to start the process. Do you keep them locally, do you keep them in a Kubernetes Secret, do you use an external Key Management System (KMS)? The most robust for production in my opinion is auto-unseal using a KMS (see [_approach_4_auto_unseal_with_kms_recommended_for_production] and the following sections). It uses an external Key Management System (KMS) to store the keys. This way, the keys are not stored in the cluster and are not exposed to the risk of being compromised.

Let’s have a look at the different options.

Approach 1: Manual Initialisation (simplest)

While manual initialisation and unsealing is the simplest approach, it is not recommended for production. Still, I would like to mention it here for completeness and at least provide a simple for loop to unseal the OpenBao service (on Kubernetes/OpenShift).

Manual unsealing is still a valid option when you have a small cluster and assume that it is not required often.

After Argo CD deploys OpenBao, the first pod, openbao-0, will not become ready until it is initialised and unsealed. Once done, the next pod, openbao-1, will try to start and wait until it is unsealed, and then the third pod, and so on.

The following, simplified script will go through the pods and unseal them one by one. It initialises the first pod and stores the unseal keys in the file openbao-init.json locally on your machine. From there it takes three different keys and unseals the OpenBao service. Since it takes a while until other pods are pulling the image and starting, we will use a sleep timer of 30 seconds between each pod. This may or may not be enough depending on your network and cluster speed. As said, this is a very simplified script:

The CLI tool oc can be replaced by kubectl in case you are using a different cluster. In addition, you will need the jq command line tool to parse the JSON file.

If your OpenBao was initialised previously, you can remove the first three commands and start with the unseal commands.

# Initialise the first pod, assuming the Pod openbao-0 is already running.
# You can skip this if OpenBao was initialised previously.
echo "Initialising OpenBao on openbao-0 (keys will be saved to openbao-init.json)..."
oc exec -it openbao-0 -n openbao -- bao operator init \
-key-shares=5 -key-threshold=3 -format=json > openbao-init.json (1)
echo "Init complete. Unseal keys and root token are in openbao-init.json."
# Unseal each pod (three times each with different keys)
echo "Unsealing pods openbao-0, openbao-1, openbao-2 (3 key steps each)..."
for i in 0 1 2; do
sleep_timer=30
SLEEPER_TMP=1
echo "Unsealing openbao-$i..."
for j in 1 2 3; do (2)
KEY=$(cat openbao-init.json | jq -r ".unseal_keys_b64[$((j-1))]")
oc exec -it openbao-$i -n openbao -- bao operator unseal $KEY
done
echo "openbao-$i unsealed. Waiting ${sleep_timer}s for next pod to be ready..."
while [[ $SLEEPER_TMP -le "$sleep_timer" ]]; do (3)
if (( SLEEPER_TMP % 10 == 0 )); then
echo -n "$SLEEPER_TMP"
else
echo -n "."
fi
sleep 1
SLEEPER_TMP=$((SLEEPER_TMP + 1))
done
echo ""
done
echo "All pods unsealed. Store init data securely (e.g. in a password manager)."
# Store init data securely (e.g. in a password manager)

1	Initialise the first pod, assuming the Pod openbao-0 is already running. This will store the keys (including the root token) in the file openbao-init.json.
2	Unseal each pod (three times each with different keys). The keys are taken from the openbao-init.json file.
3	Wait 30 seconds between each pod to give the other pods time to start and pull the image … super-fancy sleep timer.

The unseal keys as well as the root token are stored in the openbao-init.json file. This file is stored locally on your machine. Make sure to store it securely and do not share it with anyone.

Approach 2: Kubernetes Job for initialisation

A Job can run after OpenBao is deployed, initialise it if needed, and store the init output (unseal keys and root token) in a Kubernetes Secret. You can then unseal manually or with a second Job that reads from that Secret. This section assumes the init output is stored in a Kubernetes Secret and explains how to create that Secret from the Job and how to use it.

How the Secret is created

The init Job runs bao operator init and then creates the Secret using oc (or kubectl). The Secret name and key used here are openbao-init-data and init.json (the file contains the JSON output of bao operator init, including unseal_keys_b64 and root_token). One advantage of this approach is that you can integrate it into your GitOps pipeline. However, we can argue whether it is a good idea to store the unseal keys and the root token as a Secret in the same cluster where OpenBao is running.

Example structure of init.json:

{
"unseal_keys_b64": [
"key1",
"key2",
"key3",
"key4",
"key5"
],
"unseal_keys_hex": [
"hex1",
"hex2",
"hex3",
"hex4",
"hex5"
],
"unseal_shares": 5,
"unseal_threshold": 3,
"recovery_keys_b64": null,
"recovery_keys_hex": null,
"recovery_keys_shares": 0,
"recovery_keys_threshold": 0,
"root_token": "root.token"
}

You need at least unseal_threshold keys (e.g. 3) from unseal_keys_b64 to unseal each node. Store this file securely; anyone with access can unseal OpenBao and the root_token has full admin access.

RBAC Configuration

The Job’s service account (openbao-init) must be allowed to create (and optionally get/update) Secrets in the openbao namespace. Create a Role and RoleBinding:

apiVersion: v1
kind: ServiceAccount
metadata:
name: openbao-init (1)
namespace: openbao
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: openbao-init-secret-writer
namespace: openbao
rules: (2)
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create", "get", "update", "patch"]
- apiGroups: [""] (3)
resources: ["pods"]
verbs: ["get", "list"]
- apiGroups: [""] (4)
resources: ["pods/exec"]
verbs: ["create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding (5)
metadata:
name: openbao-init-secret-writer
namespace: openbao
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: openbao-init-secret-writer
subjects:
- kind: ServiceAccount
name: openbao-init
namespace: openbao

1	The ServiceAccount is used by the Job(s) to create the Secret and unseal the OpenBao pods.
2	The Role is used by the Job to create the Secret. The rules might be extended to include the permission to execute into the pods to unseal the OpenBao service (see below).
3	Permissions to get and list the Pods, required for the unseal process later
4	Permissions to execute into the Pods, required for the unseal process later
5	The RoleBinding is used by the Job to create the Secret.

The Job must also be able to reach the OpenBao API (e.g. openbao.openbao.svc:8200). This is usually possible when running in the same namespace as the OpenBao service.

The example below shows a working Job manifest that will perform the initialisation and the creation of the Secret in the openbao namespace. The manifest is already prepared for Argo CD with useful annotations. It runs an init container to initialise the OpenBao service and a main container to create the Secret. The init container writes (using the OpenBao CLI) the init.json file to a shared volume and the main container (using the kubectl command) creates the Secret from that file. In addition, the Secret containing the CA certificate is mounted as a volume and the BAO_CACERT environment variable is set to the path of the CA certificate.

Init Job that creates the Secret

apiVersion: batch/v1
kind: Job
metadata:
name: openbao-init
namespace: openbao
annotations: (1)
argocd.argoproj.io/sync-wave: "30"
argocd.argoproj.io/hook: PostSync
argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
template:
spec:
serviceAccountName: openbao-init (2)
initContainers: (3)
- name: openbao-init
image: ghcr.io/openbao/openbao:latest
command: (4)
- /bin/sh
- -c
- |
if [ -f /etc/openbao-ca/ca.crt ]; then
export BAO_CACERT=/etc/openbao-ca/ca.crt
export BAO_ADDR=https://openbao:8200
echo "Using TLS CA from /etc/openbao-ca/ca.crt"
else
export BAO_ADDR=http://openbao:8200
echo "Using plain HTTP"
fi
until bao status 2>&1 | grep -q "Initialized"; do
echo "Waiting for OpenBao..."
sleep 5
done
if bao status | grep -q "Initialized.*false"; then
echo "Initialising OpenBao..."
bao operator init -key-shares=5 -key-threshold=3 \
-format=json > /shared/init.json
echo "Init complete; init.json written to shared volume."
else
echo "OpenBao already initialised (skipping init)."
fi
volumeMounts:
- name: openbao-ca
mountPath: /etc/openbao-ca
readOnly: true
- name: shared
mountPath: /shared
containers: (5)
- name: create-secret
image: bitnami/kubectl:latest
command:
- /bin/sh
- -c
- |
if [ -f /shared/init.json ]; then
echo "Creating Secret openbao-init-data from init.json..."
kubectl create secret generic openbao-init-data \
--from-file=init.json=/shared/init.json \
-n openbao --dry-run=client -o yaml | kubectl apply -f -
echo "Initialisation complete!"
else
echo "No init.json (OpenBao already initialised or init skipped)."
fi
volumeMounts:
- name: shared
mountPath: /shared
readOnly: true
volumes:
- name: openbao-ca
secret:
secretName: openbao-ca-secret
optional: true
- name: shared (6)
emptyDir: {}
restartPolicy: Never
backoffLimit: 3

1	The annotations are used by Argo CD to trigger the Job after the OpenBao deployment.
2	The ServiceAccount is used by the Job to initialise the OpenBao service.
3	The init container runs the OpenBao CLI (wait for API, then operator init) and writes init.json to a shared volume.
4	The command is used to initialise the OpenBao service.
5	The main container runs kubectl to create the Secret from that file. This avoids needing a single image that bundles both `bao` and `kubectl`.
6	The shared volume is used to store the init.json file.

As a result the secret openbao-init-data is created in the openbao namespace. It contains the unseal keys and the root token.

Using the Secret in a Job (unseal)

A second Job can use the Secret openbao-init-data that was created by the init Job to unseal each OpenBao pod. A lot is happening here:

The init container extracts the unseal keys from the Secret using jq and writes them to a shared volume.
The main container is reading the unseal keys from the shared volume and unseals the first pod
Then the process is waiting for 30 seconds, to give the next pod time to pull the image and start
The process is repeated for the next pod and so on until all pods are unsealed.

First of all, the unseal Job needs permission to exec into the OpenBao pods. Add a Role that grants the Job’s service account create on pods/exec in the openbao namespace. To make it easier, we can extend the Role openbao-init-secret-writer that was created in the init Job.

These rules were already applied with the configuration for the init Job, but we will repeat them here for completeness.

rules: (1)
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"]

1	The role `openbao-init-secret-writer` must be extended to include the permission to execute into the pods to unseal the OpenBao service.

apiVersion: batch/v1
kind: Job
metadata:
name: openbao-unseal
namespace: openbao
annotations:
argocd.argoproj.io/sync-wave: "35"
argocd.argoproj.io/hook: PostSync
argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
template:
spec:
serviceAccountName: openbao-init (1)
initContainers:
- name: extract-keys
image: quay.io/codefreshplugins/curl-jq
command:
- /bin/sh
- -c
- |
if [ ! -f /secrets/init.json ]; then
echo "Secret not found; run init Job first."
exit 1
fi
for i in 0 1 2; do
jq -r ".unseal_keys_b64[$i]" /secrets/init.json | tr -d '\n' > "/shared/key$i" (2)
done
echo "Unseal keys extracted to shared volume."
volumeMounts:
- name: init-secret
mountPath: /secrets
readOnly: true
- name: shared
mountPath: /shared
containers: (3)
- name: unseal
image: bitnami/kubectl:latest
command: (4)
- /bin/sh
- -c
- |
if [ -f /etc/openbao-ca/ca.crt ]; then
export BAO_CACERT=/etc/openbao-ca/ca.crt
export BAO_ADDR=https://openbao:8200
echo "Using TLS CA from /etc/openbao-ca/ca.crt"
else
export BAO_ADDR=http://openbao:8200
echo "Using plain HTTP"
fi
sleep_timer=30
if [ ! -f /shared/key0 ]; then
echo "Keys not found; extract-keys init container may have failed."
exit 1
fi
echo "Unsealing pods openbao-0, openbao-1, openbao-2 (3 key steps each), ${sleep_timer}s delay between pods..."
for pod in openbao-0 openbao-1 openbao-2; do
echo "Unsealing $pod..."
for i in 0 1 2; do
key=$(cat "/shared/key$i" | tr -d '\n')
kubectl exec -n openbao "$pod" -- sh -c 'bao operator unseal "$1"' _ "$key" || true
done
echo "$pod unsealed."
if [ "$pod" != "openbao-2" ]; then
echo "Waiting ${sleep_timer}s for next pod to be ready..."
SLEEPER_TMP=1
while [ "$SLEEPER_TMP" -le "$sleep_timer" ]; do
if [ $(( SLEEPER_TMP % 10 )) -eq 0 ]; then
echo -n "$SLEEPER_TMP"
else
echo -n "."
fi
sleep 1
SLEEPER_TMP=$(( SLEEPER_TMP + 1 ))
done
echo ""
fi
done
echo "Unseal complete."
volumeMounts:
- name: shared
mountPath: /shared
readOnly: true
- name: openbao-ca
mountPath: /etc/openbao-ca
readOnly: true
volumes: (5)
- name: init-secret
secret:
secretName: openbao-init-data
- name: shared
emptyDir: {}
- name: openbao-ca
secret:
secretName: openbao-ca-secret
optional: true
restartPolicy: Never
backoffLimit: 2

1	The ServiceAccount is used by the Job to unseal the OpenBao pods. Here we are using the same ServiceAccount as the one used to initialise the OpenBao service.
2	The init container extracts unseal keys from init.json using jq and writes them to a shared volume.
3	The main container runs kubectl to exec into each OpenBao pod and run `bao operator unseal` (the OpenBao image is used inside the target pods).
4	The command is used to unseal the OpenBao service.
5	The volumes that are mounted: init-secret (contains the unseal keys), shared (contains the extracted unseal keys, shared between containers), openbao-ca (contains the CA certificate).

Both Jobs are using the same ServiceAccount and therefore the same RBAC rules. It is also possible to use a different ServiceAccount for the unseal Job.

Store init data securely. Keeping unseal keys in a cluster Secret is convenient but less secure than auto-unseal or an external secrets manager. Restrict access to the openbao-init-data Secret (e.g. RBAC and network policies) and consider rotating or moving keys to a more secure store after bootstrap.

Approach 3: Auto-unseal with Key Management Service KMS (recommended for production)

Configure a KMS (e.g. AWS KMS, Azure Key Vault, GCP Cloud KMS, Transit etc.) so that OpenBao unseals itself on every restart. You initialise once and thereafter no manual unseal step is needed.

The rest of this article will mention the different options. However, I can only demonstrate a real example using the AWS KMS option.

Auto-unseal options overview

Without auto-unseal, you must run bao operator unseal with a threshold of keys after each restart. With auto-unseal, OpenBao uses a seal backend to protect the root key and unseals itself on startup.

Supported options:

Option

Use case

Notes

1. AWS KMS

AWS (EKS, OpenShift on AWS)

IRSA or IAM user; no keys in cluster with IRSA.

2. Azure Key Vault

Azure (AKS, OpenShift on Azure)

Service principal or managed identity.

3. Google Cloud KMS

GCP (GKE)

Service account key or Workload Identity.

5. Transit

Existing Vault/OpenBao as root of trust; or a dedicated seal-only OpenBao that exists only to unseal production

External transit engine; token in a Secret.

Why use OpenBao when the seal is in KMS? Why not keep all secrets in KMS and skip OpenBao?

When using a cloud KMS (or Key Vault) for auto-unseal, a natural question is: why not store and retrieve all secrets from the KMS and run without OpenBao at all? OpenBao remains useful for operators and applications for several reasons:

KMS is for keys, not a full secrets platform. AWS KMS, Azure Key Vault keys, and GCP Cloud KMS are built to encrypt and decrypt small blobs (e.g. data encryption keys, or OpenBao’s seal blob). They are not a general-purpose secrets store with versioning, path-based access, dynamic credentials, and per-request audit. OpenBao provides that layer: applications request secrets by path, get short-lived tokens, and you get a single place to define who can read what.
Dynamic secrets. OpenBao can generate short-lived credentials on demand (e.g. database users, cloud IAM roles, PKI certificates). The KMS does not create or rotate such credentials; it only holds keys. If you need “give this pod a DB password that expires in 1 hour,” that is OpenBao’s domain, not the KMS.
Fine-grained, identity-aware access. OpenBao supports auth methods (Kubernetes, OIDC, AppRole, LDAP) and path-based policies. You can say “this service account may read only secret/data/myapp/*.” The KMS has IAM or Key Vault RBAC, but not the same model of “one API, many identities, many paths” that applications and operators use day to day.
Audit and compliance. OpenBao logs every secret read and write with identity and path. That gives a clear audit trail of who accessed which secret and when. KMS audit logs key usage, which is different from “which application read which secret at what time” in a unified way.
Abstraction and portability. Applications talk to OpenBao’s API. You can move the seal or the storage backend (or even the cloud) without changing how applications request secrets. If you put everything directly in a cloud KMS, you tie every app to that vendor’s API and limits.
Encryption as a service (Transit). OpenBao’s Transit secrets engine lets applications encrypt data with a key without ever seeing the key—useful for application-level encryption and key rotation. KMS can do something similar, but OpenBao integrates that with the same auth and audit as the rest of your secrets.

In short: the KMS is the root of trust for the seal (so OpenBao can unseal itself). OpenBao is the secrets management platform that applications and operators use for storing, retrieving, generating, and auditing secrets. Both layers complement each other.

General flow (same for all options):

Set up the seal backend (KMS key, Key Vault, static key, or transit server).
Add the appropriate seal "…" stanza to OpenBao configuration (e.g. in Helm server.ha.raft.config).
Deploy OpenBao; pods start and remain sealed until initialised.
Run one-time initialisation: bao operator init -recovery-shares=5 -recovery-threshold=3.
From then on, restarts are automatically unsealed.

Option: AWS KMS

Best for: EKS, OpenShift on AWS (ROSA), or any Kubernetes on AWS.

Create a KMS key

Create a symmetric KMS key used only for the OpenBao seal (do not use for application data).

AWS WebUI:
- search for: KMS → Create key → Symmetric, Encrypt/Decrypt
- Optionally set an alias (e.g. openbao-unseal).
- OPTIONAL: Define key administrative permissions
- OPTIONAL: Define key usage permissions
- Edit key policy

AWS CLI: (requires the AWS CLI to be installed and configured)

aws kms create-key \
--description "OpenBao auto-unseal key" \
--key-usage ENCRYPT_DECRYPT
aws kms create-alias \
--alias-name openbao-unseal \
--target-key-id <key-id-from-above>

Note the Key ID or Alias and the region (e.g. us-west-1) from the output of the command.

Key policy for OpenBao

The OpenBao process needs permission to use the key. Create an IAM policy. In the UI you can configure this directly when creating the key. When using the CLI, create a policy.json file with the following content:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": "arn:aws:kms:<REGION>:<ACCOUNT_ID>:key/<KEY_ID>" (1)
}
]
}

1	Replace `<REGION>`, `<ACCOUNT_ID>`, and `<KEY_ID>` with your values (or use the key ARN).

Create an IAM user for the seal

When using static credentials (e.g. on OpenShift or when IRSA is not available), create a dedicated IAM user that has permission only to use the KMS key above. Use that user’s access keys in the Kubernetes Secret.

AWS WebUI:
- search for: IAM → Users → Create user
- User name: e.g. openbao-seal
- DO NOT select "Provide user access to the AWS Management Console" (programmatic access only)
- Create user
- Open the created user → Permissions → Add permissions → Create inline policy (or Attach policies → Create policy)
- In the policy editor, choose JSON and paste the policy from Key policy for OpenBao, replacing <REGION>, <ACCOUNT_ID>, and <KEY_ID> with your KMS key ARN or alias
- Name the policy (e.g. OpenBaoSealKMS) and attach it to the user
- Security credentials → Create access key → Application running outside AWS (or "Third-party service") → Create access key
- Save the Access key ID and Secret access key; you will put them in the Kubernetes Secret openbao-aws-credentials.

AWS CLI: (requires the AWS CLI to be installed and configured)

Create a policy file (e.g. openbao-seal-policy.json) with the same JSON as in Key policy for OpenBao, then create the user, attach the policy, and create access keys:

# Replace REGION, ACCOUNT_ID, and KEY_ID in the policy (or use alias: arn:aws:kms:REGION:ACCOUNT_ID:alias/openbao-unseal)
aws iam create-policy \
--policy-name OpenBaoSealKMS \
--policy-document file://openbao-seal-policy.json
aws iam create-user --user-name openbao-seal
aws iam attach-user-policy \
--user-name openbao-seal \
--policy-arn arn:aws:iam::ACCOUNT_ID:policy/OpenBaoSealKMS
aws iam create-access-key --user-name openbao-seal

The last command returns AccessKeyId and SecretAccessKey; store them in the Secret openbao-aws-credentials with key AWS_REGION set to your region (e.g. us-west-1).

Create the Secret

Create the Secret openbao-aws-credentials in the openbao namespace. It contains the access key, secret access key, and region.

apiVersion: v1
kind: Secret
metadata:
name: openbao-aws-credentials
namespace: openbao
type: Opaque
stringData: (1)
AWS_ACCESS_KEY_ID: <your-access-key>
AWS_SECRET_ACCESS_KEY: <your-secret-key>
AWS_REGION: <your-region>

1	Replace `<your-access-key>`, `<your-secret-key>`, and `<your-region>` with the values from the previous steps.

Helm / Argo CD configuration (AWS)

Add the seal "awskms" block inside the same HCL config that contains listener, storage "raft". In Helm values this is typically under openbao.server.ha.raft.config.

Be sure to update the region and key ID.

The values file we used for Argo CD can be found at: OpenBao Argo CD values file

openbao:
server:
extraSecretEnvironmentVars: (1)
- envName: AWS_ACCESS_KEY_ID
secretName: openbao-aws-credentials
secretKey: AWS_ACCESS_KEY_ID
- envName: AWS_SECRET_ACCESS_KEY
secretName: openbao-aws-credentials
secretKey: AWS_SECRET_ACCESS_KEY
- envName: AWS_REGION
secretName: openbao-aws-credentials
secretKey: AWS_REGION
ha:
raft:
config: |
ui = true
seal "awskms" { (2)
region = "us-east-1"
kms_key_id = "alias/openbao-unseal"
}
listener "tcp" {
tls_disable = 0
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/openbao/tls/openbao-server-tls/tls.crt"
tls_key_file = "/openbao/tls/openbao-server-tls/tls.key"
tls_min_version = "tls12"
}
storage "raft" {
path = "/openbao/data"
}
service_registration "kubernetes" {}

1	If you use static credentials (e.g. for OpenShift), inject them via a Secret.
2	This must be added. Be sure to update your region and key ID.

The seal "awskms" block alone is not enough. OpenBao must have AWS credentials available at runtime. If you see an error such as NoCredentialProviders: no valid providers in chain or error fetching AWS KMS wrapping key information, the pod has no credentials. You must either inject static credentials from a Secret (see below) or use IRSA so the pod can assume an IAM role.

Approach 4: Init container with external secret store

Another option is to use an init container that retrieves secrets from an external source. Two use cases can be considered here:

Shamir unseal: The init container fetches the unseal keys from an external secret manager and then runs bao operator unseal (or writes keys to a file used by a sidecar/unseal script). This is the classic “external secret store” idea (e.g. using AWS Secrets Manager, HashiCorp Vault, another Kubernetes cluster).
Transit seal token: The init container fetches the token that production OpenBao uses to call the dedicated unseal service (see Dedicated unseal service (seal-only OpenBao)). That token is written to a file or shared volume so the main OpenBao process can use it in seal "transit". The token never lives in a Kubernetes Secret in Git or in plain YAML; it is fetched at pod start from the external store.

Below is a generic placeholder for the init container.

Option: init Container example

The following options are generic and more theoretical. Unlike the other options, I could not test them in a real environment yet. This may be covered in a future article.

Generic placeholder (implement according to your secret store):

# Additional values for the Helm chart
server:
extraInitContainers:
- name: fetch-secrets
image: registry.access.redhat.com/ubi9/ubi-minimal:latest
command:
- /bin/sh
- -c
- |
# Fetch unseal keys OR transit token from external secret manager
echo "Waiting for OpenBao to be ready..."
sleep 30
# Your fetch logic here (e.g. aws secretsmanager get-secret-value, vault kv get, curl to API) (1)
env:
- name: EXTERNAL_SECRET_ENDPOINT (2)
value: "https://your-external-secret-store.example.com"

1	Replace with your fetch logic.
2	Replace with your external secret store endpoint.

Option: Transit seal

When you already have a Vault or OpenBao cluster and want it to act as the root of trust (e.g. central Vault encrypts the seal key), you can use this option.

To prepare everything, you need to do the following:

Enable the Transit secrets engine on the external Vault/OpenBao and create a key (e.g. openbao-unseal).
Grant the token used by this OpenBao permission to encrypt/decrypt with that key.
Store the token in a Kubernetes Secret and inject via extraSecretEnvironmentVars

Do not put the token in the config file.

The following HCL configuration is used to configure the OpenBao cluster to use the transit seal.

seal "transit" {
address = "https://external-vault.example.com:8200"
token = "s.xxx"
disable_renewal = "false"
key_name = "openbao-unseal"
mount_path = "transit/"
}

See the OpenBao transit seal documentation for further details.

Dedicated unseal service (seal-only OpenBao)

A common and recommended pattern is to run a separate, standalone OpenBao instance whose only role is to provide the seal for your production OpenBao. In other words: the external OpenBao holds the unseal keys—via its Transit secrets engine—and the production OpenBao cluster uses the transit seal to auto-unseal by calling that external instance. No application secrets or other workloads run on the dedicated instance; it exists solely to unseal the production OpenBao.

Why use a dedicated unseal service?

Separation of concerns: The root of trust for unsealing lives outside the production cluster. If the production OpenBao is compromised or rebuilt, the seal keys remain in the dedicated instance.
Simpler operations: You unseal and manage only one small, locked-down OpenBao (the seal service). The production OpenBao unseals itself automatically via transit.
No cloud KMS required: On-premise or air-gapped environments can use this pattern instead of AWS KMS, Azure Key Vault, or GCP Cloud KMS.
Clear trust boundary: The dedicated instance can be hardened, network-isolated, and backed up independently.

Architecture (high level):

Dedicated unseal service: A standalone OpenBao (single node is often enough) with the Transit secrets engine enabled and a dedicated transit key (e.g. production-openbao-unseal). This instance is initialised and unsealed once; you keep its unseal keys and root token very secure. It does not store application secrets.
Production OpenBao: Your HA OpenBao cluster (e.g. in Kubernetes) is configured with seal "transit" pointing at the dedicated service URL and a token that has permission only to use that transit key. On startup, each production node asks the dedicated OpenBao to decrypt its seal blob and thus unseals automatically.

Migration from Shamir to auto-unseal

You might wonder if you can migrate from Shamir to auto-unseal. The answer is yes, you can.

If OpenBao was previously initialised with Shamir unseal keys and you want to switch to any auto-unseal backend, you can do the following:

Plan a short maintenance window. Raft HA will tolerate one node at a time.
Add the appropriate seal "…" block to the config and deploy (e.g. via Argo CD).

Do NOT re-run init.
Unseal the leader with the existing Shamir unseal keys.
Run seal migration on the leader:
```
bao operator unseal -migrate
```
Restart the leader. It should auto-unseal via the new seal.
Update and restart standby nodes. They should auto-unseal as well.

Resources

The Guide to OpenBao - GitOps Deployment with Argo CD - Part 5

Fri, 20 Feb 2026 00:00:00 +0000

Following the GitOps mantra "If it is not in Git, it does not exist", this article demonstrates how to deploy and manage OpenBao using Argo CD. This approach provides version control, audit trails, and declarative management for your secret management infrastructure.

Introduction

Deploying OpenBao via GitOps offers significant advantages:

Version Control: All configuration changes are tracked in Git
Audit Trail: Who changed what, when, and why
Declarative: Desired state is defined, not imperative commands
Reproducible: Same deployment process across environments
Self-healing: Argo CD ensures actual state matches desired state

However, there are challenges specific to secret management:

Initial unsealing requires manual intervention or automationxw
Root tokens and unseal keys must be handled carefully
Chicken-and-egg problem: How to store OpenBao secrets before OpenBao exists?

This article will focus on the deployment of the applicaiotns for the OpenBao deployment. The problem with the automatic unsealing will be addressed in the next article.

Prerequisites

Before you begin, ensure you have:

OpenShift GitOps (Argo CD) installed and configured
A Git repository for your cluster configuration
Understanding of the App-of-Apps pattern (see Configure App-of-Apps)
The official OpenBao Helm chart from Part 3

The (Wrapper) Helm Chart

The official OpenBao Helm Chart works well for deploying OpenBao itself. However, additional settings such as certificates are not covered there. Therefore, I created a (wrapper) Helm Chart that includes the official OpenBao Helm Chart, a Cert-Manager Helm Chart that I created a while ago, and allows you to add additional objects in the templates folder.

I will follow the same setup as discussed in Part 4. This means:

Create Issuer for Cert-Manager
Create CA Certificate for OpenBao
Create Certificate for OpenBao Server
Create Certificate for OpenBao Agent Injector
Install OpenBao Helm Chart

However, there is one significant issue with this approach: we need the certificate details (especially the CA certificate) before we can install the OpenBao Helm Chart, since the certificate must be referenced in the values file. This is a chicken-and-egg problem—particularly if you use self-signed CA certificates.

Since you typically create the CA certificate first and only once (or have it already), a separate Application for the CA certificate is a good approach. Once this is synchronised, the OpenBao Application can be updated and deployed.

This means we will need two Helm Charts: one for the CA certificate and one for OpenBao itself.

Helm Chart for the CA Certificate

Create a new Helm Chart for the CA certificate. This will use the Cert-Manager Helm Chart as a dependency and will create the:

Issuer for Cert-Manager
Request the CA certificate from the Issuer
Create the openbao namespace

The example I am using can be found at GitOps openbao-ca-certificate.

Please check the GitOps Repository Structure and subsequent articles for more information on how to structure your Git repository.

In the Chart.yaml file you can see two dependencies:

dependencies:
- name: tpl
version: ~1.0.0
repository: https://charts.stderr.at/
- name: cert-manager
version: ~2.0.3 (1)
repository: https://charts.stderr.at/
condition: cert-manager.enabled

1	The Cert-Manager Helm Chart version must be at least 2.0.3 to support the namespace parameter for the Issuer.

The tpl dependency is a library that contains templates for the namespace and other shared components (I keep this library in my Helm Charts repository). The cert-manager dependency is the Cert-Manager Helm Chart.

The values.yaml file below contains the configuration for the Cert-Manager:

All settings that are passed to a subchart (cert-manager) must be prefixed with the name of the subchart. In this case, cert-manager..

namespace:
create: true (1)
name: "openbao"
description: "OpenBao Namespace"
displayName: "OpenBao Namespace"
additionalLabels:
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/audit-version: latest
pod-security.kubernetes.io/warn: restricted
pod-security.kubernetes.io/warn-version: latest
cert-manager: (2)
enabled: true (3)
issuer:
# Name of issuer
- name: openbao-selfsigned (4)
# -- Syncwave to create this issuer
syncwave: 5 (5)
# -- Type can be either ClusterIssuer or Issuer
type: Issuer
# -- Enable this issuer.
# @default -- false
enabled: true
# -- Namespace for Issuer (ignored for ClusterIssuer). Defaults to the default namespace when not set.
namespace: openbao (6)
# -- Create a selfSigned issuer. The SelfSigned issuer doesn't represent a certificate authority as such, but instead denotes that certificates will "sign themselves" using a given private key.
selfSigned: true (7)
certificates: (8)
enabled: true
# List of certificates
certificate:
- name: openbao-ca
enabled: true
namespace: openbao
syncwave: "10"
secretName: openbao-ca-secret
duration: 87660h
dnsNames:
- openbao-ca
privateKey:
algorithm: ECDSA
size: 256
rotationPolicy: Always
isCA: true
# Reference to the issuer that shall be used.
issuerRef:
name: openbao-selfsigned
kind: Issuer

1	Create the openbao namespace with various settings.
2	Enable the Cert-Manager subchart. All settings below will be passed to the Cert-Manager subchart.
3	Enables Cert-Manager.
4	Name of the issuer.
5	Syncwave for this issuer; it must be lower than the syncwave of the certificate.
6	Namespace for the issuer; supported since version 2.0.3 of the Cert-Manager Helm Chart.
7	Creates a self-signed issuer.
8	The certificate section and all the settings. Verify the Part 4 for more information.

This chart creates the openbao namespace, which is required for the OpenBao deployment.

Helm Chart for the OpenBao Deployment

Create a new Helm Chart for the OpenBao deployment. This will use the official OpenBao Helm chart and the Cert-Manager Helm Chart as dependencies and will create the:

OpenBao deployment (including Route)
Issuer for the OpenBao server using the CA certificate
Required OpenBao certificates based on the CA certificate

I have added the full values.yaml file below. Please check the GitOps openbao/values.yaml to fetch the full chart.

Besides the two certificates added here in the values.yaml file, I made two further changes: fullnameOverride and nameOverride are both set to openbao. This is good practice for setting the name of the deployment when using Argo CD.

This values file contains the CA certificate in plain text. Whether that is acceptable is debatable—it is a public certificate, and here it is self-signed and used only in my demo environment.

# Full values.yaml file for the OpenBao deployment
########################################################
# Cert-Manager
########################################################
cert-manager:
enabled: true
issuer:
# Name of issuer
- name: openbao-ca-issuer (1)
# -- Syncwave to create this issuer
syncwave: '-1'
# -- Type can be either ClusterIssuer or Issuer
type: Issuer
# -- Enable this issuer.
# @default -- false
enabled: true
# -- Namespace for Issuer (ignored for ClusterIssuer). Defaults to the default namespace when not set.
namespace: openbao
ca:
secretName: openbao-ca-secret
certificates:
enabled: true
# List of certificates
certificate:
########################################################
# OpenBao Server TLS Certificate
########################################################
- name: openbao-server-tls (2)
enabled: true
namespace: openbao
syncwave: "0"
secretName: openbao-server-tls
duration: 8760h
renewBefore: 720h
dnsNames:
- openbao.openbao.svc
- openbao.apps.ocp.aws.ispworld.at # Route host (adjust to your domain)
- openbao
- openbao.openbao
- openbao.openbao.svc
- openbao.openbao.svc.cluster.local
- openbao-internal
- openbao-internal.openbao
- openbao-internal.openbao.svc
- openbao-internal.openbao.svc.cluster.local
- openbao-0.openbao-internal
- openbao-0.openbao-internal.openbao
- openbao-0.openbao-internal.openbao.svc
- openbao-0.openbao-internal.openbao.svc.cluster.local
- openbao-1.openbao-internal
- openbao-1.openbao-internal.openbao
- openbao-1.openbao-internal.openbao.svc
- openbao-2.openbao-internal
- openbao-2.openbao-internal.openbao
- openbao-2.openbao-internal.openbao.svc
ipAddresses:
- 127.0.0.1
- "::1"
# Reference to the issuer that shall be used.
issuerRef:
name: openbao-ca-issuer
kind: Issuer
########################################################
# Injector TLS Certificate
########################################################
- name: injector-certificate (3)
enabled: true
namespace: openbao
syncwave: "0"
secretName: injector-tls
duration: 24h
renewBefore: 144m
dnsNames:
- openbao-agent-injector-svc
- openbao-agent-injector-svc.openbao
- openbao-agent-injector-svc.openbao.svc
- openbao-agent-injector-svc.openbao.svc.cluster.local
# Reference to the issuer that shall be used.
issuerRef:
name: openbao-ca-issuer
kind: Issuer
########################################################
# OpenBao Deployment
########################################################
openbao: (4)
# Override the full name of the deployment via Argo CD
fullnameOverride: openbao
nameOverride: openbao
global:
# Enable OpenShift-specific settings
openshift: true
# -- The namespace to deploy to. Defaults to the `helm` installation namespace.
namespace: openbao
# Required when TLS is enabled: tells the chart to use HTTPS for readiness/liveness
# probes and for in-pod API_ADDR (127.0.0.1:8200). Otherwise you get "client sent
# an HTTP request to an HTTPS server" from the probes.
tlsDisable: false
server:
extraEnvironmentVars:
BAO_CACERT: /openbao/tls/openbao-server-tls/ca.crt
# High Availability configuration
ha:
enabled: true
replicas: 3
# Raft storage configuration
raft:
enabled: true
setNodeId: true
config: |
ui = true
listener "tcp" {
tls_disable = 0
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/openbao/tls/openbao-server-tls/tls.crt"
tls_key_file = "/openbao/tls/openbao-server-tls/tls.key"
tls_min_version = "tls12"
telemetry {
unauthenticated_metrics_access = "true"
}
}
storage "raft" {
path = "/openbao/data"
retry_join {
leader_api_addr = "https://openbao-0.openbao-internal:8200"
leader_tls_servername = "openbao-0.openbao-internal"
leader_ca_cert_file = "/openbao/tls/openbao-server-tls/ca.crt"
}
retry_join {
leader_api_addr = "https://openbao-1.openbao-internal:8200"
leader_tls_servername = "openbao-1.openbao-internal"
leader_ca_cert_file = "/openbao/tls/openbao-server-tls/ca.crt"
}
retry_join {
leader_api_addr = "https://openbao-2.openbao-internal:8200"
leader_tls_servername = "openbao-2.openbao-internal"
leader_ca_cert_file = "/openbao/tls/openbao-server-tls/ca.crt"
}
}
service_registration "kubernetes" {}
telemetry {
prometheus_retention_time = "30s"
disable_hostname = true
}
route:
enabled: true
host: openbao.apps.ocp.aws.ispworld.at
tls:
# Route terminates client TLS; backend can use reencrypt or passthrough
termination: reencrypt
insecureEdgeTerminationPolicy: Redirect
destinationCACertificate: |
-----BEGIN CERTIFICATE-----
MIIBWzCCAQCgAwIBAgIQNdbg4KIu9oi6dDClE8drmjAKBggqhkjOPQQDAjAAMB4X
DTI2MDIxODA5NDIxNFoXDTM2MDIxODIxNDIxNFowADBZMBMGByqGSM49AgEGCCqG
SM49AwEHA0IABIeYw35/kEHvyctLtOA5xMlyQNxUtXtfBbZMUfPh6AN5MFjIGuNS
cn07a3EpSpfY6/3DaPpu+4wYNFlc+/qDNYajXDBaMA4GA1UdDwEB/wQEAwICpDAP
BgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQgyRk5Vv9K0VULcCgga5mRg4O9kTAY
BgNVHREBAf8EDjAMggpvcGVuYmFvLWNhMAoGCCqGSM49BAMCA0kAMEYCIQCwQ7lZ
Q0jzUjJFzpTGkQjU2+OB159LIQMSbSQ7dz8nVQIhAIDa7f87tjQxDxbJio+/vJx2
awFaWnueGOOQpvwCcV/+
-----END CERTIFICATE-----
extraVolumes:
- type: secret
name: openbao-server-tls
path: /openbao/tls
readOnly: true
extraVolumeMounts:
- name: openbao-server-tls
mountPath: /openbao/tls
readOnly: true
# Resource requests and limits
resources:
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 1Gi
cpu: 1000m
# Persistent volume for data
dataStorage:
enabled: true
size: 10Gi
# storageClass: "gp3-csi"
# Injector configuration
injector:
enabled: true
replicas: 2 # HA for the injector too
certs:
secretName: injector-tls
# For a private CA: set caBundle to the CA cert (PEM) so the Kubernetes API server trusts the injector webhook. E.g. oc get secret openbao-ca-secret -n openbao -o jsonpath='{.data.ca\.crt}' | base64 -d
caBundle: "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJXekNDQVFDZ0F3SUJBZ0lRTmRiZzRLSXU5b2k2ZERDbEU4ZHJtakFLQmdncWhrak9QUVFEQWpBQU1CNFgKRFRJMk1ESXhPREE1TkRJeE5Gb1hEVE0yTURJeE9ESXhOREl4TkZvd0FEQlpNQk1HQnlxR1NNNDlBZ0VHQ0NxRwpTTTQ5QXdFSEEwSUFCSWVZdzM1L2tFSHZ5Y3RMdE9BNXhNbHlRTnhVdFh0ZkJiWk1VZlBoNkFONU1GaklHdU5TCmNuMDdhM0VwU3BmWTYvM0RhUHB1KzR3WU5GbGMrL3FETllhalhEQmFNQTRHQTFVZER3RUIvd1FFQXdJQ3BEQVAKQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlFneVJrNVZ2OUswVlVMY0NnZ2E1bVJnNE85a1RBWQpCZ05WSFJFQkFmOEVEakFNZ2dwdmNHVnVZbUZ2TFdOaE1Bb0dDQ3FHU000OUJBTUNBMGtBTUVZQ0lRQ3dRN2xaClEwanpVakpGenBUR2tRalUyK09CMTU5TElRTVNiU1E3ZHo4blZRSWhBSURhN2Y4N3RqUXhEeGJKaW8rL3ZKeDIKYXdGYVdudWVHT09RcHZ3Q2NWLysKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo="
certName: tls.crt
keyName: tls.key
# UI configuration
ui:
enabled: true

1	Issuer for OpenBao. The syncwave is set to '-1' to create the issuer before the certificate request and the OpenBao deployment.
2	Certificate for the OpenBao server, with all DNS names and IP addresses used to reach it. The syncwave is set to '0' so the certificate is created after the issuer.
3	Certificate for the OpenBao Agent Injector, with all DNS names used to reach it. The syncwave is set to '0' so the certificate is created after the issuer.
4	OpenBao deployment configuration; the two overrides fullnameOverride and nameOverride are both set to openbao. See Part 4 for more information.

Creating Argo CD Applications

Whilst the Helm Charts are ready, we need to create the Argo CD Applications for the CA certificate and the OpenBao deployment. If you read our blog carefully, you will notice that these Application resources are created automatically when I add something to the folder clusters/management-cluster :) This is done by leveraging ApplicationSet resources and is fully described in the GitOps Repository Structure and subsequent articles.

For the sake of simplicity, I will show the created Application resources below. The setup is the same for both; they simply target a different path in the Git repository.

# Full Application resource for the OpenBao CA Certificate
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: in-cluster-openbao-ca-certificate
namespace: openshift-gitops
spec:
destination:
name: in-cluster (1)
namespace: default (2)
info:
- name: Description
value: ApplicationSet that Deploys on Management Cluster Configuration (using Git Generator)
project: in-cluster (3)
source:
path: clusters/management-cluster/openbao-ca-certificate (4)
repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops' (5)
targetRevision: main
syncPolicy:
retry:
backoff:
duration: 5s
factor: 2
maxDuration: 3m
limit: 5
---
# Full Application resource for the OpenBao deployment
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: in-cluster-openbao
namespace: openshift-gitops
spec:
destination:
name: in-cluster (1)
namespace: openbao (2)
info:
- name: Description
value: ApplicationSet that Deploys on Management Cluster Configuration (using Git Generator)
project: in-cluster (3)
source:
path: clusters/management-cluster/openbao (6)
repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops' (5)
targetRevision: main
syncPolicy:
retry:
backoff:
duration: 5s
factor: 2
maxDuration: 3m
limit: 5

1	Target cluster; here, the local cluster
2	Namespace of the target cluster; here, the OpenBao deployment will be installed.
3	Argo CD Project (must exist)
4	Path to the Git repository for the CA certificate
5	URL of the Git repository
6	Path to the OpenBao deployment

This will create the following Applications: - in-cluster-openbao-ca-certificate - in-cluster-openbao

The ApplicationSet adds the prefix in-cluster- to each Application name so that they remain unique in Argo CD.

Figure 1. Argo CD: OpenBao Argo CD Applications

The first Application to synchronise is in-cluster-openbao-ca-certificate.

It creates the following:

Namespace
CA Issuer
CA Certificate

Then synchronise in-cluster-openbao. It creates:

OpenBao deployment
OpenBao Agent Injector
OpenBao Server TLS Certificate
OpenBao Agent Injector TLS Certificate

OpenBao is running—what next?

Deploying OpenBao via GitOps gives you version control and declarative management for your secret management infrastructure.

Whilst OpenBao is running and managed by Argo CD, the next step is to configure it: you will need to handle initialisation (for a new cluster) and unsealing (see Part 3 and Part 4). That manual approach does not scale. In the next article, I will discuss ways to automate initialisation and unsealing.

Key takeaways:

Use sync waves to control deployment order
Consider auto-unseal for production (Part 6)
Store initialisation data securely outside Git

Resources

Helm Charts Repository Updates

Fri, 12 Dec 2025 00:00:00 +0000

This page shows the latest updates to the stderr.at Helm Charts Repository. The charts are designed for OpenShift and Kubernetes deployments, with a focus on GitOps workflows using Argo CD.

The content below is dynamically loaded from the Helm repository and always shows the most recent changes.

Quick Links

Resource	Link
Helm Repository	https://charts.stderr.at/
GitHub Source	https://github.com/tjungbauer/helm-charts
ArtifactHub	https://artifacthub.io/packages/search?repo=tjungbauer
Example GitOps Repo	https://github.com/tjungbauer/openshift-clusterconfig-gitops

Resource

Link

Helm Repository

https://charts.stderr.at/

GitHub Source

https://github.com/tjungbauer/helm-charts

ArtifactHub

https://artifacthub.io/packages/search?repo=tjungbauer

Example GitOps Repo

https://github.com/tjungbauer/openshift-clusterconfig-gitops

[Ep.15] OpenShift GitOps - Argo CD Agent

Wed, 14 Jan 2026 00:00:00 +0000

OpenShift GitOps based on Argo CD is a powerful tool to manage the infrastructure and applications on an OpenShift cluster. Initially, there were two ways of deployment: centralized and decentralized (or distributed). Both methods had their own advantages and disadvantages. The choice was mainly between scalability and centralization. With OpenShift GitOps v1.19, the Argo CD Agent was finally generally available. This agent tries to solve this problem by bringing the best of both worlds together. In this quite long article, I will show you how to install and configure the Argo CD Agent with OpenShift GitOps using hub and spoke architecture.

Classic Deployment Models:

Prior to the Argo CD Agent, there were two classic and often used deployment models: centralized and decentralized.

Centralized Model

In a centralized deployment, all changes are applied to a single, central Argo CD instance, often installed on a management cluster. This is the traditional way of deploying Argo CD, at least in my opinion. With this model, you have a single pane of glass to manage all your clusters. You have a single UI and will see all your clusters in one place, which makes this model very convenient when you have multiple clusters. However, the scalability of this model is limited. Organizations with a huge number of clusters or Argo CD applications would hit some boundaries at some point. A sharding configuration would help, but only to a certain extent. The performance would degrade significantly. In addition, this model creates a Single Point of Failure. If this instance is down, the company loses the ability to manage their clusters through Argo CD.

I often saw or used this model for the cluster configuration.

Decentralized Model

In a decentralized deployment, multiple instances of Argo CD, often one for each cluster, are installed. With this approach, the issue with scalability is solved. Moreover, the Single Point of Failure is eliminated as well, since a broken instance will not affect the other instances. However, the disadvantages of this model are that the complexity for the operational teams will increase, since they need to manage multiple instances now. Also, the single pane of glass for management is lost, as there are multiple UIs to manage.

I often saw or used this model for the application deployment.

The - not so secret - Argo CD Agent

The Argo CD Agent, released as generally available in OpenShift GitOps v1.19, is a new way to use Argo CD. It tries to solve the challenges of the classic deployment models by combining the best of both worlds. The Agent allows you to have a single UI in a central control plane, while the application controller is distributed across the fleet of clusters. Agents on the different clusters will communicate with the central Argo CD instance.

The Agent model introduces a hub and spoke architecture:

Control plane cluster (hub) - The control plane cluster is the central cluster that manages the configuration for multiple spokes.
Workload cluster (spoke) - The workload cluster is the cluster that runs the application workloads deployed by Argo CD.

Each Argo CD Agent on a cluster manages the local Argo CD instance and ensures that applications, AppProjects, and secrets remain synchronized with their source of truth.

The official documentation describes a comparison between the classic deployment models and the Argo CD Agent: GitOps Architecture - Argo CD Agent Comparison

Argo CD Agent Modes

The Argo CD Agent supports two modes of operation: Managed and Autonomous. The mode determines where the authoritative source of truth for the Application .spec field resides.

Managed mode — the control plane/hub defines Argo CD applications and their specifications.
Autonomous mode — each workload cluster/spoke defines its own Argo CD applications and their specifications.

A mixed mode is also possible.

Managed Mode

Using the managed mode means that the control plane is the source of truth and is responsible for the Argo CD application resources and their distribution across the different workload clusters. Any change on the hub cluster will be propagated to the spoke clusters. Any changes made on the spoke/workload cluster will be reverted to match the control plane configuration.

Autonomous Mode

Using this mode, the Argo CD applications are defined on the workload clusters, which serve as their own source of truth. The applications are synchronized back to the control plane for observability. Changes made on the workload cluster are not reverted, but will appear on the control plane. On the other hand, you cannot modify applications directly from the control plane.

Security

The Argo CD Agent uses mTLS certificates to communicate between the hub and the spoke clusters.

The certificate must be created and managed by the user.

Argo CD Agent Installation

The agent consists of two components that are responsible to synchronize the Argo CD applications between the hub and the spoke clusters.

Principal - Deployed on the control plane cluster together with Argo CD. Here the central UI (and API) can be found.
Agent - Deployed on the workload clusters to synchronize the Argo CD applications.

The installation of both is done differently. But before we dive into the installation, let’s have a look at the terminology to understand the different components and their roles.

This is a quote from the official documentation (Argo CD Agent Terminologies)

Principal namespace - Specifies the namespace where you install the Principal component. This namespace is not created by default, you must create it before adding the resources in this namespace. In Argo CD Agent CLI commands, this value is provided using the --principal-namespace flag.
Agent namespace - Specifies the namespace hosting the Agent component. This namespace is not created by default, you must create it before adding the resources in this namespace. In Argo CD Agent CLI commands, this value is provided using the --agent-namespace flag.
Context - A context refers to a named configuration in the oc CLI that allows you to switch between different clusters. You must be logged in to all clusters and assign distinct context names for the hub and spoke clusters. Examples for cluster names include principal-cluster, hub-cluster, managed-agent-cluster, or autonomous-agent-cluster.
Principal context - The context name you provide for the hub (control plane) cluster. For example, if you log in to the hub cluster and rename its context to principal-cluster, you specify it in Argo CD Agent CLI commands as --principal-context principal-cluster.
Agent context - The context name you provide for the spoke (workload) cluster. For example, if you log in to a spoke cluster and rename its context to autonomous-agent-cluster, you specify it in Argo CD Agent CLI commands as --agent-context autonomous-agent-cluster.

A Word about the Setup

To create some kind of real customer scenario, I have created two clusters:

The cluster where we installed the Principal component. This will be the Hub/Management/Principal cluster (We have too many words for this…)
A separate cluster that will be the Agent cluster. This will be the Spoke/Workload cluster.

The first one I have installed on AWS, and the second one is a Bare Metal Single node cluster. Both can reach each other via the Internet.

We are using different contexts for the different clusters for the command line. The argocd-agentctl tool knows the flags --principal-context and --agent-context to switch between the different clusters. Be sure to create the resources on the correct cluster.

Prerequisites

Before we start with the installation of the Principal or Agent component, we need to ensure that the following prerequisites are met:

We have two OpenShift test clusters.
Both clusters can reach each other
OpenShift GitOps Operator is already installed (possible configuration modifications are described in the following sections)

Because of my main focus on OpenShift GitOps, we will try to deploy cluster configurations and not just workload. Therefore, the example Argo CD applications will configure cluster settings. (A banner on the top and bottom of the UI)

Configure OpenShift GitOps Subscription on the Hub Cluster

The OpenShift GitOps Operator installs by default an Argo CD instance. In this test we will disable this, as we do not need that instance. Moreover and even more important, we need to tell the Operator for which namespaces it should feel responsible for. In this case, we will tell the Operator to be responsible for all namespaces on the cluster.

We need to modify the Subscription openshift-gitops-operator and add the following environment variables:

spec:
config:
env:
- name: DISABLE_DEFAULT_ARGOCD_INSTANCE (1)
value: 'true'
- name: ARGOCD_CLUSTER_CONFIG_NAMESPACES (2)
value: '*'

1	Optional: Disable the default Argo CD instance.
2	Tell the Operator for which namespaces it should feel responsible for. In this case, all Namespaces. This is important for a namespace-scoped Argo CD instance, which we will install in the next step.

Activate the Principal Component

To activate the Principal components we first need a cluster (the Hub) where OpenShift GitOps is installed. On this cluster, there might be a running instance of Argo CD already. However, at this time an existing Argo CD cannot be used. Instead, a new Argo CD instance must be created. (This is because the controller must not be activated)

In my case, I will install the Principal component in the namespace argocd-principal.

To create an Argo CD instance we need to create the following configuration:

At this very stage, the principal component must be installed in a separate Argo CD instance, since the controller must not be activated. Therefore we create a new Argo CD instance in a new namespace.

apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
name: hub-argocd
namespace: argocd-principal
spec:
controller:
enabled: false (1)
argoCDAgent:
principal:
enabled: true (2)
auth: "mtls:CN=([^,]+)" (3)
logLevel: "info"
namespace:
allowedNamespaces: (4)
- "*"
tls:
insecureGenerate: false (5)
jwt:
insecureGenerate: false
sourceNamespaces: (6)
- "argocd-agent-bm01"
server:
route:
enabled: true (7)

1	Disable the controller component.
2	Enable the Principal component.
3	Authentication method for the Principal component.
4	Allowed namespaces for the Principal component.
5	Insecure generation of the TLS certificate.
6	Specifies the sourceNamespaces configuration. (Such a list might already exist)
7	Enable the Route for the Principal component.

This will start a Pod called hub-gitops-agent-principal in the namespace argocd-principal. However, this pod will fail at this moment and that is fine.

Pod hub-gitops-agent-principal is failing with:
{"level":"info","msg":"Setting loglevel to info","time":"2026-01-19T13:45:57Z"}
time="2026-01-19T13:45:57Z" level=info msg="Loading gRPC TLS certificate from secret argocd-principal/argocd-agent-principal-tls"
time="2026-01-19T13:45:57Z" level=info msg="Loading root CA certificate from secret argocd-principal/argocd-agent-ca"
time="2026-01-19T13:45:57Z" level=info msg="Loading resource proxy TLS certificate from secrets argocd-principal/argocd-agent-resource-proxy-tls and argocd-principal/argocd-agent-ca"
[FATAL]: Error reading TLS config for resource proxy: error getting proxy certificate: could not read TLS secret argocd-principal/argocd-agent-resource-proxy-tls: secrets "argocd-agent-resource-proxy-tls" not found (1)

1	The secret is not yet available.

The Pod is failing at the moment because the different secrets for authentication, are not yet available. The Secrets are created in a later step, because some settings, such as the principal hostname and resource proxy service names, are available only after the Red Hat OpenShift GitOps Operator enables the principal component.

At this point the Operator created the Route object already:

Route: hub-gitops-agent-principal
Hostname: https://hub-gitops-agent-principal-argocd-principal.apps.ocp.aws.ispworld.at
Service: hub-gitops-agent-principal

Configure the AppProject

If you configured the AppProject with sourceNamespaces, you need to add the following to the AppProject (for example to the default AppProject). This must match exactly the namespaces you have created for the Agent.

spec:
sourceNamespaces:
- "argocd-agent-bm01"

You can also use this patch command:

oc patch appproject default -n argocd-principal --type='merge' \
-p '{"spec": {"sourceNamespaces": ["argocd-agent-bm01"]}}' --context aws

Restart the Argo CD Pods to apply the changes.

Download argocd-agentctl

To create the required secrets, we need to download the argocd-agentctl tool.

This can be found at: https://developers.redhat.com/content-gateway/rest/browse/pub/cgw/openshift-gitops/

Download and install it for your platform.

Create Required Secrets

The following steps will create the required secrets for the Principal component. In this example, we will create our own CA and certificates. This is suitable for development and testing purposes. For production environments, you should use certificates issued by your organization’s PKI or a trusted certificate authority.

Use your company’s CA and certificates for production environments.

Initialize the Certificate Authority (CA)

To create a certificate authority (CA) that signs other certificates, we need to run the following command:

argocd-agentctl pki init \
--principal-namespace argocd-principal \ (1)
--principal-context aws

1	The namespace where the Principal component is running.

This will initialize the CA and store it in the secret argocd-principal/argocd-agent-ca. The certificate looks like:

"Certificate Information:
Common Name: argocd-agent-ca
Subject Alternative Names:
Organization: DO NOT USE IN PRODUCTION
Organization Unit:
Locality:
State:
Country:
Valid From: January 15, 2026
Valid To: January 15, 2036
Issuer: argocd-agent-ca, DO NOT USE IN PRODUCTION
Serial Number: 1 (0x1)"

Generate Service Certificate for the Principal

To generate the server certificate for the Principal’s gRPC service, run the following command:

argocd-agentctl pki issue principal \
--principal-namespace argocd-principal \
--principal-context aws \
--dns "<YOUR PRINCIPAL HOSTNAME>" (1)

1	The hostname of the Principal service. This must match with the hostname of the Principal’s route (spec.host) or, in case a LoadBalancer Service is used, with .status.loadBalancer.ingress.hostname.

Generate the Resource Proxy Certificate

The resource proxy service requires a certificate as well. Since the proxy will run on the same cluster as the Principal, we can use the service name directly. This is generated by the following command:

argocd-agentctl pki issue resource-proxy \
--principal-namespace argocd-principal \
--principal-context aws \
--dns hub-argocd-agent-principal-resource-proxy (1)

1	The service name for the resource-proxy. This must match with the service name of the Resource Proxy service.

Generate the JWT Signing Key

Generate the RSA private key for the JWT signing key by running the following command:

argocd-agentctl jwt create-key \
--principal-namespace argocd-principal \
--principal-context aws

This will generate the RSA private key and store it in the secret argocd-principal/argocd-agent-jwt.

Verify the Principal Component

Now the principal pod should be running successfully.

If the Pod still shows an error, wait a few moments or restart the Pod.

In the logs you should see the following:

{"level":"info","msg":"Setting loglevel to info","time":"2026-01-19T14:09:31Z"}
time="2026-01-19T14:09:31Z" level=info msg="Loading gRPC TLS certificate from secret argocd-principal/argocd-agent-principal-tls"
time="2026-01-19T14:09:31Z" level=info msg="Loading root CA certificate from secret argocd-principal/argocd-agent-ca"
time="2026-01-19T14:09:31Z" level=info msg="Loading resource proxy TLS certificate from secrets argocd-principal/argocd-agent-resource-proxy-tls and argocd-principal/argocd-agent-ca"
time="2026-01-19T14:09:31Z" level=info msg="Loading JWT signing key from secret argocd-principal/argocd-agent-jwt"
time="2026-01-19T14:09:31Z" level=info msg="Starting argocd-agent (server) v99.9.9-unreleased (ns=argocd-principal, allowed_namespaces=[*])" module=server

This concludes the configuration of the Principal component. There are a lot of steps to create the required secrets, but this is only done once. A GitOps-friendly way to achieve this might be done using a Kubernetes Job (if you consider this as GitOps-friendly… which I do).

Activate the Agent Component

After the Principal component is configured, you can activate one or more Agents (spoke or workload clusters) and connect them with the Hub.

The prerequisites are:

The Principal component is configured and running.
You have access to both the Principal and Agent clusters.
The argocd-agentctl CLI tool is installed and accessible from your environment.
The helm CLI is installed and configured. Ensure that the helm CLI version is later than v3.8.0.
OpenShift GitOps Operator is installed and configured on the Agent cluster.

Yes, a separate Helm Chart will be used to install the Agent component on the target cluster. This time it is not a Chart that I created, but one provided by Red Hat. :)

Create Agent Secret on Principal Cluster

We first need to create an agent on the Principal cluster.

argocd-agentctl agent create "argocd-agent-bm01" \ (1)
--principal-context "aws" \ (2)
--principal-namespace "argocd-principal" \ (3)
--resource-proxy-server "hub-argocd-agent-principal-resource-proxy:9090" (4)

1	A (unique) name for the Agent.
2	The context name for the Principal cluster. In my case it is "aws".
3	The namespace where the Principal component is running.
4	The resource proxy server URL. This is the URL of the Principal’s resource proxy service including the port (9090).

This will create the secret cluster-argocd-agent-bm01 with the label argocd.argoproj.io/secret-type: cluster in the Argo CD namespace.

Create the Agent Namespace on the Agent Cluster

Be sure that the target namespace on the agent or workload cluster exists. If not, create it first.

oc create namespace argocd-agent-bm01 --context bm (1)

1	The name of the namespace.

Propagate the Principal CA to the Agent Cluster

To copy the CA certificate from the principal to the agent cluster the following command is used:

argocd-agentctl pki propagate \
--agent-context bm \ (1)
--principal-context aws \ (2)
--principal-namespace argocd-principal \ (3)
--agent-namespace argocd-agent-bm01 (4)

1	The context name for the Agent cluster.
2	The context name for the Principal cluster.
3	The namespace where the Principal component is running.
4	The namespace where the Agent component is running.

This will copy the CA certificate from the Principal cluster to the Agent cluster into the namespace and secret argocd-agent-bm01/argocd-agent-ca.

Only the certificate is copied. The private key is not copied.

Generate a client certificate for the Agent

Now we need to create, based on the imported CA, a client certificate for the Agent. This is done by the following command:

argocd-agentctl pki issue agent "argocd-agent-bm01" \
--principal-context "aws" \
--agent-context "bm" \
--agent-namespace "argocd-agent-bm01" \
--principal-namespace "argocd-principal"

This will create the secret argocd-agent-client-tls on the workload cluster, containing a certificate and a key, signed by the CA certificate imported from the Principal cluster.

Configure OpenShift GitOps Subscription on the Spoke Cluster

We need to modify the Subscription openshift-gitops-operator and add the following environment variables:

spec:
config:
env:
- name: DISABLE_DEFAULT_ARGOCD_INSTANCE (1)
value: 'true'
- name: ARGOCD_CLUSTER_CONFIG_NAMESPACES (2)
value: '*'

1	Disable the default Argo CD instance.
2	Tell the Operator for which namespaces it should feel responsible for. In this case, all Namespaces.

Create Argo CD Instance on the Agent Cluster

To create a minimalistic Argo CD instance on the Agent cluster, we can use the following Argo CD configuration:

apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
name: agent-argocd
namespace: argocd-agent-bm01 (1)
spec:
server:
enabled: false

1	The namespace where the Argo CD instance is running. This is also the name of the Agent we have created earlier on the principal cluster.

This will create a lightweight Argo CD instance in the namespace argocd-agent-bm01.

oc get pod -n argocd-agent-bm01 --context bm
NAME READY STATUS RESTARTS AGE
agent-argocd-application-controller-0 1/1 Running 0 2m49s
agent-argocd-redis-5f6759f6fb-2fdnt 1/1 Running 0 2m49s
agent-argocd-repo-server-7949d97dfd-dsk6b 1/1 Running 0 2m49s

Installing the Agent

To install the agent we will use a Helm Chart provided by Red Hat. This will install the Agent component on the target cluster.

As a reminder, we have two modes of operation for the Agent:

Managed mode — the control plane/hub defines Argo CD applications and their specifications.
Autonomous mode — each workload cluster/spoke defines its own Argo CD applications and their specifications.

Create Required Network Policy

Before we start with the actual installation of the Agent, we need to ensure that the Redis instance on the spoke cluster is accessible for the Agent. We need to create a NetworkPolicy accordingly:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: argocd-agent-bm01-redis-network-policy
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: agent-argocd-redis (1)
ingress:
- ports:
- protocol: TCP
port: 6379
from:
- podSelector:
matchLabels:
app.kubernetes.io/name: argocd-agent-agent
policyTypes:
- Ingress

1	The name of the Redis instance. The label is based on <instance name>-redis.

Apply the NetworkPolicy to the spoke cluster:

oc apply -f network-policy.yaml -n argocd-agent-bm01 --context bm

Add the Helm Chart Repository

Add the Helm repository:

helm repo add openshift-helm-charts https://charts.openshift.io/
helm repo update

Install a managed Agent with the Helm Chart

Install the agent in the managed mode using the Helm Chart. The following parameters are used:

namespaceOverride - The namespace where the Agent is running.
agentMode - The mode of the Agent.
server - The server URL of the Principal component. This is the spec.host setting of the Principal’s route.
argoCdRedisSecretName - The name of the Redis secret.
argoCdRedisPasswordKey - The key of the Redis password.
redisAddress - The address of the Redis instance.

helm install redhat-argocd-agent openshift-helm-charts/redhat-argocd-agent \
--set namespaceOverride=argocd-agent-bm01 \
--set agentMode="managed" \
--set server="serverURL of principal route" \
--set argoCdRedisSecretName="agent-argocd-redis-initial-password" \
--set argoCdRedisPasswordKey="admin.password" \
--set redisAddress="agent-argocd-redis:6379" \
--kube-context "bm"

With this chart a pod on the spoke cluster will be created and will start to synchronize the Argo CD applications between the hub and the spoke cluster.

Verify the Managed Agent

To verify the Agent in managed mode we need to create an Argo CD Application on the hub cluster. We can try the following Application. The Application is taken from the openshift-clusterconfig-gitops repository and simply adds a banner to the top of the OpenShift UI. I typically use this as a quick test if GitOps is working.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: branding
namespace: argocd-agent-bm01 (1)
spec:
destination:
namespace: default
server: 'https://hub-argocd-agent-principal-resource-proxy:9090?agentName=argocd-agent-bm01' (2)
project: default
source:
path: clusters/management-cluster/branding
repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops'
targetRevision: main
syncPolicy:
automated:
prune: true
selfHeal: true

1	The namespace of the agent
2	The server URL of the Principal component. This is the URL plus the port and the agentName. As an alternative you can also use name: argocd-agent-bm01 which is the name of the cluster and might be easier to read.

Apply the Application to the hub cluster:

oc apply -f application.yaml -n argocd-agent-bm01 --context aws

Since the application will try to automatically synchronize the configuration, the status will change to Synced after a few seconds:

status:
resources:
- group: console.openshift.io
kind: ConsoleNotification
name: topbanner
status: Synced
version: v1

and the (top) banner will be visible on the UI:

Figure 1. Banner on the top of the UI

On the hub cluster, the Argo CD Application will be Synced:

oc get applications --context aws -A
NAMESPACE NAME SYNC STATUS HEALTH STATUS
argocd-agent-bm01 branding-banner Synced Healthy

Install an autonomous Agent with the Helm Chart

Let’s cleanup the first installation of the Chart (managed agent) in order to not have any conflicts.

helm uninstall redhat-argocd-agent --kube-context "bm"

Install the agent in the autonomous mode using the Helm Chart. The following parameters are used:

namespaceOverride - The namespace where the Agent is running.
agentMode - The mode of the Agent.
server - The server URL of the Principal component. This is the spec.host setting of the Principal’s route.
argoCdRedisSecretName - The name of the Redis secret.
argoCdRedisPasswordKey - The key of the Redis password.
redisAddress - The address of the Redis instance.

The only difference to the managed mode is the agentMode parameter.

helm install redhat-argocd-agent-autonomous openshift-helm-charts/redhat-argocd-agent \
--set namespaceOverride=argocd-agent-bm01 \
--set agentMode="autonomous" \
--set server="serverURL of principal route" \
--set argoCdRedisSecretName="agent-argocd-redis-initial-password" \
--set argoCdRedisPasswordKey="admin.password" \
--set redisAddress="agent-argocd-redis:6379" \
--kube-context "bm"

With this chart a pod on the spoke cluster will be created and will start to synchronize the Argo CD applications between the hub and the spoke cluster.

Verify the Autonomous Agent

To verify the Agent in autonomous mode we need to create an Argo CD Application on the spoke cluster.

We can try the following Application. It is basically the same as we used for the test for the managed mode, except that this time we will add a banner on the bottom of the UI.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: branding-bottom-banner
namespace: argocd-agent-bm01 (1)
spec:
destination:
namespace: default
server: 'https://kubernetes.default.svc' (2)
project: default
source:
path: clusters/management-cluster/branding-bottom
repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops'
targetRevision: main
syncPolicy:
automated:
prune: true
selfHeal: true

1	The namespace of the agent
2	The server URL of the local cluster, since in autonomous mode the application is managed locally.

Apply the Application to the spoke cluster:

oc apply -f application.yaml -n argocd-agent-bm01 --context bm

Like the managed mode, the status will change to Synced after a few seconds and the (this time) bottom banner will be visible on the UI:

Figure 2. Banner on the bottom of the UI

Moreover, the Argo CD Application will appear on the hub cluster:

oc get applications --context aws -A
NAMESPACE NAME SYNC STATUS HEALTH STATUS
argocd-agent-bm01 branding-bottom-banner Synced Healthy
argocd-agent-bm01 branding-banner Synced Healthy

Troubleshooting

During the installation and configuration of the Argo CD Agent, you might encounter some issues. Here are some issues I encountered during the tests:

Principal Pod Fails to Start

If the Principal pod fails to start with errors about missing secrets, verify that all required secrets have been created:

oc get secrets -n argocd-principal | grep argocd-agent

You should see the following secrets:

argocd-agent-ca - The CA certificate
argocd-agent-principal-tls - The Principal’s TLS certificate
argocd-agent-resource-proxy-tls - The Resource Proxy’s TLS certificate
argocd-agent-jwt - The JWT signing key

If any of these are missing, re-run the corresponding argocd-agentctl command to create them.

Redis Errors in the Principal Pod

When you see errors like the following in the logs of the Principal pod, ensure that the Argo CD instance does not have the controller enabled in that namespace (set spec.controller.enabled: false). Hopefully, this will change in the future.

time="2026-01-20T04:57:10Z" level=error msg="unexpected lack of '_' namespace/name separate: 'app|managed-resources|branding|1.8.3'" connUUID=3c13b30b-9f84-4af6-93d8-e1c03c4c7898 function=redisFxn module=redisProxy

Limitations

While the Argo CD Agent brings significant improvements, there are some limitations to be aware of:

Separate Argo CD Instance Required

Currently, the Principal component cannot be installed alongside an existing Argo CD instance where the application controller is enabled. You must create a separate Argo CD instance with the controller disabled (spec.controller.enabled: false). To me, this is one of the biggest limitations. However, this will be addressed in the future and is tracked in the issue: https://github.com/argoproj-labs/argocd-agent/issues/708

Manual Certificate Management

The mTLS certificates must be created and managed manually by the user. There is no automatic certificate rotation or renewal. For production environments, you should integrate with your organization’s PKI infrastructure and implement a certificate rotation strategy.

Summary

The Argo CD Agent provides a powerful solution for managing multiple clusters with Argo CD. By combining the benefits of centralized management with distributed application controllers, it addresses the scalability and single point of failure challenges of traditional deployment models.

While the initial setup requires several steps, especially around certificate management, the resulting architecture offers a robust foundation for GitOps at scale.

[Ep.14] Reusable Argo CD Application Template

Thu, 17 Jul 2025 00:00:00 +0000

When working with Argo CD at scale, you often find yourself creating similar Application manifests repeatedly. Each application needs the same basic structure but with different configurations for source repositories, destinations, and sync policies. Additionally, managing namespace metadata becomes tricky when you need to conditionally control whether Argo CD should manage namespace metadata based on sync options.

In this article, I’ll walk you through a reusable Helm template that solves these challenges by providing a flexible, DRY (Don’t Repeat Yourself) approach to creating Argo CD Applications. This template is available in my public Helm Chart library and can easily be used by anyone.

The Problem

Traditional Argo CD Application manifests suffer from several issues:

Repetitive Code: Each application requires similar boilerplate YAML
Configuration Validation: Manual validation of required fields across multiple applications
Maintenance Overhead: Changes to common patterns require updates across multiple files

But I can do this manually, right?

Of course, nobody prevents you from creating an Argo CD Application manifest manually or using the UI to enter the values there, but sometimes you just want to get things done faster or help your team with a consistent way to create Argo CD Applications. Often teams do not want to learn about a new tool like Argo CD and maybe you want to automate the creation by using a CI/CD pipeline.

The Solution: A Reusable Helm Template

I already work with a common template library for all of my Helm Charts. The idea is to have repeatable snippets in my tpl charts and I can reuse them in all other charts. Recently, I started testing the templating of entire Kubernetes manifests.

To address the above issues, I’ve created a comprehensive Helm template that will render an Argo CD Application. But let’s start with the end result. Your team wants to create a new Argo CD Application. They can do this either via the UI, via the CLI, by creating the YAML file manually, or through pull request or CI/CD integration. Ultimately, the minimum required information is:

The name of the application
The namespace of the application
The source repository
The destination repository

So something like this:

argocd_applications:
my-app: (1)
namespace: "openshift-gitops"
source: (2)
repositoryURL: "https://github.com/argoproj/argocd-example-apps.git"
targetRevision: "HEAD"
path: "guestbook"
destination: (3)
server: "https://kubernetes.default.svc" (4)
namespace: "guestbook"
my-second-app:
....

1	This key will become the name of the Application
2	The source repository, this is the repository that contains the Helm chart or the Kubernetes manifests
3	The destination, that defines the cluster and namespace where the application will be deployed
4	Either server or name must be set, but not both.

With the above values multiple applications can be created at once.

Integrating the Template

The values from the above example can be used to create the Application manifest.

All the developers (or CI/CD pipelines) need to do is to define the values and create one template to include the source template. This will look like this:

{{- if .Values.argocd_applications }}
{{- range $name, $config := .Values.argocd_applications }} (1)
{{- if $config.enabled | default false }} (2)
---
{{- include "tpl.argocdApplication" (dict "name" $name "spec" $config) -}} (3)
{{- end }}
{{- end }}
{{- end }}

1	Iterate over the applications, defining $name and $config
2	Only include the application if it is enabled
3	Include the template with the name of the application and the specification

This is everything you need to do to create an Argo CD Application. Let’s take a look at the template.

Key Features of the template

1. Flexible Destination Configuration

The template supports both server URL and cluster name destinations with validation:

destination:
{{- if and ($spec.destination.server) ($spec.destination.name) }}
{{ fail "destination.server and destination.name cannot be set at the same time" }}
{{- else }}
{{- if $spec.destination.server }}
server: {{ $spec.destination.server }}
{{- else if $spec.destination.name }}
name: {{ $spec.destination.name }}
{{- else }}
server: https://kubernetes.default.svc
{{- end }}
{{- end }}
namespace: {{ $spec.destination.namespace | required "destination.namespace is required" }}

2. Comprehensive Sync Policy Support

The template handles all Argo CD sync policy features:

Automated sync with prune, selfHeal, and allowEmpty options
Flexible sync options array
Retry configuration with backoff strategies
Conditional managed namespace metadata

3. Conditional Namespace Management

The intelligent handling of managedNamespaceMetadata. The template only includes this section when it CreateNamespace= option is set to true:

{{- if and $spec.syncPolicy.syncOptions (not (has "CreateNamespace=false" $spec.syncPolicy.syncOptions)) }}
{{- if $spec.syncPolicy.managedNamespaceMetadata }}
managedNamespaceMetadata:
{{- with $spec.syncPolicy.managedNamespaceMetadata.labels }}
labels:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with $spec.syncPolicy.managedNamespaceMetadata.annotations }}
annotations:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- end }}
{{- end }}

4. Other Features

I created the template to be as flexible as possible. However, I did not include everything in this template, only the most important features (from my point of view). Currently, the following is possible:

Create a template for an Argo CD Application using Git
Create a template for an Argo CD Application using Helm defining all possible Helm parameters, like additional values files or other options.
Using a single source
Set required annotations and labels

5. Not possible (currently)

Along with the supported features, there are some features that are currently not possible:

Defining multiple sources
Configure Kustomize settings

However, if you feel this needs to be added, please let me know and create an issue. I can then try to add it.

Why This Approach Works

1. DRY Principle

Instead of repeating the same YAML structure across multiple applications, you define it once and reuse it everywhere.

2. Intelligent Defaults

The template provides sensible defaults (like openshift-gitops namespace) while allowing customization when needed.

3. Validation

Built-in validation ensures required fields are present and conflicting configurations are caught early.

4. Conditional Logic

The template handles complex scenarios like namespace management automatically, reducing the chance of misconfigurations.

Real-World Benefits

In practice, this template has several advantages:

Consistency: All applications follow the same pattern
Maintainability: Changes to common patterns are made in one place
Safety: Validation prevents common misconfigurations
Flexibility: Supports the full range of Argo CD features
Development Guidelines: Ensure all developers are using the same process

Real-World Examples

1. Define a UI Branding

I would like to define a top banner in the OpenShift Console. Everything is defined in the clusters/management-cluster/branding folder in my git repository.

All I need to do is to define the following values:

argocd_applications:
my-branding:
enabled: true
namespace: "openshift-gitops"
source:
repositoryURL: "https://github.com/tjungbauer/openshift-clusterconfig-gitops"
targetRevision: "main"
path: "clusters/management-cluster/branding"
destination:
name: in-cluster
namespace: "default"

2. Define a UI Branding with custom Helm value

Like above I would like to define a top banner in the OpenShift Console. This time I want to use a custom Helm value to define the background color of the banner.

argocd_applications:
my-branding-custom-helm:
enabled: true
namespace: "openshift-gitops"
source:
repositoryURL: "https://github.com/tjungbauer/openshift-clusterconfig-gitops"
targetRevision: "main"
path: "clusters/management-cluster/branding"
helm:
parameters:
- name: generic-cluster-config.console.console_banners.topbanner.backgroundcolor
value: '#FF9843'
destination:
name: in-cluster
namespace: "default"

3. Full-Blown Example

A full example with all features can be found in my Git repository at: values_example_ArgoCD-Application.yaml

What about Validation?

Above I mentioned that the template is able to validate the values. This is true for the most important parts.

For example, try to define the following values:

[...]
destination:
name: in-cluster
server: "https://kubernetes.default.svc"
namespace: "default"

name and server are not allowed to be set at the same time.

Helm (and Argo CD which is using Helm) will validate the values and fail with an error.

Error: destination.server and destination.name cannot be set at the same time

Conclusion

This Argo CD Application template demonstrates how Helm’s templating capabilities can solve real-world GitOps challenges. By combining conditional logic, validation, and sensible defaults, we create a tool that’s both powerful and easy to use.

The conditional namespace management feature alone saves hours of debugging why Argo CD isn’t behaving as expected with namespace metadata. When you combine this with the DRY benefits and built-in validation, you get a robust foundation for managing Argo CD applications at scale.

Whether you’re managing a few applications or hundreds, this template pattern will help you maintain consistency, reduce errors, and improve your team’s GitOps experience.

[Ep.13] ApplicationSet with Matrix Generator

Thu, 17 Apr 2025 00:00:00 +0000

During my day-to-day business, I am discussing the following setup with many customers: Configure App-of-Apps. Here I try to explain how I use an ApplicationSet that watches over a folder in Git and automatically adds a new Argo CD Application whenever a new folder is found. This works great, but there is a catch: The ApplicationSet uses the same Namespace default for all Applications. This is not always desired, especially when you have different teams working on different Applications.

Recently I was asked by the customer if this can be fixed and if it is possible to define different Namespaces for each Application. The answer is yes, and I would like to show you how to do this.

The Current Situation

Currently, I am (or was) using the following ApplicationSet to watch over a folder in Git. The ApplicationSet uses the Matrix Generator to create a new Argo CD Application for each folder found in the Git repository. It also uses the list operator to define the targetCluster:

 generatormatrix:
# Git: Walking through the specific folder and take whatever is there.
- git:
directories:
- path: clusters/management-cluster/*
repoURL: *repourl
revision: *branch
# List: simply define the targetCluster. The name of the cluster must be known by Argo CD
- list:
elements:
# targetCluster is important, this will define on which cluster it will be rolled out.
# The cluster name must be known in Argo CD
- targetCluster: *mgmtclustername

This will create a new Application for any subfolder found in the clusters/management-cluster/ folder any every Application in Argo CD will be configured with the same target namespace: default.

Technically, this is not a problem, as I define the exact namespace in the different Helm Charts, but it is not always desired.

I personally recommend defining the Namespace in the Helm Charts, since especially for the cluster configuration, sometimes there is no clear target Namespace or multiple Namespaces are modified.

What did not work

The first idea was to use the Matrix generator and define the targetNamespace in list.elements and if the namespace is not defined, use a default one. So similar like this:

 generatormatrix:
# Git: Walking through the specific folder and take whatever is there.
- git:
directories:
- path: clusters/management-cluster/*
repoURL: *repourl
revision: *branch
# List: simply define the targetCluster. The name of the cluster must be known by Argo CD
- list:
elements:
# targetCluster is important, this will define on which cluster it will be rolled out.
# The cluster name must be known in Argo CD
- targetCluster: *mgmtclustername
path: clusters/management-cluster/cert-manager
targetNamespace: cert-manager
- targetCluster: *mgmtclustername
path: clusters/management-cluster/*
targetNamespace: default

To make is short: This does not work

The matrix operator walks over all folders and creates a cartesian product of the elements. This means, it will create a new Application for each folder and each element in the list. So if you have 10 folders and 2 elements in the list, you will end up with 20 Applications. This is not what we want. We want to create a new Application for each folder and define the targetNamespace in the Git repository.

The second test was the use of the Merge generator. This did not work as well, as it was not possible to define a default Namespace.

The Solution

The solution is to use the Git FILES generator and define the targetNamespace in the Git repository. This is done by creating a file called config.json in each subfolder. The content of the file is simple:

{
"namespace": "default",
"environment": "in-cluster"
}

The advantage is that it is possible to define multiple parameters in that file. However, the disadvantage is that this file must be created, otherwise the ApplicationSet will ignore the folder and will not create a new Application. I think this is a small disadvantage, and the file is easy to maintain.

Bringing everything together now opens two possibilities:

This will require helper-argocd version 2.0.41 or higher.

Option 1: Keep Matrix Generator and use Git File sub-generator

The first option simply replaces the git directory generator with the git file generator. The rest of the ApplicationSet remains unchanged.

I like this option somehow better than the second one, because I can keep everything as I had it before, the only thing is to create the config.json file in each subfolder and change two lines in the ApplicationSet.

 # Switch to set the namespace to '.namespace' ... must be defined in config.json
use_configured_namespace: true (1)
# Definition of Matrix Generator. Only 2 generators are supported at the moment
generatormatrix:
# Git: Walking through the specific folder and take whatever is there.
- git:
files: (2)
- path: clusters/management-cluster/**/config.json (3)
repoURL: *repourl
revision: *branch
# List: simply define the targetCluster. The name of the cluster must be known by Argo CD
- list:
elements:
# targetCluster is important, this will define on which cluster it will be rolled out.
# The cluster name must be known in Argo CD
- targetCluster: *mgmtclustername

1	Switch to use the configured namespace. This is important, otherwise the namespace is set to "default". This was added for backward compatibility.
2	The git file generator is used instead of the git directory generator.
3	The path is changed to the config.json file. The ** is important, as it defines to look into every subfolder.

The config.json can be shortened to:

{
"namespace": "default"
}

Option 2: Switch to plain Git File Generator

The second option is to switch to the plain Git generator. This removes the Matrix generator, but also requires defining the targetCluster in the config.json file. This is not a problem, as the config.json file can be used to define multiple parameters.

 generatorgit: (1)
# Git: Walking through the specific folder and take whatever is there.
- files:
- clusters/management-cluster/**/config.json
repourl: *repourl
revision: *branch

1	No Matrix but Git generator instead.

Here the full config.json file is required, otherwise the targetCluster is not defined:

{
"namespace": "default",
"environment": "in-cluster"
}

Full working example

Source: https://github.com/tjungbauer/openshift-clusterconfig-gitops/blob/main/base/argocd-resources-manager/values.yaml

applicationsets:
######################################
# MATRIX GENERATOR EXAMPLE Git Files #
######################################
# The idea behind the GIT Generate (File) is to walk over a folder, for example /clusters/management-cluster and fetch a config.json from each folder.
# This is more or less similar as the Matrix generator (see below), but reqires a bit more configuration ... the config.json.
# The advantage is that you can configure individual namespaces for example in this config.json and provide an additional information
mgmt-cluster-matrix-gitfiles:
enabled: true
# Description - always usful
description: "ApplicationSet that Deploys on Management Cluster Configuration (using Git Generator)"
# Any labels you would like to add to the Application. Good to filter it in the Argo CD UI.
labels:
category: configuration
env: mgmt-cluster
# Using go text template. See: https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/GoTemplate/
goTemplate: true
argocd_project: *mgmtclustername
environment: *mgmtclustername
# preserve all resources when the application get deleted. This is useful to keep that workload even if Argo CD is removed or severely changed.
preserveResourcesOnDeletion: true
# Switch to set the namespace to '.namespace' ... must be defined in config.json
use_configured_namespace: true
# Definition of Matrix Generator. Only 2 generators are supported at the moment
generatormatrix:
# Git: Walking through the specific folder and take whatever is there.
- git:
files:
- path: clusters/management-cluster/**/config.json
repoURL: *repourl
revision: *branch
# List: simply define the targetCluster. The name of the cluster must be known by Argo CD
- list:
elements:
# targetCluster is important, this will define on which cluster it will be rolled out.
# The cluster name must be known in Argo CD
- targetCluster: *mgmtclustername
syncPolicy:
autosync_enabled: false
# Retrying in case the sync failed.
retries:
# number of failed sync attempt retries; unlimited number of attempts if less than 0
limit: 5
backoff:
# the amount to back off. Default unit is seconds, but could also be a duration (e.g. "2m", "1h")
# Default: 5s
duration: 5s
# a factor to multiply the base duration after each failed retry
# Default: 2
factor: 2
# the maximum amount of time allowed for the backoff strategy
# Default: 3m
maxDuration: 3m

Conclusion

In this blog post I have shown you how to use the ApplicationSet with the Matrix generator and define individual Namespaces for each Application. This is done by using the Git File generator and defining a config.json file in each subfolder. The config.json file can be used to define multiple parameters, but it is required to create the file in each subfolder. This is a small disadvantage, but I think it is worth the effort. The advantage is that you can define individual Namespaces for each Application, and you can use the same ApplicationSet for all your Applications.

I hope this blog post was helpful and you learned something new. If you have any questions or comments, please feel free to reach out to me. I am happy to help you.

[Ep.12] Using Kustomize Post-Renderer

Sun, 13 Oct 2024 00:00:00 +0000

Lately I came across several issues where a given Helm Chart must be modified after it has been rendered by Argo CD. Argo CD does a helm template to render a Chart. Sometimes, especially when you work with Subcharts or when a specific setting is not yet supported by the Chart, you need to modify it later … you need to post-render the Chart.

In this very short article, I would like to demonstrate this on a real-live example I had to do. I would like to inject annotations to a Route objects, so that the certificate can be injected. This is done by the cert-utils operator. For the post-rendering the Argo CD repo pod will be extended with a sidecar container, that is watching for the repos and patches them if required.

Everything below is using OpenShift Gitops Operator. This is based on Argo CD, but instead of directly modifying the repo Deployment, we will modify the Argo CD Custom Resource.

In the future it will be easier to inject certificates into a Route, by defining a Secret. This as currently a TechPreview feature (OpenShift 4.17). Creating a route with externally managed certificate

The Route Object

Imagine we have the following Route object, rendered via Helm template:

---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
name: my-route
namespace: my-namespace
spec:
host: my.route.apps.cluster.name
port:
targetPort: http
tls:
insecureEdgeTerminationPolicy: Redirect
termination: edge
to:
kind: Service
name: my-service
weight: 100
wildcardPolicy: None

The cert-manager Operator requested a certificate which can be found in the Secret "my-certificate ". To let the cert-utils Operator inject the data from the certificate automatically, we need to add annotations to that Route object.

This injection is usually a good idea, since we do not want to define certificate and (private) key directly in the Route object using our Chart.

apiVersion: route.openshift.io/v1
kind: Route
metadata:
annotations: (1)
cert-manager.io/cluster-issuer: my-issuer
cert-utils-operator.redhat-cop.io/certs-from-secret: my-certificate

1	Two annotations shall be added to the Route object.

In this example certificates have to be ordered. No wildcard certificate is available.

Post-Rendering

To modify the output after it has been rendered by Argo CD we will use Kustomize patch feature. This means, after the template has been rendered, we send it to Kustomize and let it patch it.

Let’s go through the steps one-by-one:

Create a kustomization.yaml Place the following file next to your Chart.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: my-namespace
resources:
- ./all.yaml (1)
patches:
- patch: |
- op: add (2)
path: /metadata/annotations
value:
cert-manager.io/cluster-issuer: my-issuer
cert-utils-operator.redhat-cop.io/certs-from-secret: my-certificate
target:
kind: Route
name: my-route (3)

1	The all.yaml file will be created by the helm template command.
2	Add the annotations to the Route object.
3	The name of the Route object.

This will patch the Route object. You can test this locally by execute the command: helm template . > all.yaml && kustomize build && rm all.yaml

Create an empty file called my-cmp-plugin into the folder next to the Chart.yaml I will explain in a bit why I chose to use this approach.
Create the following ConfigMap in the OpenShift GitOps namespace (for example openshift-gitops)

kind: ConfigMap
apiVersion: v1
metadata:
name: my-cmp-plugin
namespace: openshift-gitops
data:
plugin.yaml: |-
apiVersion: argoproj.io/v1alpha1
kind: ConfigManagementPlugin
metadata:
name: my-cmp-plugin (1)
spec:
version: v1.0
init: (2)
command: [sh, -c, 'echo "Initializing my-plugin-cmp..."', 'helm dependency build || true']
generate: (3)
command: [sh, -c, "helm template . --name-template $ARGOCD_APP_NAME --namespace $ARGOCD_APP_NAMESPACE --include-crds > all.yaml && kustomize build"]
discover: (4)
find:
glob: "**/my-cmp-plugin"

1	The name of the plugin.
2	The init command will be executed once, when the plugin is loaded.
3	The generate command will be executed every time the plugin is called.
4	The discovery command will be executed to find the plugin.

This will execute the command to generate a helm template, pipe the output into all.yaml and let Kustomize patch the output. The "discovery" part is looking for a specific file in the repository. I thought this might be useful to pin down this plugin to specific repositories only. However, there are other ways to implement this. You could omit this part and define the name of the plugin inside the Argo CD Application too for example.

Patching Argo CD Repo server

Now it is time to patch our repo server specification of the Argo CD custom resource. The following should do it:

As image for the sidecar container, I am using Gerald Nunn’s tool image. You can use your own image, as long as Helm and Kustomize are available.

apiVersion: argoproj.io/v1alpha1
kind: ArgoCD
metadata:
name: openshift-gitops
namespace: openshift-gitops
spec:
[...]
repo:
- configMap: (1)
name: my-cmp-plugin
name: my-cmp-plugin
sidecarContainers: (2)
- name: my-cmp-plugin
command: [/var/run/argocd/argocd-cmp-server]
env:
- name: APP_ENV
value: prod
image: quay.io/gnunn/tools:latest (3)
imagePullPolicy: Always
securityContext:
runAsNonRoot: true
volumeMounts: (4)
- mountPath: /var/run/argocd
name: var-files
- mountPath: /home/argocd/cmp-server/plugins
name: plugins
- mountPath: /tmp
name: tmp
- mountPath: /home/argocd/cmp-server/config/plugin.yaml
subPath: plugin.yaml
name: my-cmp-plugin
volumes: (5)
- configMap:
name: cloudbees-cmp-plugin
name: cloudbees-cmp-plugin

1	The name of the ConfigMap that was created in step 2.
2	The sidecar container specification.
3	The image that is used for the sidecar container.
4	The volume mounts for the sidecar container.
5	The volumes for the sidecar container.

As soon as the repo Pod has been patched a 2nd container inside the Pod will be started as a sidecar. This will take the ConfigMap that was created in step 2 and mount it. As soon as a repo is found where this patch shall be executed, Argo CD will perform the actions defined in the ConfigMap, resulting in the output of the helm template and the patched output of Kustomize.

---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
name: my-route
namespace: my-namespace
annotations: (1)
cert-manager.io/cluster-issuer: my-issuer
cert-utils-operator.redhat-cop.io/certs-from-secret: my-certificate
spec:
host: my.route.apps.cluster.name
port:
targetPort: http
tls:
insecureEdgeTerminationPolicy: Redirect
termination: edge
to:
kind: Service
name: my-service
weight: 100
wildcardPolicy: None

1	The annotations that are added to the Route.

This is it; this will patch our resource. Such post-renderer can be used for other patches as well. For example, to remove certain items from an object.

2nd Example

In my real-live example I had the problem that the path was empty in the Helm Chart and OpenShift automatically removed that, which was shown as out-of-sync in Argo CD.

So I am using the patch to remove the path.

Only do this if you are sure the element is really empty!

I extended the kustomization.yaml with

 - op: remove
path: /spec/path

so it looks like:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: my-namespace
resources:
- ./all.yaml
patches:
- patch: |
- op: add
path: /metadata/annotations
value:
cert-manager.io/cluster-issuer: my-issuer
cert-utils-operator.redhat-cop.io/certs-from-secret: my-certificate
- op: remove (1)
path: /spec/path
target:
kind: Route
name: my-route

1	The patch that removes the path.

This 2nd patch will completely remove the /spec/path from the Route object named my-route.

Further information:

Example, which was the base of my patch: Plugin Sidecar
G.Nunn’s tools image (Thanks for everything): https://quay.io/repository/gnunn/tools

[Ep.11] Managing Certificates

Thu, 04 Jul 2024 00:00:00 +0000

The article SSL Certificate Management for OpenShift on AWS explains how to use the Cert-Manager Operator to request and install a new SSL Certificate. This time, I would like to leverage the GitOps approach using the Helm Chart cert-manager I have prepared to deploy the Operator and order new Certificates.

I will use an ACME Letsencrypt issuer with a DNS challenge. My domain is hosted at AWS Route 53.

However, any other integration can be easily used.

Before we start

Before we start, be sure that Route 53 is configured correctly. The required settings and commands are described at Configure an AWS User for Accessing Route 53

Deploy the Operator

The first step is to deploy the Operator to our cluster. This is done using GitOps and the Helm Chart is located at my Helm repository: https://github.com/tjungbauer/openshift-clusterconfig-gitops/tree/main/clusters/management-cluster/cert-manager

The configuration looks like below. It takes care to:

Deploy the Operator cert-manager-operator
Verify if the Operator has been deployed successfully
Configure Cert-Manager
1. Create a ClusterIssuer using route53 integration. (You can configure any other configuration too)
2. Patch the Operator with "overrideArgs". This is required for AWS Route 53 where we need to define which DNS resolvers shall be used.

All these settings are handed over to the appropriate sub-charts. Like helper-operator, helper-status-checker and cert-manager

# Install Operator Compliance Operator
# Deploys Operator --> Subscription and Operatorgroup
# Syncwave: 0
helper-operator: (1)
operators:
compliance-operator:
enabled: true
syncwave: '0'
namespace:
name: cert-manager-operator
create: true
subscription:
channel: stable-v1
approval: Automatic
operatorName: openshift-cert-manager-operator
source: redhat-operators
sourceNamespace: openshift-marketplace
operatorgroup:
create: true
notownnamespace: false
helper-status-checker: (2)
enabled: true
checks:
- operatorName: cert-manager-operator
namespace:
name: cert-manager-operator
serviceAccount:
name: "status-checker-cert-manager"
cert-manager: (3)
certManager:
enable_patch: true
overrideArgs: (4)
- '--dns01-recursive-nameservers-only'
- --dns01-recursive-nameservers=ns-362.awsdns-45.com:53,ns-930.awsdns-52.net:53
issuer: (5)
- name: letsencrypt-prod
type: ClusterIssuer
enabled: true
syncwave: 20
acme:
email: tjungbau@redhat.com
solvers:
- dns01: (6)
route53:
accessKeyIDSecretRef:
key: access-key-id
name: prod-route53-credentials-secret
region: us-west-1
secretAccessKeySecretRef:
key: secret-access-key
name: prod-route53-credentials-secret
selector:
dnsZones:
- aws.ispworld.at

1	Installing the Operator
2	Verify if the Operator has been successfully deployed
3	Configure the Cert-Manager Operator
4	Override the DNS resolver args
5	Configure the ClusterIssuer
6	Use the solver dns01.route53.

Verify the README of the Helm Chart cert-manager for additional possibilities in the configuration.

One additional piece is missing before we can finally start the deployment.

As you can see in the values file above the accessKey and secretAccessKey are stored in the secret named prod-route53-credentials-secret.

This means, that a secret is required with the keys that have been provided by AWS when you configured the Route 53 access:

kind: Secret
apiVersion: v1
metadata:
name: prod-route53-credentials-secret
namespace: cert-manager (1)
data:
access-key-id: <AccessKey>
secret-access-key: <Secret Access Key>
type: Opaque

1	Namespace of the Secret, here the Operator is managing the Certificate Controller.

I stored this Secret as SealedSecret and put it into the cluster configuration folder. From here, Argo CD will pick it up and deploy it.

Never, never ever store a Secret object directly in Git. Secret objects are not encrypted but encoded. Everybody could decode the data. With Sealed Secrets or any other Secret Management, you are able to prepare these objects and store or retrieve them.

Finally, with these settings, the Operator can be deployed. This is managed by OpenShift GitOps (Argo CD). As soon as the Operator is ready, we can start requesting certificates as we automatically created the ClusterIssuer (letsencrypt-prod)

Figure 1. Deploying and Configuring Cert-Manager Operator

Two certificates are of special interest :

Default IngressController of OpenShift
OpenShift’s API

Therefore, let’s request and configure them.

Requesting a Certificate

The chart cert-manager can render a Certificate resource as well. I tried to support any possible setting. However, not everything, especially the non-stable ones, is available yet.

The official Cert-Manager documentation explains how to create such Certificate Resource.

The chart README explains which settings are supported by the chart.

Not every setting is required and some will set default values. The minimum parameters are: name, namespace, secretName, dnsNames and reference to an issuer.

Requesting IngressController Certificate

The default IngressController of OpenShift listens on the wildcard domain *.apps.clustername.basedomain. In my examples, you will see *.apps.ocp.aws.ispworld.at

The IngressController configuration must be modified to reference the Secret object the cert-manager will generate once the Certificate has been successfully requested. The cluster configuration Ingresscontroller defines the required parameters:

---
# -- Define ingressControllers
# Multiple might be defined.
ingresscontrollers: (1)
# -- Name of the IngressController. OpenShift initial IngressController is called 'default'.
- name: default
# -- Enable the configuration
# @default -- false
enabled: true
# -- Number of replicas for this IngressController
# @default -- 2
replicas: 3
# -- The name of the secret that stores the certificate information for the IngressController
# @default -- N/A
defaultCertificate: router-certificate (2)
# -- Bind IngressController to specific nodes
# Here as example for Infrastructure nodes.
# @default -- empty
#nodePlacement:
# NodeSelector that shall be used.
# nodeSelector: (3)
# key: node-role.kubernetes.io/infra
# value: ''
# # -- Tolerations, required if the nodes are tainted.
# tolerations:
# - effect: NoSchedule
# key: node-role.kubernetes.io/infra
# operator: Equal
# value: reserved
# - effect: NoExecute
# key: node-role.kubernetes.io/infra
# operator: Equal
# value: reserved
certificates:
enabled: true
# List of certificates
certificate: (4)
- name: router-certificate
enabled: true
namespace: openshift-ingress
syncwave: "0"
secretName: router-certificate (5)
dnsNames: (5)
- apps.ocp.aws.ispworld.at
- '*.apps.ocp.aws.ispworld.at'
# Reference to the issuer that shall be used.
issuerRef: (6)
name: letsencrypt-prod
kind: ClusterIssuer

1	Configuration for the IngressController
2	Reference to the Secret that will store the Certificate
3	Optional tolerations that can be configured for the IngressController
4	List of Certificates to order
5	List of domainnames for the IngressController. Here 2 are used, the wildcard domain and the base domain of that wildcard.
6	Reference to the issuer (in this case ClusterIssuer)

This will request the Certificate:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: router-certificate
namespace: openshift-ingress
spec:
dnsNames:
- apps.ocp.aws.ispworld.at
- '*.apps.ocp.aws.ispworld.at'
duration: 2160h0m0s
issuerRef:
kind: ClusterIssuer
name: letsencrypt-prod
privateKey:
algorithm: RSA
encoding: PKCS1
rotationPolicy: Always
secretName: router-certificate

It may take a while until the Certificate request is approved.

The IngressController will update the reference to the secret and restart the ingress pods:

apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
name: default
namespace: openshift-ingress-operator
spec:
[...]
defaultCertificate:
name: router-certificate

Once all pods have been successfully restarted, open a new browser, or reload or open a private window to verify the certificate that is provided by the application.

Requesting APIServer Certificate

Requesting the certificate for the OpenShift API follows the same rules as for the IngressController. The example can be found at: Clusterconfig APIServer

The values file may look like the following for example:

In this case, not only the Certificate is requested but the ETCD encryption is also enabled. The reason for that is, that both settings are done in the same Kubernetes resource (apiserver). If we split this up into 2 Argo CD Applications one of them will always show a warning that the same resource is managed by another Argo CD Application.

---
# -- Using subchart generic-cluster-config
generic-cluster-config:
apiserver: (1)
enabled: true
# audit configuration
audit:
profile: Default
# Configure a custom certificate for the API server
custom_cert:
enabled: true
cert_names: (2)
- api.ocp.aws.ispworld.at
secretname: api-certificate (3)
etcd_encryption: (4)
enabled: true
encryption_type: aesgcm (5)
# -- Namespace where Job is executed that verifies the status of the encryption
namespace: kube-system
serviceAccount:
create: true
name: "etcd-encryption-checker" (6)
cert-manager: (7)
enabled: true
certificates:
enabled: true
# List of certificates
certificate:
- name: api-certificate
enabled: true
namespace: openshift-config
syncwave: "0"
secretName: api-certificate
dnsNames:
- api.ocp.aws.ispworld.at
# Reference to the issuer that shall be used.
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer

1	Settings for the APIServer object.
2	The name of the domain the certificate will be responsible for.
3	Reference to the Secret that will store the Certificate
4	Enable ETCD encryption
5	Tpee of encryption
6	Service Account that will be created and used by a Job that will verify when and if the encryption has been finished successfully.
7	Settings for the Certificate. Similar to the settings of the IngressController.

The configuration is more or less similar to the IngressController. Again the APIServer will restart and once done, the Certificate is used by the cluster.

Conclusion

With this, very short, article I have tried to easily explain how to deploy the Cert-Manager Operator and request Certificates. Different Helm Charts are used, but the main one is cert-manager.

The cluster configuration repository https://github.com/tjungbauer/openshift-clusterconfig-gitops then use this chart to configure the required resources.

With the support of this Helm Chart anybody in the Cluster can request Certificates which are then managed by the Cert-Manager Operator.

[Ep.10] Update Cluster Version

Fri, 07 Jun 2024 00:00:00 +0000

During a GitOps journey at one point, the question arises, how to update a cluster? Nowadays it is very easy to update a cluster using CLI or WebUI, so why bother with GitOps in that case? The reason is simple: Using GitOps you can be sure that all clusters are updated to the correct, required version and the version of each cluster is also managed in Git.

All you need is the channel you want to use and the desired cluster version. Optionally, you can define the exact image SHA. This might be required when you are operating in a restricted environment.

Overview

Since OpenShift 4 it is very easy to update a cluster. It can either be done using the Web UI Administration → Cluster Settings or via the command line using the command oc adm upgrade.

There are 2 main configurations for an upgrade:

channel: This declares the update strategy tied to versions of OpenShift.
The desired version: This is the target version the cluster should be updated to.

To get the current clusterversion you can use the following command:

❯ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.14.1 True False 4m38s Cluster version is 4.14.1

Here we can see that the version is currently 4.14.1 and no upgrade is currently progressing.

Now we want to update to the latest possible version.

Where to find the available updates?

Before you start the update, you will need to fetch the possible available updates.

This information can be gathered with the command oc adm upgrade or oc get clusterversion/version -o yaml.

The output will look like the following:

Cluster version is 4.14.1
Upstream is unset, so the cluster will use an appropriate default.
Channel: stable-4.14 (available channels: candidate-4.14, eus-4.14, fast-4.14, stable-4.14, candidate-4.15, fast-4.15, stable-4.15) (1)
Recommended updates: (2)
VERSION IMAGE
4.14.27 quay.io/openshift-release-dev/ocp-release@sha256:4d30b359aa6600a89ed49ce6a9a5fdab54092bcb821a25480fdfbc47e66af9ec
4.14.26 quay.io/openshift-release-dev/ocp-release@sha256:4fe7d4ccf4d967a309f83118f1a380a656a733d7fcee1dbaf4d51752a6372890
[...]

1	Current Channel and available channels
2	List of available updates

The command describes the current channel (stable-4.14) and a list of possible updates. (The list is much longer).

The latest available version is 4.14.27 and the list of possible channels is all the 4.14 channels, but also 4.15 channels. This means we could change the channel here as well.

It is your responsibility to find the correct and supported upgrade paths. Not all upgrades to any version are supported. Also, do not rely on consecutive path numbers, some versions were never available.

Channels

Channels help users to define the timing and level of support for their environment. The following channels typically exist:

fast: This channel is fully supported but not yet fully tested and can be used to quickly update to the latest GA release. For example, when a certain bug is triggered in the current version.
stable: This is the latest stable version. Versions here are added after enough data points have been collected and therefore have some delay.
candidate: This channel offers unsupported early access to specific versions.
eus: All even-numbered versions offer an Externed Update Support (EUS) channel. They allow EUS-to-EUS updates and have a longer support phase.

Helm Chart - update-clusterversion

Now we want to update the version to the latest of the current channel 4.14.27 and change the channel to stable-4.15. The Helm Chart update-clusterversion has been created to help with a cluster update.

It will modify the clusterversion resource.

The required values are quite simple:

channel: stable-4.15
desiredVersion: 4.14.27
image: ''

Be sure that you choose the correct and available updates.

This is everything we need. As soon as this is synchronized into the cluster, the update process will be started by the cluster.

GitOps Synchronization

When we verify the current version in the OpenShift UI, we will see the possibility of an upgrade:

Figure 1. Cluster before the update process starts

In OpenShift GitOps we have the Application to start the update process:

Figure 2. Syncing new version

After a few seconds OpenShift will start with the update:

Figure 3. Progressing Cluster Update

This can also be verified via the command line

❯ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.14.1 True True 95s Working towards 4.14.27: 116 of 860 done (13% complete), waiting on etcd, kube-apiserver

Eventually, the cluster update process finishes successfully. We are now on version 4.14.27 and using the channel stable-4.15.

We can see that the next upgrade with be to version 4.15.15. This means we can directly upgrade to this version. This is possible because we switched the channel to stable-4.15.

Other channels, like candidate-4.15 might offer different, but not recommended, versions.

❯ oc adm upgrade
Cluster version is 4.14.27
Upstream is unset, so the cluster will use an appropriate default.
Channel: stable-4.15 (available channels: candidate-4.14, candidate-4.15, eus-4.14, fast-4.14, fast-4.15, stable-4.14, stable-4.15)
Recommended updates:
VERSION IMAGE
4.15.15 quay.io/openshift-release-dev/ocp-release@sha256:bb1182cd9001d6811dea8c5823235c17b9a316cce3bb13c51325250c14b46787

What about the SHA for the image?

The SHA for the image field in the ClusterVersion resource is required in certain scenarios to provide a precise reference to the container image that represents the OpenShift version you want to update to.

Such scenarios could be for example:

Offline or Restricted Networks In environments where clusters are running in offline or restricted network conditions, specifying the exact image SHA ensures that the cluster updates to a specific, known image that has been pre-pulled and is available within the network.
Precise Version Control Using the SHA ensures that the exact image version is used for the update, providing a higher level of precision and control.
Custom or Private Registries If you are using custom or private container registries, specifying the image SHA can help avoid ambiguities and ensure that the correct image is pulled from the correct registry.
Specific Compliance or Security Requirements Some regulations might require precise specification of container images, including SHAs, to ensure traceability and verifiability of the software components being deployed.

Conclusion

With this very simple method, it is easy to manage the version of multiple clusters via GitOps. Especially, where there is a bigger cluster fleet it will become essential to ensure which cluster has which version. Disconnected environments can also use the image setting to specify the exact SHA of an available update.

The current Helm Chart is very small and limited. It was created to quickly show the main use case: a simple and straightforward cluster upgrade.

Other possible options that might be required in the future. If you find anything missing, please let me know.

Please note, to always verify which version and channel is available and always consult the official documentation before an upgrade to find the latest release notes.

References: Updating Clusters

[Ep.9] Multiple Sources for Applications

Sun, 02 Jun 2024 00:00:00 +0000

Argo CD or OpenShift GitOps uses Applications or ApplicationSets to define the relationship between a source (Git) and a cluster. Typically, this is a 1:1 link, which means one Application is using one source to compare the cluster status. This can be a limitation. For example, if you are working with Helm Charts and a Helm repository, you do not want to re-build (or re-release) the whole chart just because you made a small change in the values file that is packaged into the repository. You want to separate the configuration of the chart with the Helm package.

The most common scenarios for multiple sources are (see: Argo CD documentation):

Your organization wants to use an external/public Helm chart
You want to override the Helm values with your own local values
You don’t want to clone the Helm chart locally as well because that would lead to duplication and you would need to monitor it manually for upstream changes.

This small article describes three different ways with a working example and tries to cover the advantages and disadvantages of each of them. They might be opinionated but some of them proved to be easier to use and manage.

Option 1: Multisource by Argo CD

Specifying multiple sources for an application is a beta feature.

The first option I would like to demonstrate is the "official way" by Argo CD. It was introduced in Argo CD version 2.6 and while this option is still in the Beta phase, it is one of the most requested features and was added to the release candidate of version 2.11. The article by Argo CD gives more details of this release. The official documentation by Argo CD can be found here

In my repository I am using this option as an example to create the App-of-Apps. The App-of-Apps created an Application, that uses multiple sources.

The values files from https://github.com/tjungbauer/openshift-clusterconfig-gitops/blob/main/base/argocd-resources-manager/values.yaml
The Helm Chart helper-argocd from https://charts.stderr.at/

Let’s see the working example:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: argocd-resources-manager
namespace: openshift-gitops
spec:
destination:
namespace: openshift-gitops
server: 'https://kubernetes.default.svc'
info:
- name: Description
value: >-
This is the starting point which will initialize all applicationsets or
argocd applications
project: default
sources: (1)
- chart: helper-argocd (2)
helm:
valueFiles: (3)
- $values/base/argocd-resources-manager/values.yaml
repoURL: 'https://charts.stderr.at/' (4)
targetRevision: 2.0.28 (5)
- ref: values (6)
repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops'
targetRevision: main
syncPolicy:
automated:
prune: true
selfHeal: true

1	Using sources instead of singular source
2	I want to use the Helm Chart helper-argocd
3	The chart will use the values file(s) from github.com/tjungbauer/openshift-clusterconfig-gitops/base/argocd-resources-manager/values.yaml. $value (can only be specified at the beginning of the path) resolves to the root of the values file repository
4	The URL of the Helm Chart repository
5	The version of the Helm chart helper-argocd
6	The reference to the values file, defining the repository URL and target revision.

Argo CD does not currently support using another Helm chart as a source for value files.

With this option, it is possible to separate the values files from the Helm Chart itself. Whenever I want to change something in the configuration, I simply change the values file without being required to release a new Chart version.

Advantages

Allows easily to define an Application with multiple sources, using Argo CD features.
No additional tool (see other options below) is required
Was added to the release candidate of Argo CD v2.11

Disadvantages

Currently a Beta feature and thus is not supported and has some limitations, such as lacking support of CLI and UI.
Does not allow using additional local specifications. At least as far as I know.
Can be complex to configure and manage

Option 2: Wrapper Helm Chart

With this option, which I extensively use, a wrapper Helm Chart is used to define local values files while calling additional (sub) Helm Charts as a dependency. This wrapper could simply define the values files and nothing else (being empty instead) or even define additional files, such as Sealed Secrets or things that are not provided by the sub-charts.

An example of an empty chart would be: setup-acs
An example of a chart that defines additional local files would be: generic-clusterconfig

The first one is using sub-charts to build the required specifications:

dependencies:
- name: rhacs-setup
version: ~1.0.0
repository: https://charts.stderr.at/
- name: helper-operator
version: ~1.0.23
repository: https://charts.stderr.at/
- name: helper-status-checker
version: ~4.0.0
repository: https://charts.stderr.at/
condition: helper-status-checker.enabled

The values file specifies the configuration for these sub-charts.

The second example also uses sub-charts, but additionally defines local files such as a SealedSecret for htpasswd.

As you can see throughout my repository I am using this option almost all the time. It proved to be quite simple, especially if you prefer working with Helm Charts such as I do. However, you must take care of the settings in the values file. Specifications you would like to be presented in a sub-chart must be put into the correct place.

For example:

Everything underneath

helper-operator:
operators:
rhacs-operator:
[...]

will be used by the chart helper-operator. While everything underneath:

helper-status-checker:
enabled: true

will be used by the chart helper-status-checker.

Advantages

Easy to use, at least for myself
Allows defining additional, local files that are not provided by the sub-charts

Disadvantages

A wrapper Chart must be created, that at least defines: Chart.yaml, templates folder and values.yaml
The configuration must be done correctly and all settings for a sub-chart must be forwarded to the sub-chart.

Option 3: Using Kustomize with Helm enabled

The third option I would like to show is using Kustomize. This tool can be used to call a Helm Chart when the option --enable-helm is activated in Argo CD. I am using one example for the IngressController.

Here the values file is placed into the local folder and the Kustomize.yaml is configured as:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
helmCharts:
- name: ingresscontroller
repo: https://charts.stderr.at
valuesFile: values.yaml

This simply defines the use of the chart ingresscontroller from the Helm repository with the local values file.

Unlike Option #2 you do not need to take care about the settings for sub-charts and which settings are passed to which chart. Like Option #2 you can also define additional files that shall be rendered using out-of-the-box Kustomize possibilities.

Advantages

Easy to use
Allows defining additional, local files that are not provided by the sub-charts
No need to take care of sub-charts and correctly pass the settings to a sub-chart

Disadvantages

combines two different tools, which might become confusing
requires specific option to be enabled --enable-helm

Conclusion

As this short article demonstrates, there are multiple ways to work with multiple sources and therefore to separate the values file from the actual Helm Chart. There might be even more options, but these are the ones I was seeing at customers.

Which one do I use? When you look at my repository you see that I mainly use Option #2. Actually, I completely moved from Option #3 to Option #2 a few months ago, because this proved to be clearer for customers, especially when they are new to Kustomize and Helm. That way, only one tool is used and must be managed. Option #3 proved to be more complex in such a case.

What about Option #1? While I am using it to showcase this feature it is still in a TechPreview phase. However, I do not think that it will completely replace the other options, because it is more complex to configure.

However, in the end, it is all about personal preferences. Use the tool that you feel most comfortable with :).

[Ep.8] Installing OpenShift Logging

Fri, 24 May 2024 00:00:00 +0000

OpenShift Logging is one of the more complex things to install and configure on an OpenShift cluster. Not because the service or Operators are so complex to understand, but because of the dependencies logging has. Besides the logging operator itself, the Loki operator is required, the Loki operator requires access to an object storage, that might be configured or is already available.

In this article, I would like to demonstrate the configuration of the full stack using an object storage from OpenShift Data Foundation. This means:

Installing the logging operator into the namespace openshift-logging
Installing the Loki operator into the namespace openshift-operators-redhat
Creating a new BackingStore and BucketClass
Generating the Secret for Loki to authenticate against the object storage
Configuring the LokiStack resource
Configuring the ClusterLogging resource

All steps will be done automatically. In case you have S3 storage available, or you are not using OpenShift Data Foundation, the setup will be a bit different. For example, you do not need to create a BackingStore or the Loki authentication Secret.

Prerequisites

OpenShift 4
Argo CD (OpenShift GitOps) deployed
OpenShift Data Foundation (ODF) deployed and ready to provide object storage.
Enough available compute resources to deploy LokiStack. Verify the official OpenShift Logging documentation to see which option might need which resources.

For ODF it would be enough to deploy object storage only, instead of the full storage stack based on Ceph. In this case, the so-called MultiCloudObjectGateway option is used, which creates (virtualizes) object storage on top of an existing StorageClass

If ODF object storage based on Noobaa should be used, then it makes sense to think about the data retention process, which will take care of removing old data from the storage. It is recommended to configure this directly on the object storage, because this is much more compute-friendly, then letting OpenShift Logging take care of that. The configuration depends on the object storage vendor. In the case of Noobaa, I have prepared a separate article: Noobaa Bucket Data Retention Lifecycle

Introduction

The main resources of OpenShift Logging are the three custom resources: ClusterLogging, ClusterLogForwarder and LokiStack. The first two are provided by the OpenShift Logging Operator, the last one is provided by the Loki Operator. ClusterLogForwarder is an optional configuration. It allows us to forward logs to external destinations, such as Splunk, or to forward the OpenShift audit logs to Loki. (They are not stored by default). The LokiStack resource requires an available object storage to be able to start its workloads.

In my case, I would like to configure everything automatically. This means, that I also want to configure the object or S3 storage and create the required authentication secret for Loki without manual intervention. This can be easily done using ODF.

The Configure App-of-Apps installed an Argo CD Application called in-cluster-setup-openshift-logging:

Figure 1. Argo CD Application: setup-openshift-logging

This Argo CD Application uses the following path to find the Helm Chart: setup-openshift-logging

This Helm chart is a wrapper chart that uses sub-charts as dependencies to install and configure the operator as well as to do some OpenShift Jobs on top, for example, creating the required Secret for LokiStack.

The deployment workflow will go through the sub-charts and look like the following:

Figure 2. Deployment Workflow

While this looks quite huge and complex, the idea of the sub-charts is quite simple: Do a small specific task, that can be reused by other charts. For example, the NetworkObservability Operator also required an object storage and Loki. I can easily reuse the sub-charts without repeating the logic behind them.

Installing OpenShift Logging Stack

Analyzing Chart.yaml

Let’s examine the Chart.yaml file to see which dependencies are used:

The file looks like the following. The Chart has a lot of dependencies on sub-charts, that have been created to make specific, small and defined operations re-useable for multiple Charts. A total number of 6 sub-charts are used:

dependencies:
- name: helper-operator (1)
version: ~1.0.18
repository: https://charts.stderr.at/
- name: helper-status-checker (2)
version: ~4.0.0
repository: https://charts.stderr.at/
condition: helper-status-checker.enabled
- name: openshift-logging (3)
version: ~2.0.0
repository: https://charts.stderr.at/
- name: helper-loki-bucket-secret (4)
version: ~1.0.0
repository: https://charts.stderr.at/
condition: helper-loki-bucket-secret.enabled
- name: helper-objectstore (5)
version: ~1.0.0
repository: https://charts.stderr.at/
condition: helper-objectstore.enabled
- name: helper-lokistack (6)
version: ~1.0.0
repository: https://charts.stderr.at/
condition: helper-lokistack.enabled

1	Dependency: Helper Operator
2	Dependency: Helper Status Checker
3	Dependency: OpenShift Logging
4	Dependency: Helper Loki Bucket Secret
5	Dependency: Helper Objectstore
6	Dependency: Helper Lokistack

Verify the READMEs of the different Charts for detailed information on how to configure them.

Configuration of the Chart

To configure OpenShift Logging the values file of the wrapper Chart must be prepared accordingly.

The important thing here is, that any value that should be bypassed to a sub-chart is defined under the name of the sub-chart. For example, everything under helper-operator: will be sent to the helper-operator Chart and is used there for its configuration.

Let’s walk through the configuration for each sub-chart in the order they are required:

Installing the Operator

The first thing to do is to deploy the Operators themselves. For OpenShift Logging two Operators are required:

OpenShift Logging
Loki

Loki might be installed already due to a different dependency. Maybe you have deployed the Network Observability Operator previously. In that case, OpenShift Logging is required only.

The Helm Chart helper-operator is responsible for deploying the Operators. In the following example, I will deploy both Operators (Logging and Loki) and enable the console plugin for the OpenShift Logging operator:

The console plugin will only work when the whole stack, this means when Logging itself, has been rolled out.

helper-operator:
console_plugins: (1)
enabled: true
plugins: (2)
- logging-view-plugin
operators:
cluster-logging-operator: (3)
enabled: true (4)
syncwave: '0' (5)
namespace: (6)
name: openshift-logging
create: true
subscription: (7)
channel: stable
source: redhat-operators
approval: Automatic
operatorName: cluster-logging
sourceNamespace: openshift-marketplace
operatorgroup: (8)
create: true
notownnamespace: false
loki-operator: (9)
enabled: true
namespace: (10)
name: openshift-operators-redhat
create: true
subscription: (11)
channel: stable-5.8
approval: Automatic
operatorName: loki-operator
source: redhat-operators
sourceNamespace: openshift-marketplace
operatorgroup: (12)
create: true
notownnamespace: true

1	Activate Console Plugin. This will trigger a Kubernetes Job, that will modify the current list of console plugins and add the new plugin to it.
2	List of plugins that should be added by the Job. The name of that plugin must be known. In the case of OpenShift Logging it is called logging-view-plugin
3	Key of the first operator: cluster-logging-operator. Everything below here will define the settings for the Logging Operator.
4	Is this Operator enabled yes/no.
5	Syncwave for the Operator deployment. (Subscription and OperatorGroup etc.) This should be early enough for other tasks.
6	The Namespace where the Operator shall be deployed and if this namespace shall be created.
7	Configuration of the Subscription resource. This defines the channel (version) that shall be used and whether the approval of the installPlan shall happen automatically or not.
8	Configuration of the OperatorGroup. Typically, you will need one when you create a new Namespace. Notownnamespace defines whether or not the targetNamespace is configured for this Operator or if the Operator is available in any Namespace.
9	Key of the second Operator: loki-operator. Everything below here will define the settings for the Logging Operator.
10	The Namespace where the Operator shall be deployed, must be openshift-operators-redhat and if this namespace shall be created.
11	Configuration of the Subscription resource. This defines the channel (version) that shall be used and whether the approval of the installPlan shall happen automatically or not.
12	Configuration of the OperatorGroup

The approval setting can either be Automatic or Manual. If the Operator requires approval to be installed, then this must either be done manually (via WebUI or CLI) or using the helper-status-checker chart which automatically can approve existing installPlans (explained in the next section). This is helpful, to automatically deploy the first version of the Operator without the need for manual intervention.

Verify the README at Helper Operator to find additional possible configurations. Also, verify the separate article Operator Installation with Argo CD to understand why I am verifying the status of the Operator installation.

Verifying the Operator Deployment

An Operator deployment can take some time and before you continue to configure the operator’s CRDs you must be sure that the installation finished successfully. Otherwise, the synchronization in Argo CD will fail because the CRD is not ready.

There are mainly two tactics to really verify the status of the Operator:

Simply retry a failed sync in Argo CD. This can be done automatically x-times.
Verify if the Operator installation succeeded by starting a Kubernetes Job that monitors the status.

(Custom) Health checks in Argo CD proved to be not 100% accurate because sometimes the Operator says it is "Ready" but the CRD still cannot be configured for some seconds. Looking at you Compliance Operator ….

I chose the second option, simply because I could also add a second Job that approved pending installPlans in case the deployment was set to manual approval.

The Helm Chart helper-status-checker has two main purposes:

Start a Kubernetes Job to verify the status of one or multiple Operator installation(s)
Optional: start a Kubernetes Job to approve the installPlan(s)

An example configuration, that verifies two Operators, looks like the following:

helper-status-checker:
enabled: true (1)
approver: false (2)
# List of checks that shall be performed.
checks:
- operatorName: cluster-logging (3)
# -- OPTIONAL: Name of subscription that shall be approved. In some cases the name of the Subscription is different to the name of the operator.
# @default --operatorName
subscriptionName: cluster-logging-operator (4)
namespace: (5)
name: openshift-logging
serviceAccount: (6)
name: "status-checker-logging"
- operatorName: loki-operator (7)
namespace:
name: openshift-operators-redhat
serviceAccount:
name: "status-checker-loki"

1	Enable the status checker.
2	Enable the installPlan approver. Only required if the approval strategy for an Operator is set to Manual.
3	Verify the status of the first Operator cluster-logging
4	Sometimes the name of the Subscription differs from the Operator name. Logging is such a case. To be able to find which Subscription should be verified, the subscriptionName must be defined here.
5	Namespace for OpenShift Logging
6	Name of the ServiceAccount that will be created to verify the status of the logging operator.
7	Settings for the 2nd operator: Loki. This one is running in a different Namespace and must be verified there.

Verify the README at Helper Operator Status Checker to find additional possible configurations.

At this stage, the Operators have been deployed and they have been verified if the deployment was finished successfully.

Now the real complex part can start…

Creating a new BackingStore for OpenShift Data Foundation

If you want to use a different storage solution or you have a bucket already, you can skip this section and simply create the LokiStack Secret manually.

In the case that ODF is used and a BackingStore together with a BucketClass shall be created another sub-chart called Helper ObjectStore can be used.

It will help you to create a:

BackingStore
BucketClass
StorageClass
BucketClaim

This fully automates the creation of the bucket and the required Class when using ODF. As a prerequisite, OpenShift Data Foundation (ODF) must be configured and available of course.

This is completely optional. If you want to use a different storage solution and have the buckets ready, you can simply create the Secret that Loki requires to authenticate at the storage. In this case, you can ignore this and the next section.

The following example will create a BackingStore with the size of 700Gi for our OpenShift Logging. A bucket named logging-bucket is created and can be used to store the logs.

helper-objectstore:
enabled: true
syncwave: 1 (1)
backingstore_name: logging-backingstore (2)
backingstore_size: 700Gi (3)
limits_cpu: 500m (4)
limits_memory: 2Gi
pvPool: (5)
numOfVolumes: 1
type: pv-pool
baseStorageClass: gp3-csi (6)
storageclass_name: logging-bucket-storage-class (7)
bucket: (8)
enabled: true
name: logging-bucket
namespace: openshift-logging
syncwave: 2
storageclass: logging-bucket-storage-class

1	Syncwave to create the BackingStore.
2	Name of the Backingstore.
3	Size of the BackingStore. 700Gi is good enough for testing Logging. Keep in mind that data retention should be configured separately for Noobaa.
4	Limit for CPU and Memory for the Noobaa (BackingStore) pod. They might need to be adjusted since the original ones are quite small for bigger buckets.
5	Pool of Persistent Volumes. Currently pv-pool is supported by the chart only.
6	The basic storage class that shall be used to virtualize ODF object storage on.
7	The name of the StorageClass that will be created and used by the BackingStore.
8	The configuration of the Bucket and its namespace and storageClass (defined at <7>)

Eventually, the BackingClass and the BucketClaim are created and ready.

Figure 3. Ready BackingStore and bound BucketClaim

Custom Argo CD Health Check for BackingStore

The creation of the BackingStore is a process that will take several minutes. Storage must be prepared, and several services must be started. To let Argo CD wait until the BackingStore is fully operational, instead of blindly continuing with the deployment of Loki and Logging, a custom Health Check in Argo CD might help.

The following health check should be placed into the Argo CD resource. Be aware, that there might be others already defined.

The status of the BackingStore resource inside Argo CD will continue progressing until the status of the resource becomes Ready.

Due to different syncwaves, Argo CD will wait for the Ready-status before it continues deploying Loki and Logging.

 resourceHealthChecks:
- check: |
hs = {}
if obj.status ~= nil then
if obj.status.phase ~= nil then
if obj.status.phase == "Ready" then
hs.status = "Healthy"
hs.message = obj.status.phase
return hs
end
end
end
hs.status = "Progressing"
hs.message = "Waiting for BackinbgStore to complete"
return hs
group: noobaa.io
kind: BackingStore

Generating Secret for LokiStack

If you want to use a different storage solution or you have a bucket already, you can skip this section and simply create the LokiStack Secret manually.

Creating the BackingStore and the BucketClaim will generate a Secret and a ConfigMap inside the target namespace. These hold the information about the connection to the object storage. Both resources are named as the bucket. The Secret contains the keys: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY while the ConfigMap stores the information about the URL, region etc.

While this is all we need to connect to the object store, Loki itself unfortunately requires a different Secret with a specific format. Before Loki can be configured, this Secret must be created, containing the keys: access_key_id, access_key_secret, bucketnames, endpoint and region (could be empty)

To automate the process another Helm Chart Helper Loki Bucket Secret has been created (we have too few charts) that has the only task to wait until the object store has been created, read the ConfigMap and the Secret and create the required Secret for Loki for us. Easy …

helper-loki-bucket-secret:
enabled: true
syncwave: 3
namespace: openshift-logging (1)
secretname: logging-loki-s3 (2)
bucket:
name: logging-bucket (3)

1	Namespace we are working in
2	The name of the Secret that shall be created
3	The name of the bucket that was created in the previous step to find the source information.

A Kubernetes Job is created, that will mount the created Secret and ConfigMap, read their values and create the Secret we need. It will simply execute the following command:

oc create secret generic {{ .secretname }} --from-literal access_key_id=${bucket_user} \
--from-literal access_key_secret=${bucket_secret} \
--from-literal bucketnames=${bucket_name} \
--from-literal endpoint=https://${bucket_host} \
--from-literal region=${bucket_region} \

This is completely optional. If you want to use a different storage solution and have the buckets ready, you can simply create the Secret (Sealed or inside a Vault) and put it into the wrapper chart. In this case, you can ignore this section.

Configuring the LokiStack

Up until now, all we did was the deployment of the Operators, verifying if they were ready, creating the object storage and the Secret that will be required by Loki. At this point, we can configure Loki by creating the resource LokiStack. This will start a lot of Pods (depending on your selected size). Loki itself then takes care to push the logs into the object store and to query them etc.

Believe it or not, but there is another Helm Chart called Helper LokiStack this will configure the service as we need. The configuration can become very big and the following example shows the main settings. Please consult the README of the Chart Helper LokiStack or the values file from our wrapper chart setup-openshift-logging. Especially, the pod placement using tolerations might be interesting, as it must be set per component individually.

helper-lokistack:
enabled: true (1)
name: logging-loki
namespace: openshift-logging
syncwave: 3
# -- This is for log streams only, not the retention of the object store. Data retention must be configured on the bucket.
global_retention_days: 4
storage: (2)
# -- Size defines one of the supported Loki deployment scale out sizes.
# Can be either:
# - 1x.demo
# - 1x.extra-small (Default)
# - 1x.small
# - 1x.medium
# @default -- 1x.extra-small
size: 1x.extra-small
# Secret for object storage authentication. Name of a secret in the same namespace as the LokiStack custom resource.
secret: (3)
name: logging-loki-s3
# -- Storage class name defines the storage class for ingester/querier PVCs.
# @default -- gp3-csi
storageclassname: gp3-csi (4)
# -- Mode defines the mode in which lokistack-gateway component will be configured.
# Can be either: static (default), dynamic, openshift-logging, openshift-network
# @default -- static
mode: openshift-logging (5)
# -- Control pod placement for LokiStack components. You can define a list of tolerations for the following components:
# compactor, distributer, gateway, indexGateway, ingester, querier, queryFrontend, ruler
podPlacements: {}

1	Basic settings, like Namespace, name of the resource and syncwave.
2	Size of the LokiStack. Depending on the selected size more or less compute resources will be required. 1x.demo is for testing only and is not supported for production workload.
3	Name of the Secret that was created in the previous step (or manually)
4	StorageClass that is required for additional workload. This is NOT the object storage.
5	Mode for the LokiStack Gateway to store the data. Possible values are static, dynamic, openshift-logging and openshift-network.

Custom Argo CD Health Check for LokiStack

As for the BackingStore resource, the LokiStack resource can take a couple of minutes before it is ready. Moreover, it can easily break when there are not enough computing resources available in the cluster. Therefore, I suggest creating another custom health check for Argo CD, to let it wait until the resource is ready. Only when it is ready, Argo CD will continue with the synchronization. Add the following to the resourceHealthChecks in your Argo CD resource.

 - check: |
hs = {}
if obj.status ~= nil and obj.status.conditions ~= nil then
for i, condition in ipairs(obj.status.conditions) do
if condition.type == "Degraded" and condition.reason == "MissingObjectStorageSecret" then (1)
hs.status = "Degraded"
hs.message = "Missing Bucket Secret"
end
if condition.type == "Pending" and condition.reason == "PendingComponents" and condition.status == "True" then (2)
hs.status = "Progressing"
hs.message = "Some LokiStack components pending on dependencies"
end
if condition.type == "Ready" and condition.reason == "ReadyComponents" then (3)
hs.status = "Healthy"
hs.message = "All components are ready"
end
end
return hs
end
hs.status = "Progressing" (4)
hs.message = "Waiting for LokiStack to deploy."
return hs
group: loki.grafana.com
kind: LokiStack

1	In LokiStack resources, if the fields 'status.conditions.condition.type' is "Degraded" and 'status.conditions.condition.reason' is MissingObjectStoreSecret then set the synchronization in Argo CD to Degraded.
2	In LokiStack resources, if the fields 'status.conditions.condition.type' is "Pending" and 'status.conditions.condition.reason' is PendingComponents and 'status.conditions.condition.status' is True then set the synchronization in Argo CD to Progressing.
3	In LokiStack resources, if the fields 'status.conditions.condition.type' is "Ready" and 'status.conditions.condition.reason' is ReadyComponents then set the synchronization in Argo CD to Healthy.
4	Per default set the status to Progressing.

Configuring ClusterLogging

Finally, the time … or should I say syncwave … has come to actually deploy the Logging components. The Operators are deployed, the object storage has been created and LokiStack is running.

The following settings will start the deployment of the ClusterLogging resource. As usual, please read the README of the Chart OpenShift Logging to find additional settings, such as tolerations etc.

openshift-logging:
loggingConfig:
enabled: true
syncwave: '4' (1)
logStore: (2)
type: lokistack
lokistack: logging-loki
visualization: (3)
type: ocp-console
collection: (4)
type: vector

1	The next syncwave, should be after LokiStack deployment.
2	Define the logStore (LokiStack) and its type (Loki or Elasticsearch). Please note that Elasticsearch as storage is deprecated and will be removed in the future. In my chart, I already removed the support for Elasticsearch
3	Type of virtualisation: should be ocp-console since Kibana and Elasticsearch are deprecated.
4	Type of collection: should be vector since Fluentd and Elasticsearch are deprecated.

This will deploy the ClusterLogging resource and OpenShift Logging is finally deployed. In the WebUI of OpenShift, you should now see at Observe > Logs the log files for the cluster.

Figure 4. OpenShift Logging

For individual Pods, a new tab called Aggregated Logs is available too:

Figure 5. Aggregated Logs tab

Custom Argo CD Health Check for ClusterLogging

One last thing to mention is the 3rd health check for Argo CD I usually configure that provides a proper response in the UI when the Logging stack is in a healthy state. The following will verify if the status is "Ready":

 - check: |
hs = {}
hs.status = "Progressing"
hs.message = "Progressing ClusterLogging"
if obj.status ~= nil and obj.status.conditions ~= nil then
for i, condition in ipairs(obj.status.conditions) do
if condition.type == "Ready" then
hs.status = "Healthy"
hs.message = "ClusterLogging is ready"
end
end
return hs
end
return hs
group: logging.openshift.io
kind: ClusterLogging

Tips and Tricks

Anchors in yaml files: Several parameters in the values file will repeat themselves. For example, the name of the LokiStack resource. Typically, I define this as an anchor on the top of the yaml files and then reference it inside the file. This way I see these anchors at the top and can easily change them there:

For example:

lokistack: &lokistackname logging-loki
[...]
helper-lokistack:
[...]
name: *lokistackname
openshift-logging:
loggingConfig:
[...]
logStore:
lokistack: *lokistackname

Object Storage Data Retention: The object storage is configured with a size of 700Gi, but without any lifecycle management. For object storage, the lifecycle (or data retention) is done on the bucket itself, not by the service. Please read the article Noobaa Bucket Data Retention Lifecycle to find out how to configure the data retention.

Conclusion

OpenShift Logging with all its dependencies, especially when you also want to use OpenShift Data Foundation and automate the bucket creation, is for sure one of the most complex Argo CD Applications I have created. I wanted to create one Application that completely deploys Logging for me, without manual interference. It will become much easier when you do not need to create the ODF bucket and the Secret for Loki. However, in such a case you define the Bucket somewhere else and must create the Secret manually (and put it into the wrapper Helm Chart for example). So probably the effort just shifts to somewhere else.

I hope this article was somehow understandable. I am always happy for Feedback, GitHub issues or Pull Requests.

One last thing, OpenShift Logging also supports the forwarding of logs. This is currently not supported by the Helm Chart per se. I would suggest creating such a resource and storing it in the wrapper Chart. Just be sure that the syncwave is after the ClusterLogging deployment and it will install the resource accordingly.

[Ep.7] Configure Buckets in MinIO

Fri, 17 May 2024 00:00:00 +0000

MinIO is a simple, S3-compatible object storage, built for high-performance and large-scale environments. It can be installed as an Operator to Openshift. In addition, to a command line tool, it provides a WebUI where all settings can be done, especially creating and configuring new buckets. Currently, this is not possible in a declarative GitOps-friendly way. Therefore, I created the Helm chart minio configurator, that will start a Kubernetes Job, which will take care of the configuration.

Honestly, when I say I have created it, the truth is, that it is based on an existing MinIO Chart by Bitnami, that does much more than just set up a bucket. I took out the bucket configuration part, streamlined it a bit and added some new features, which I required.

This article shall explain how to achieve this.

Prerequisites

Argo CD (OpenShift GitOps) deployed
MinIO including a deployed tenant that is waiting for buckets

Introduction

After MinIO and the Tenant have been deployed, we can configure and update a bucket, users, policies and more. Since I do not want to do this manually, the Helm Chart that will be described here creates a Kubernetes Job that leverages the mc command line tool to execute certain tasks automatically. The chart will take care of:

creating a lifecycle policy
creating an access policy
creating a new user/group. User credentials might be added directly to the values file or, better, are imported as a secret
attaching policies to a user/group
creating a bucket
set a quota for a bucket
set tags for a bucket
enable versioning for a bucket
enable object locking for a bucket (be aware that this can only be enabled during the bucket creation)
enable bucket replication to a target cluster/bucket
execute possible extra commands that are configured in the values file

To perform all these tasks Bitnami released a container image: docker.io/bitnami/minio:2024.5.1-debian-12-r0 They are updating this image regularly.

Actually, the image can be used to deploy the minio server. At this moment, we are interested in the command line tool only. Bitnami also managing a minio-client image, that can be tested and used. However, I left the original image, which is working very well.

The Values

All settings are explained in more detail at: https://github.com/tjungbauer/helm-charts/tree/main/charts/minio-configurator

The Job and everything that is required, are executed inside the Tenant namespace. In the following examples, this will be minio-tenant-namespace

Basic Settings

The basic settings are the following. They will define the namespace of the Tenant, the name of the ServiceAccount, the URL of the tenant, Argo CD Hook settings and the image that shall be used for the deployment.

name: minio-provisioner (1)
namespace: minio-tenant-namespace (2)
synwave: 5 (3)
argoproj: (4)
hook: Sync
hook_delete_policy: HookSucceeded
image:
url: docker.io/bitnami/minio:2024.5.1-debian-12-r0 (5)
# Information of the Minio Cluster
miniocluster: (6)
url: minio-tenant-api-url
port: 443
skip_tls_verification: true (7)
# Specifies whether a ServiceAccount should be created
serviceAccount: (8)
create: true
name: "minio-provisioner"

1	Name of the Kubernetes provisioner Job resource.
2	Namespace of the MinIO Tenant.
3	Syncwave of the provisioner Job.
4	Possible Argo CD hook configuration.
5	The container image the provisioner Job will use.
6	The URL of the minio console. This will be used to set the "alias" for the mc command
7	Skip verification of TLS for the mc command. This will disable the TLS check for any mc command the Job will execute.
8	Information about the ServiceAccount

Authentication Settings

To be able to authenticate against MinIO credentials must be provided. This happens, typically, in the form of a Secret:

auth:
useCredentialsFiles: true (1)
secretName: minio-provisioner (2)

1	Shall a secret mounted as a file be used (preferred)
2	Name of the Secret

The Secret itself requires specific keys and should look like the following:

kind: Secret
apiVersion: v1
metadata:
name: minio-provisioner (1)
namespace: minio-tenant-namespace (2)
data:
root-password: <base64 string> (3)
root-user: <base64 string> (4)
type: Opaque

1	Name of the Secret as mentioned in the minio-configurator values files
2	Name of the Namespace as mentioned in the minio-configurator values files
3	Password to access MinIO
4	User to access MinIO

The Secret must exist upfront and is not created by the Helm Chart. Either pick it from a Vault or create a Sealed Secret to be able to store it in Git.

The credentials are called root-. Any user that has permission to configure buckets is sufficient here. Still, the keys must be named that way.

Creating MinIO Policies

MinIO uses Policy-Based Access Control to define which actions can be performed on certain resources by an authenticated user. A policy can be created by the command mc admin policy. Our Kubernetes Job will take the configuration from the values file and mount the information as a JSON file, that will be imported into MinIO.

The following specification shows the example for OpenShift Logging:

provisioning:
enabled: true (1)
policies:
- name: openshift-logging-access-policy (2)
statements:
- resources: (3)
- "arn:aws:s3:::openshift-logging"
- "arn:aws:s3:::openshift-logging/*"
effect: "Allow" (4)
actions:
- "s3:*" (5)

1	In general, enable the provisioning or not
2	Name of the policy. Multiple can be defined and assigned to a user or group.
3	Define the resources the policy should manage access to.
4	Define the effect: Allow or Deny (default)
5	The actions that are allowed. Here: any s3: action

Multiple policies can be defined in the values file, and it is very important to exactly define the resources, the effect and the actions. The above configuration will allow the user that has the policy assigned:

All s3 actions to the bucket openshift-logging and everything inside this bucket (thus two resources)

All actions are defined at: MinIO Access Management.

Another example would be the following:

 policies:
- name: custom-bucket-specific-policy
statements:
- resources:
- "arn:aws:s3:::my-bucket"
actions:
- "s3:GetBucketLocation"
- "s3:ListBucket"
- "s3:ListBucketMultipartUploads"
- resources:
- "arn:aws:s3:::my-bucket/*"
effect: "Allow"
actions:
- "s3:AbortMultipartUpload"
- "s3:DeleteObject"
- "s3:GetObject"
- "s3:ListMultipartUploadParts"
- "s3:PutObject"

This policy defines the actions in a fine granular way:

To the bucket my-bucket we have three allowed actions (GetBucketLocation, ListBucket and ListBucketMultipartUploads)
To everything inside the bucket (/*) we can also Delete, Get, Put objects etc.

Creating a User

The policy that has been created must be assigned to a user (or a group) to be effective. Such a user requires a username, a password and a list of policies that shall be assigned.

The required information can be added directly to the values file like this:

This is NOT the recommended way!

 # users:
# - username: test-username (1)
# password: test-password (2)
# disabled: false (3)
# policies: (4)
# - readwrite
# - consoleAdmin
# - diagnostics
# # When set to true, it will replace all policies with the specified.
# # When false, the policies will be added to the existing.
# setPolicies: false
# @default -- []

1	Username
2	clear text password
3	Shall the user be created or not
4	List of policies that shall be assigned

As mentioned above: Defining a list of users directly in the values file is not recommended as it would mean that the passwords are stored in clear text.

Instead, a list of Secrets can be defined:

 usersExistingSecrets:
- minio-users

The defined Secrets require a specific structure and can be encrypted and stored in Git or a Vault.

The data structure is the following:

apiVersion: v1
kind: Secret
metadata:
name: minio-users (1)
type: Opaque
stringData:
username1: | (2)
username=username (3)
password=password (4)
disabled=false (5)
policies=openshift-logging-access-policy,readwrite,consoleAdmin,diagnostics (6)
setPolicies=false (7)

1	Name of the Secret as referenced in the values file.
2	List of users, distinguished by the key "username1", "username2", etc.
3	Username
4	Password
5	Enabled or disabled
6	List of policies to assign to the user
7	Replace or add the policies to an (existing) user.

Built-In Policies

MinIO provides several Built-In Policies that can be attached to a user or group.

The following policies will always exist: (Please verify the official documentation for further information)

consoleAdmin

Grants complete access to all S3 and administrative API operations against all resources on the MinIO deployment.

s3:*
admin:*

readonly

Grants read-only permissions on any object on the MinIO deployment. The GET action must apply to a specific object without requiring any listing.

s3:GetBucketLocation
s3:GetObject

readwrite

Grants read and write permissions for all buckets and objects on the MinIO server.

s3:*

diagnostics

Grants permission to perform diagnostic actions on the MinIO deployment.

admin:ServerTrace
admin:Profiling
admin:ConsoleLog
admin:ServerInfo
admin:TopLocksInfo
admin:OBDInfo
admin:BandwidthMonitor
admin:Prometheus

writeonly

Grants write-only permissions to any namespace (bucket and path to object) the MinIO deployment.

s3:PutObject

Provisioning Groups

Users can be combined into groups and instead of assigning policies to individual users, we can assign them to a whole group. The idea is the same as for users, except, that we define a list of members for that group:

 groups
- name: test-group (1)
disabled: false (2)
members: (3)
- username
policies: (4)
- readwrite
# When set to true, it will replace all policies with the specified.
# When false, the policies will be added to the existing.
setPolicies: false (5)

1	Name of the group.
2	Enabled or disabled.
3	List of users that are members of this group.
4	List of policies that are assigned to this group.
5	Replace or add the policies to an (existing) user.

Configure the Bucket

Finally, we can configure the bucket itself. A bucket will have a specific configuration, a lifecycle a quota etc. A list of buckets with different configurations can be defined in the values files.

The only mandatory information is the name of the bucket. It is not required to configure a lifecycle or quota etc.

Let us analyse the following example, which tries to cover all possible settings:

 buckets:
- name: mybucket (1)
region: my-region (2)
versioning: Versioned (3)
withLock: false (4)
bucketReplication: (5)
enabled: true
targetClusterUrl: replication-target-cluster
targetClusterPort: 443
targetBucket: replication-target-bucket
replicationSettings: (6)
- existing-objects
credSecretName: replication-credentials (7)
lifecycle:
- id: name-of-lifecycle (8)
prefix: test-prefix (9)
disabled: false
expiry: (10)
days: 30 # or date
nonconcurrentDays: 10
- id: name-of-second-lifecycle
disabled: false
expiry:
deleteMarker: true
nonconcurrentDays: 10
quota: (11)
type: set
size: 1024Gib
tags: (12)
key1: value1

1	Name of the bucket.
2	Region of the bucket
3	Enable versioning (https://docs.min.io/docs/minio-client-complete-guide.html#ilm). Allowed options are: Versioned, Suspended or Unchanged.
4	Enable object Locking
5	Configure bucket replication to a target cluster and a target bucket
6	Define the settings for the bucket replication can be: delete, delete-marker or existing-objects: https://min.io/docs/minio/linux/administration/bucket-replication/enable-server-side-one-way-bucket-replication.html
7	Name of the Secret that stores the credentials for the replication
8	Define a list of lifecycle policies for the bucket: https://min.io/docs/minio/linux/administration/object-management/object-lifecycle-management.html
9	A prefix that can be defined
10	Define the expiration. This can be defined as days OR as a date, for example "2021-11-11T00:00:00Z"
11	Set a quota for the bucket: https://docs.min.io/docs/minio-admin-complete-guide.html#bucket
12	Define additional tags for the bucket https://docs.min.io/docs/minio-client-complete-guide.html#tag

Replication Secret

The definition above defines a bucket replication. To authenticate at the target cluster, we need to provide a username and a password. This is stored inside a secret:

apiVersion: v1
kind: Secret
metadata:
name: replication-user
type: Opaque
stringData:
username: username
password: password

This defines a whole bunch of settings. Except for the bucket name, none is mandatory.

Example OpenShift-Logging Bucket

The following is a more realistic example, for defining a bucket used for OpenShift Logging:

It defines the bucket name, with a lifecycle of 4 days and a quota of 1TB:

 buckets:
- name: openshift-logging
lifecycle:
- id: logging-retention
disabled: false
expiry:
days: 4
quota:
type: set
size: 1024GiB

Additional Settings

Finally, there are some additional settings, I would like to mention here. They are completely optional, but might be interesting:

Automatically clean up the provisioning job after it has finished:

 cleanupAfterFinished:
enabled: false
seconds: 600

Define resources for the provisioning job. For example:

resources:
requests:
cpu: 2
memory: 512Mi
limits:
cpu: 3
memory: 1024Mi

Typically, I leave this to resources: {}

Take care of the pod placement and define a nodeSelector and tolerations, for example:

 nodeSelector: {}
tolerations:
- effect: NoSchedule
key: infra
operator: Equal
value: reserved
- effect: NoExecute
key: infra
operator: Equal
value: reserved

Conclusion

With this Helm chart by Bitnami, with a little modification from my side, it is possible to create and update buckets, policies, users etc. There is no need, to perform any modification manually in the MinIO WebUI.

I am currently using this chart for several bucket configurations, with sometimes more and sometimes fewer settings in the values file. Keep in mind, that many settings, especially for the bucket itself, are completely optional and are not required to create a new bucket. (For example, lifecycle). Please check out the source of the Helm Chart and the values file to get further information: minio configurator.

If you have any feedback or miss something, feel free to create a pull request or an issue :)

[Ep.6] Setup & Configure Advanced Cluster Security

Sun, 28 Apr 2024 00:00:00 +0000

Today I want to demonstrate the deployment and configuration of Advanced Cluster Security (ACS) using a GitOps approach. The required operator shall be installed, verified if it is running and then ACS shall be initialized. This initialization contains the deployment of several components:

Central - as UI and as a main component of ACS
SecuredClusters - installs a Scanner, Controller pods etc.
Console link into OpenShift UI - to directly access the ACS Central UI
Job to create an initialization bundle to install the Secured Cluster
Job to configure authentication using OpenShift

Let’s start …

Prerequisites

Argo CD (OpenShift GitOps) deployed
App-Of-Apps deployed

Introduction

The main components of ACS are the two custom resources: Central and SecuredCluster. The recommended way to deploy ACS is to use the Operator (alternatives would be the command line or Helm Chart). When I first came across ACS I thought about how to automate the full deployment. I did not want to install the Operator, then the Central, then manually create a so-called init-bundle, then deploy the Secured Cluster, then find the route that has been used, then find the secret that stores the initial administrator credentials and then, finally, log into ACS and activate OpenShift authentication.

As you can see there are a lot of tasks to do before I can start a customer demo.

At the same time, I started to dig into GitOps and I thought this would be a good option to create my very first Helm Chart.

Long story short, I have now a Helm Chart (actually three, because I outsourced some of the features into sub-charts) that automatically does all these things above. Once I am synchronizing my Argo CD Application everything will happen automatically.

The Configure App-of-Apps installed an Argo CD Application called in-cluster-setup-acs:

Figure 1. Argo CD Application: setup-acs

This Argo CD Application used the following path to find the Helm Chart: setup-acs

This Helm chart is a wrapper chart that uses sub-charts as dependencies to install and configure the operator as well as to do some OpenShift Jobs on top, for example, creating a ConsoleLink or creating an init-bundle.

Please check out the article Setup Compliance Operator on why I am using a wrapper chart.

Installing Advanced Cluster Security

Analysing Chart.yaml

Let’s examine the Chart.yaml file to see which dependencies are used:

The file looks like the following. Three sub-charts are defined as required to deploy and configure the ACS. This is pretty much the same as it was for the Setup Compliance Operator.

apiVersion: v2
name: setup-acs
description: Deploys Advanced Cluster Security (ACS) on target cluster. If enabled Central will be deployed too.
version: 1.0.0
dependencies:
- name: rhacs-setup (1)
version: ~1.0.0 (2)
repository: https://charts.stderr.at/
- name: helper-operator (3)
version: ~1.0.23
repository: https://charts.stderr.at/
- name: helper-status-checker (4)
version: ~4.0.0
repository: https://charts.stderr.at/
condition: helper-status-checker.enabled (5)

1	Dependency: RHACS Setup
2	Version that will be used. The "~" means that the latest version of 1.0.X will be used.
3	Dependency: Helper Operator
4	Dependency: Helper Status Checker
5	Only use this dependency when "enabled" is set

Verify the READMEs of the different Charts for detailed information on how to configure them.

Configuration of the Chart

To configure Advanced Cluster Security the values file of the wrapper Chart must be prepared accordingly.

Check out the example values file I use to configure ACS and the README to find further information about the possible settings that can be done.

Let’s check the main example, to quickly start:

Installing and verifying the Operator

The first thing to do is to deploy the Operator and to verify if the Operator installation finished successfully.

The two Helm Charts helper-operator and helper-status-checker are responsible to do so.

They are configured as follows:

helper-operator:
operators:
rhacs-operator: (1)
enabled: true (2)
syncwave: '0' (3)
namespace:
name: rhacs-operator (4)
create: true
subscription:
channel: stable (5)
approval: Automatic
operatorName: rhacs-operator
source: redhat-operators
sourceNamespace: openshift-marketplace
operatorgroup: (6)
create: true
# rhacs does not support to monitor own namespace,
# therefore the spec in the OperatorGroup must be empty
notownnamespace: true
# Subchart helper-status-checker
# checks if ACS operator is ready
helper-status-checker:
enabled: true (7)
checks: (8)
- operatorName: rhacs-operator (9)
namespace:
name: rhacs-operator (10)
syncwave: 3
serviceAccount:
name: "status-checker-acs" (11)

1	Key that can be freely defined. Theoretically, you can deploy multiple operators at once.
2	Is this Operator enabled yes/no.
3	Syncwave for the Operator deployment. (Subscription and OperatorGroup etc.)
4	The Namespace where the Operator shall be deployed and if this namespace shall be created.
5	Configuration of the Subscription resource.
6	Configuration of the OperatorGroup
7	Enable status checker or not. Default: false
8	List of operators to check. Typically, only one is checked, but there could be more.
9	Name of the Operator to check (same as for helper-operator)
10	Namespace where the Operator has been installed (same as for helper-operator)
11	Name of the ServiceAccount that will be created to check the status.

Verify the READMEs at Helper Operator and Helper Operator Status Checker to find additional possible configurations.

Also verify the separate article Operator Installation with Argo CD to understand why I am verifying the status of the Operator installation.

Configuring Advanced Cluster Security

Besides the deployment of the Operator, the configuration of ACS is the most important part here. The ACS Operator provides two custom resources: Central and SecuredCluster. On the Central cluster both CRDs are required. On any other (spoke) cluster, the SecuredCluster resource is enough.

In the following example, I am going to configure both Central and SecuredCluster. Since the values file is quite huge I removed most of the additional comments, to keep this article short and readable. You can read the example values file or the README at Advanced Cluster Security Chart to find additional possible configurations. Especially, if you like to add tolerations or set resource limits.

#########################################
# Settings for Advanced Cluster Security
#########################################
rhacs-setup:
rhacs:
namespace: (1)
name: stackrox
syncwave: '0'
descr: 'Red Hat Advanced Cluster Security'
################
# CENTRAL of ACS
################
# Settings for the Central of ACS
central: (2)
enabled: true
syncwave: '3'
egress:
connectivityPolicy: Online
###############
# CENTRAL DB
###############
# Settings for Central DB, which is responsible for data persistence.
db: (3)
# -- Set Central DB resources.requests for a DEMO environment to save resources.
resources:
requests:
cpu: '1'
memory: '1Gi'
# -- If you want this component to only run on specific nodes, you can
# configure tolerations of tainted nodes.
tolerations: {}
# - effect: NoSchedule
# key: infra
# operator: Equal
# value: reserved
# - effect: NoSchedule
# key: infra
# operator: Equal
# value: reserved
###############
# SCANNER
###############
scanner: (4)
enabled: true
analyzer:
# The following settings are NOT suitable for a production environment
autoscaling:
status: "Disabled"
max: 1
min: 1
# When autoscaling is disabled, the number of replicas will always be
# configured to match this value.
replicas: 1
tolerations: {}
###############
# SCANNER DB
###############
db:
tolerations: {}
#################
# SECURED CLUSTER
#################
secured_cluster: (5)
enabled: true
syncwave: '4'
clustername: local-cluster
sensor:
tolerations: {}
admissioncontrol:
listenOn:
creates: true
events: true
updates: true
tolerations: {}
# -- Basic settings for ACS authentication
# This configuration is done by a Job, that will configure the OpenShift oauth for ACS.
basic_acs_settings: (6)
auth_provider: 'OpenShift'
auth_provider_type: 'openshift'
min_access_role: 'None'
syncwave: 5
####################################################
# Additional settings for Central and possible Jobs
####################################################
job_vars: (7)
max_attempts: 20
job_init_bundle: (8)
enabled: true
syncwave: '3'
consolelink: (9)
enabled: true
syncwave: '3'
location: ApplicationMenu
text: Advanced Cluster Security
section: Observability

1	Create the Namespace stackrox and install the ACS resources there.
2	Enable the Central during Syncwave 3 and set the connectivityPolicy to Online
3	The Central DB and its configuration. Here the resource requests are modified to allow a small installation on the DEMO environment. Also, tolerations might be set here, as well a PVCs etc.
4	Settings for the Scanner and its databases. Again, tolerations might be configurated here, but also, not shown in this example, resource limits and requests and other settings. Since I am configuring for a DEMO environment, I disabled the autoscaler and set the replica to 1.
5	The SecuredCluster is the 2nd CRD that is provided by ACS Operator. It is installed after the Central (thus a higher Syncwave). The most important setting here is the clustername. In our "local" example, the name is set to local-cluster.
6	Some basic settings, that will configure the OpenShift authentication and the minimum role for authenticated users (None)
7	Some default settings for Jobs that are started by this Helm chart.
8	The Job that initializes the creation of the init-bundle
9	The Job and its configuration to generate a direct link to ACS in the OpenShift UI.

With this ACS is about to be installed on the cluster. Let’s see what will happen during the synchronization.

Deploying Advanced Cluster Security (ACS)

Let’s hit the sync button inside OpenShift GitOps. This will start the whole process, walking through the syncwaves and the hooks that have been defined.

Figure 2. Syncing Argo CD

Since hooks are used, you must sync the whole Argo CD Application. As you can see inside Argo CD, the hooks are not shown, because they will only appear when their time has come (and will disappear afterward again). This means, that if you perform a selective sync, Argo CD does not know when it should start such a hook and they are never triggered.

The Operator installation is now started. At the moment the Operator has the status Installing. Currently, no CRDs (Central or SecuredCluster) are available yet. If we would just let Argo CD continue, it would try to create the Central configuration, based on a CRD which does not yet exist. Thus, the syncing process will fail and therefore the status-checker is going to verify if the installation was truly successful.

Figure 3. Operator is installed

The Status Checker is a simple Pod that is triggered by a Kubernetes Job. If waits until the status of the Operator is Succeeded. Until this is the case, Argo CD waits before it continues with the synchronization. (It waits until the hook ends the Job)

Figure 4. Status Checker

In the logs file of the Pod, we can see that the Operator is ready.

Figure 5. Status Checker Logs

And indeed, the status of the Operator is now Succeeded. Now it is time for Argo CD to continue the synchronization.

Figure 6. Operator Ready

The next step is to create the Central CRD. This will deploy the UI of ACS and the local Scanner. You can also see two other Jobs that have been created by the Helm Charts create_cluster_link and create_cluster_init_bundle. They will finish when the Central becomes ready.

Until the Central becomes ready, these two additional Jobs may show errors. Do not worry, OpenShift will reschedule them.

Figure 7. Installing Central

You can also see two other Jobs that have been created by the Helm Charts create_cluster_link and create_cluster_init_bundle. They will finish when the Central becomes ready.

Until the Central becomes ready, these two additional Jobs may show errors. Do not worry, OpenShift will reschedule them.

Figure 8. Init Job is waiting for Central

Once the Central has been deployed, the second CRD SecuredCluster will be added. This will trigger the installation of the Collectors and other components.

Figure 9. Installing SecuredCluster

Eventually, all Pods are running at the end. The additional Jobs are completed and ACS is ready to take care of the cluster security.

Figure 10. All Pods running

We can now use the console link that was created.

Figure 11. ACS ConsoleLink

If you disabled the creation of the ConsoleLink you would need to find the Route that ACS Operator created. Honestly, I do not know why the Operator does not create such ConsoleLink out of the box.

Since I am lazy I created another Job that automatically configures authentication via OpenShift for ACS. This way, we can simply use our OpenShift credentials to login.

Figure 12. ACS Login

And that’s it, we can now use ACS, which was deployed fully automatically.

Figure 13. Advanced Cluster Security

Conclusion

As you can see Advanced Cluster Security was completely installed. This included the Operator, the Central, the creation and installation of an init-bundle, the creation of a ConsoleLink, the configuration of the SecuredCluster CRD and the initial configuration of the auth provider inside ACS. You can now start using ACS or add additional clusters.

Speaking of additional clusters: The create-cluster-init-bundle created three certificates: collector-tls, sensor-tls and admission-control-tls. They are required so that the SecuredClusters can communicate with the Central. You could now create a separate init-bundle for each SecuredCluster, which is not really easy to automate, or you simply take these created secrets and put them into your GitOps and re-use them for any other SecuredCluster.

All these steps and configurations seem quite complicated but honestly, it is straightforward. I install any Operator using Helper Operator and in most cases. I also use Helper Operator Status Checker to find additional possible configurations. Both require simple configuration only, which you would need to know anyway when you create the Subscription object manually. Once done, you can repeat this for any other Operator.

The real magic happens when the Operator is configured at the same time because this is very individual to the Operator.

[Ep.5] Setup & Configure Compliance Operator

Thu, 25 Apr 2024 00:00:00 +0000

In the previous articles, we have discussed the Git repository folder structure and the configuration of the App-Of-Apps. Now it is time to deploy our first configuration. One of the first things I usually deploy is the Compliance Operator. This Operator is recommended for any cluster and can be deployed without any addition to the Subscription.

In this article, I will describe how it is installed and how the Helm Chart is configured.

Prerequisites

Argo CD (OpenShift GitOps) deployed
App-Of-Apps deployed

Introduction

As a reminder, at Git repository folder structure I described my preferred folder structure. I would like to deploy the Compliance Operator in the Management Cluster now. All my examples can be found at GitHub repository OpenShift Clusterconfig GitOps. The folder clusters/management-cluster/setup-compliance-operator is the one I am interested in.

Inside this folder, you will find another Helm Chart. The Helm Chart has no local templates, instead, it uses dependencies to call other (sub-) charts. However, the values.yaml is the main part to configure everything.

In case you want to have any local template, that you do NOT want to integrate into one of the sub-charts, you can easily do so, by storing them in the templates folder.

Why "empty" Helm Charts?

Actually, it would be possible to use the Helm Chart of the Chart repository directly, without creating a separate chart, that does nothing else than using dependency charts.

The reasons why I am using such an "empty" Chart are the following (in no particular order):

With that way it is possible to add templates (i.e. SealedSecrets) and modify the values-file without packaging and releasing a new Chart version every time you change a small thing.
The Multi-Source Option, which allows you to use a Helm Chart from repository A and a values file from repository B is still a TechPreview feature (Argo CD 2.10). I am using this for the App-of-Apps already, but I did not do this for all charts. This feature is on the list for Argo CD version 2.11 to become globally available.

As an alternative, it is also possible to mix Kustomize and Helm. That way you only need a kustomization.yaml file and reference to a Helm Chart. In the folder clusters/management-cluster/ingresscontroller I have such an example.

Installing Compliance Operator

Analysing Chart.yaml

As any Helm Chart a Chart.yaml file exists, that stores the basic information. The most important ones for now are the dependencies.

The file looks like the following. Three sub-charts are defined as required to deploy and configure the Compliance Operator.

apiVersion: v2
name: setup-compliance-operator
description: Deploy and configure the Compliance Operator
version: 1.0.1
dependencies:
- name: compliance-operator-full-stack (1)
version: ~1.0.0 (2)
repository: https://charts.stderr.at/
- name: helper-operator (3)
version: ~1.0.21
repository: https://charts.stderr.at/
- name: helper-status-checker (4)
version: ~4.0.0
repository: https://charts.stderr.at/
condition: helper-status-checker.enabled (5)

1	Dependency: Compliance Operator Full Stack
2	Version that will be used. The "~" means that the latest version of 1.0.X will be used.
3	Dependency: Helper Operator
4	Dependency: Helper Status Checker
5	Only use this dependency when "enabled" is set

Verify the READMEs of the different Charts for detailed information on how to configure them.

As you can see three other Helm Charts are used to actually deploy and configure the Operator.

Configuration of the Chart

To configure the Compliance Operator, the values files must be prepared accordingly.

The following is a full example of the values I typically use.

# Install Operator Compliance Operator
# Deploys Operator --> Subscription and Operatorgroup
helper-operator:
operators:
compliance-operator:
enabled: true
syncwave: '0'
namespace:
name: openshift-compliance
create: true
subscription:
channel: stable
approval: Automatic
operatorName: compliance-operator
source: redhat-operators
sourceNamespace: openshift-marketplace
operatorgroup:
create: true
notownnamespace: true
# Verify if the Operator has been successfully deployed
helper-status-checker:
enabled: true
checks:
- operatorName: compliance-operator
namespace:
name: openshift-compliance
serviceAccount:
name: "status-checker-compliance"
# Setting for the Compliance Operator
compliance-operator-full-stack:
compliance:
namespace:
name: openshift-compliance
syncwave: '0'
descr: 'Red Hat Compliance'
scansettingbinding:
enabled: true
syncwave: '3'
profiles:
- name: ocp4-cis-node
kind: Profile # Could be Profile or TailedProfile
- name: ocp4-cis
kind: Profile
scansetting: default

Let us walk through the settings in more detail.

Installing the Operator

The first thing to do is to deploy the Operator. Two resources are relevant to install an Operator:

Subscription
OperatorGroup

Both objects should be deployed at the very beginning of Argo CD synchronisation. This is done by setting the Syncwave to 0.

The main settings are the operatorName, the channel (which is the version of the operator) and the approval (which defines if the Operator is updated automatically or manually).

In addition, a Namespace object is deployed, because this Operator should run in its very own namespace.

This will start the Operator installation process.

helper-operator:
operators:
compliance-operator: (1)
enabled: true (2)
syncwave: '0' (3)
namespace:
name: openshift-compliance (4)
create: true
subscription: (5)
channel: stable # Version of the Operator
approval: Automatic # Automatic or Manual
operatorName: compliance-operator # Name of the Operator
source: redhat-operators
sourceNamespace: openshift-marketplace
operatorgroup: (6)
create: true
notownnamespace: true

1	Key that can be freely defined. Theoretically, you can deploy multiple operators at once.
2	Is this Operator enabled yes/no.
3	Syncwave for the Operator deployment. (Subscription and OperatorGroup etc.)
4	The Namespace where the Operator shall be deployed and if this namespace shall be created.
5	Configuration of the Subscription resource.
6	Configuration of the OperatorGroup

Verify the README at Helper Operator to find additional possible configurations.

Verify the Status of the Operator

After Argo CD creates the subscription and operatorgroup resources (and namespace), OpenShift will start the installation of the Operator. This installation will take a while but Argo CD does not see this. All it sees is that the Subscription resource is available and it tries to continue with the configuration of the Operator. Here it will fail because the CRDs are not available yet.

Therefore, I created a mechanism to verify if an Operator is ready or not.

Also verify the separate article Operator Installation with Argo CD that addresses the problem in more detail.

All it does is to start a small Job inside OpenShift and to verify the status of the Operator installation. If everything is fine, the Job will end successfully and Argo CD will continue with the next syncwave. Argo CD Hook and syncwaves are required here. The Job should be started after the Subscription/OperatorGroup resources have been created, which means any syncwave after "0".

The following annotations will be used by the Job:

 argocd.argoproj.io/hook: Sync (1)
argocd.argoproj.io/hook-delete-policy: HookSucceeded (2)
argocd.argoproj.io/sync-wave: {{ .syncwave | default 1 | quote }} (3)

1	Hooks are ways to run scripts before, during, and after a Sync operation.
2	Deletes the OpenShift Job again. The hook resource is deleted after the hook succeeded (e.g. Job/Workflow completed successfully).
3	Syncwave: can be configured. Must be after helper-operator (default 0) and before the Operator is configured further. Default value is 1.

The configuration for hepler_status_checker will look like the following:

# Verify if the Operator has been successfully deployed
helper-status-checker:
enabled: true (1)
checks: (2)
- operatorName: compliance-operator (3)
namespace:
name: openshift-compliance (4)
serviceAccount:
name: "status-checker-compliance" (5)

1	Enable status checker or not. Default: false
2	List of operators to check. Typically, only one is checked, but there could be more.
3	Name of the Operator to check (same as for helper-operator)
4	Namespace where the Operator has been installed (same as for helper-operator)
5	Name of the ServiceAccount that will be created to check the status.

Verify the README at Helper Operator Status Checker to find additional possible configurations.

Configuring Compliance Operator

Finally, the Operator has been deployed and has been verified. Now the time is right to configure the Operator with any configuration we would like. This means, using CRDs to do whatever the Operator offers.

This is reflected in the following part of the values file. All these settings are handed over to the sub-chart compliance-operator-full-stack.

Verify the README at Compliance Operator Chart to find additional possible configurations. Especially, if you like to do Tailored Profiles.

The compliance operator requires a so-called ScanSettingBinding that uses Profiles which are used to check the cluster compliance once a day. In this case, I am using CIS Benchmarks. There are two profiles:

ocp4-cis-node: will check the node operating system for missing but suggested configuration.
ocp4-cis: will check the OpenShift cluster for missing but suggested configuration.

# Setting for the Compliance Operator
compliance-operator-full-stack: (1)
compliance:
namespace:
name: openshift-compliance (2)
syncwave: '0'
descr: 'Red Hat Compliance'
scansettingbinding: (3)
enabled: true
syncwave: '3'
profiles: (4)
- name: ocp4-cis-node
kind: Profile # Could be Profile or TailedProfile
- name: ocp4-cis
kind: Profile
scansetting: default

1	Handing everything that comes below to the sub-chart compliance-operator-full-stack
2	Namespace where the configuration should be deployed. The Syncwave at this point could be omitted.
3	The configuration for the ScanSettingBinding. It is enabled (default = false) and has a Syncwave AFTER the helper-status-checker.
4	The list of profiles that shall be used. These must exist. The Compliance Operator offers several profiles. I usually use these two for full CIS compliance check.

Conclusion

With this configuration, the Compliance Operator will not only be installed but also configured with the same Argo CD Application. All you need to do is to synchronize Argo CD and let the magic happen. After a few minutes, everything should be in sync.

Figure 1. Sync Compliance Operator

Inside OpenShift the Operator is configured and starts doing its job:

Figure 2. Configured Compliance Operator

This concludes the deployment of the Compliance Operator. For further information about the Operator itself, please read the documentation or articles:

Also, be sure to check out the READMEs of the different Charts:

If you have any questions or problems, feel free to create a GitHub issue at any time.

[Ep.4] Configure App-of-Apps

Tue, 02 Apr 2024 00:00:00 +0000

In the article Install GitOps to the cluster OpenShift GitOps is deployed using a shell script. This should be the very first installation and the only deployment that is done manually on a cluster. This procedure automatically installs the so-called App-of-Apps named Argo CD Resources Manager which is responsible for all further Argo CD Applications and ApplicationSets. No other configuration should be done manually if possible.

This article will demonstrate how to configure the App-of-Apps in an easy and declarative way, using ApplicationSet mainly.

Prerequisites

At this stage, the OpenShift cluster with the openshift-gitops operator and the App-of-Apps must be deployed. Your Argo CD should look somehow like this:

Figure 1. Argo CD: Initial Applications

But how do all these Applications end up in Argo CD and how can you add additional ones?

Understanding the Argo CD Resources Manager

For any further references, I am using the GitHub repository OpenShift Clusterconfig GitOps

The Argo CD Resources Manager is, in fact, the App-of-Apps. Its configuration file can be found in the directory base/argocd-resources-manager. It is simply a values file and uses the Helm Chart helper-argocd to create additional Applications or ApplicationSets for Argo CD.

Analyzing the example values file

The example values file seems to be huge and confusing at first look. But it is quite easy to understand … trust me :)

Let’s walk through the file bit by bit:

Defining Header Variables

At the top of the file, some variables are defined as so-called anchors. These definitions might be used multiple times and are defined at the top to allow us to find and change them easily.

I define here the clusters and the information about the GitHub repository for example:

mgmt-cluster: &mgmtcluster https://kubernetes.default.svc (1)
mgmt-cluster-name: &mgmtclustername in-cluster (2)
production-cluster: &prodcluster https://api.ocp.aws.ispworld.at:6443
production-cluster-name: &prodclustername prod
repourl: &repourl 'https://github.com/tjungbauer/openshift-clusterconfig-gitops' (3)
repobranch: &branch main (4)

1	Define the API URL for the cluster as configured in Argo CD. (The local cluster where Argo CD is running might be called kubernetes.default.svc)
2	Define the short name of the cluster as configured in Argo CD. (The local cluster where Argo CD is running might be called in-cluster)
3	Define the URL to the GitHub repository that is used in this file.
4	Define the git branch that will be used.

Any additional anchor can be used to define values, that should show up at the very top and/or are used multiple times and you do not want to write them each time.

Whenever you see a value defined as *repourl for example, such an anchor is used.

Understanding Naming Conventions

Each Application or ApplicationSet must have a unique name inside Argo CD. Whenever Applications are generated by ApplicationSet a prefix with the name of the cluster is usually added.

In the values file multiple Applications or ApplicationSets can be defined. They are all bypassed to the Helm Chart which takes care of everything. The name that will be used for an ApplicationSet for example will be the (yaml) key of the definition.

For example, ApplicationSets are defined as:

applicationsets:
mgmt-cluster:
...
enable-etcd-encryption:
...

The keys, for example, mgmt-cluster or enable-etcd-encryption are used as names of the ApplicationSets. This way you do not need to take care of unique names, as YAML will already complain if some kwy is used twice.

Enable/Disable

Like with all of my Helm Charts, I added a switch to enable (or disable) a certain configuration. This way, you can easily remove Applications without actually deleting the specification.

The default value is false, so you actively need to set it to true

 mgmt-cluster:
# Is the ApplicationSet enabled or not
enabled: true

Supported Generators

The helm chart helper-argocd supports the following generators currently:

Matrix Generator
List Generator
Git Generator (for files)
Cluster Generator

Additional generators might be added in the future (ping me or create a pull request), but I found these the most useful ones.

But what is the difference between these generators from the configuration point of view? The different generators require different configurations and therefore provide different placeholders for variables. While the Git generator might use variables that are defined in a file that it finds {{environment}} the List (or Cluster) generator is using {{url}} to define the target cluster.

This might make the specification of an ApplicationSet quite complex … and that’s the whole reason for creating the helper-argocd

Example ApplicationSet - Matrix Generator

The first ApplicationSet I would like to show is probably the most important one. As described in GitOps Repository Structure I am using a folder structure like clusters/management-cluster/ and in this folder I am defining any configuration that is applicable for that specific cluster. If I want to add a new cluster, I simply create a new folder (and a new App-of-Apps configuration). With this, you will always see which settings a specific cluster has without much hassle.

The idea is to walk over this folder and automatically create a new Argo CD Application for any sub-folder that is found. This has the advantage, that whenever I want to create an additional configuration for a cluster, I simply add another sub-folder and the ApplicationSet will automatically create a new Argo CD application.

To achieve this the so-called Matrix Generator is used. This generator combines two (currently two are possible only) generators. In our case, it combines:

git generator: to walk over the folder and get and sub-folder
list generator: to define the target cluster

The snippet of the configuration will look like the following:

 # Definition of Matrix Generator. Only 2 generators are supported at the moment
generatormatrix: (1)
# Git: Walking through the specific folder and take whatever is there.
- git: (2)
directories:
- path: clusters/management-cluster/*
- path: clusters/management-cluster/waves
exclude: true
repoURL: *repourl
revision: *branch
# List: simply define the targetCluster. The name of the cluster must be known by Argo CD
- list: (3)
elements:
# targetCluster is important, this will define on which cluster it will be rolled out.
# The cluster name must be known in Argo CD
- targetCluster: *mgmtclustername

1	Using matrix generator
2	The first generator is Git: It will observe any changes in the folder clusters/management-cluster and will create a new Argo CD Application if a new sub-folder is found. However, it excludes the folder clusters/management-cluster/waves/
3	The second generator is List: It simply defines the target cluster where the Application that is created by the ApplicationSet shall be deployed.

Now let us bring the whole example together:

 mgmt-cluster: (1)
# Is the ApplicationSet enabled or not
enabled: true (2)
# Description - always useful
description: "ApplicationSet that Deploys on Management Cluster Configuration (using Matrix Generator)" (3)
# Any labels you would like to add to the Application. Good to filter it in the Argo CD UI.
labels: (4)
category: configuration
env: mgmt-cluster
# Using go text template. See: https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/GoTemplate/
goTemplate: true (5)
argocd_project: *mgmtclustername (6)
# preserve all resources when the application get deleted. This is useful to keep that workload even if Argo CD is removed or severely changed.
preserveResourcesOnDeletion: true (7)
# Definition of Matrix Generator. Only 2 generators are supported at the moment
generatormatrix: (8)
# Git: Walking through the specific folder and take whatever is there.
- git:
directories:
- path: clusters/management-cluster/*
- path: clusters/management-cluster/waves
exclude: true
repoURL: *repourl
revision: *branch
# List: simply define the targetCluster. The name of the cluster must be known by Argo CD
- list:
elements:
# targetCluster is important, this will define on which cluster it will be rolled out.
# The cluster name must be known in Argo CD
- targetCluster: *mgmtclustername
syncPolicy: (9)
autosync_enabled: false

1	Key of the ApplicationSet inside the yaml specification, that will be used as object name
2	Is the ApplicationSet enabled or not (Default: false)
3	A useful description
4	Labels that can be used to filter
5	Enable the usage of Go Template for this ApplicationSet
6	The Argo CD project (not OpenShift project) the ApplicationSet belongs to
7	Be sure that resources are not deleted when deleting the ApplicationSet. I found this quite useful … otherwise, all Applications the ApplicationSet created will be removed INCLUDING the resources they have created.
8	The specification of the matrix generator
9	Any kind of syncPolicy … in this case automatic synchronization of the Applications that are created is disabled.

Based on these settings the helper-argocd helm chart will render an ApplicationSet object automatically. As mentioned above it will be called mgmt-cluster and creates an Application for any sub-folder it finds in clusters/management-cluster.

Any new folder that is added will automatically create a new Application. You do not need to configure anything else.

The full objects will look like this:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: mgmt-cluster (1)
namespace: openshift-gitops
labels:
app.kubernetes.io/instance: argocd-resources-manager
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: helper-argocd
category: configuration
env: mgmt-cluster
helm.sh/chart: helper-argocd-2.0.28
spec:
generators: (2)
- matrix:
generators:
- git:
directories:
- path: clusters/management-cluster/*
- exclude: true
path: clusters/management-cluster/waves
repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops'
revision: main
- list:
elements:
- targetCluster: in-cluster
goTemplate: true
goTemplateOptions:
- missingkey=error
syncPolicy:
preserveResourcesOnDeletion: true
template:
metadata:
name: '{{ .targetCluster }}-{{ .path.basenameNormalized }}' (3)
spec:
destination: (4)
name: '{{ .targetCluster }}'
namespace: default
info:
- name: Description
value: ApplicationSet that Deploys on Management Cluster Configuration (using Matrix Generator)
project: in-cluster
source: (5)
path: '{{ .path.path }}'
repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops'
targetRevision: main

1	Name of the object == name of the key in the values file definition
2	Configuration of the matrix generator
3	Name of the Applications that this ApplicationSet will generate. In this case, it will concat the name of the target cluster and the name of the path.
4	Target cluster
5	Definition of the source for the Application

Example ApplicationSet - Git Generator

Now let us take a look at a second example using the git generator. The basic idea is quite similar and just a few minor changes must be made to our configuration.

In the following object, a git file generator is used to observe a specific folder and look for the file named values.yaml. For each file that it found an Application is created.

This example is also explained in my article at Project onboarding using GitOps and Helm

 # Tenant Onboarding (using Git Generator)
onboarding-tenant-workload: (1)
# Is the ApplicationSet enabled or not
enabled: true
# Description - always useful
description: "Onboarding Workload to the cluster"
# Any labels you would like to add to the Application. Good to filter it in the Argo CD UI.
labels:
catagory: tenant-onboarding
# Path to the Git repository. The default URL and revision are defined as anchors at the beginning of the file, but could be overwritten here.
path: clusters/all/project-onboarding (2)
repourl: *repourl
targetrevision: *branch
# Using go text template. See: https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/GoTemplate/
goTemplate: true
# Helm configuration. A list of helm values files
helm: (3)
per_cluster_helm_values: false
value_files:
- '/{{ .path.path }}/values.yaml'
- /tenants/values-global.yaml
# Generator: currently list, git and cluster are possible.
# either "generatorlist", "generatorgit" or "generatorclusters"
# Define the repository that shall be checked for configuration file
generatorgit: (4)
- repourl: *repourl
targetrevision: *branch
files:
- tenants/**/values.yaml (5)
# preserve all resources when the application gets deleted. This is useful to keep that workload even if Argo CD is removed or severely changed.
preserveResourcesOnDeletion: true

1	Name of the ApplicationSet
2	The repo URL and path which shall be read for the ApplicationSet
3	A list of values files, that shall be used.
4	The specification of the Git generator
5	The path that shall be observed by this ApplicationSet. ** will return all files and directories recursively.

Again, the Helm chart helper-argocd will render an ApplicationSet for us.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: onboarding-tenant-workload
namespace: openshift-gitops
labels:
app.kubernetes.io/instance: argocd-resources-manager
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: helper-argocd
catagory: tenant-onboarding
helm.sh/chart: helper-argocd-2.0.28
spec:
generators:
- git:
files:
- path: tenants/**/values.yaml
repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops'
revision: main
goTemplate: true
goTemplateOptions:
- missingkey=error
syncPolicy:
preserveResourcesOnDeletion: true
template:
metadata:
name: '{{ index .path.segments 1 | normalize }}-{{ .path.basename }}'
spec:
destination:
name: '{{ .environment }}'
namespace: default
info:
- name: Description
value: Onboarding Workload to the cluster
project: default
source:
helm:
valueFiles:
- '/{{ .path.path }}/values.yaml'
- /tenants/values-global.yaml
path: clusters/all/project-onboarding
repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops'
targetRevision: main

Example ApplicationSet - List Generator

At this point, we have seen two examples of ApplicationSets defined for helper-argocd. The List generator will be very easy to understand as it simply uses a list of target clusters to render the ApplicationSet.

The following snippet demonstrates that all you need to set are the clustername and clusterurl.

Please also verify the article Argo CD and Release Management with Helm Charts and ApplicationSets to understand the usage of the setting chart_version.

 # List of clusters
# "clustername" (string): Is the name of the cluster a defined in Argo CD
# "clusterurl" (string): Is the URL of the cluster API
# "chart_version" (string, optional): Defines which chart version shall be deployed on each cluster.
generatorlist:
- clustername: *mgmtclustername
clusterurl: *mgmtcluster

This is all the magic. The rendered ApplicationSet will look like:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: install-sonarqube
namespace: openshift-gitops
labels:
app.kubernetes.io/instance: argocd-resources-manager
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: helper-argocd
category: project
helm.sh/chart: helper-argocd-2.0.28
spec:
generators:
- list:
elements:
- cluster: in-cluster
url: 'https://kubernetes.default.svc'
template:
metadata:
name: '{{ cluster }}-install-sonarqube'
spec:
destination:
namespace: sonarqube
server: '{{ url }}'
info:
- name: Description
value: Install Sonarqube
project: '{{ cluster }}'
source:
chart: sonarqube
helm:
releaseName: sonarqube
repoURL: 'https://charts.stderr.at/'
targetRevision: 1.0.1

The list generator can be used to deploy on ALL clusters too. Simply define generatorlist: []

The example above also demonstrates how to use a Helm chart instead of a git repository.

What about Applications?

The examples above show the usage of ApplicationSet and recently I migrated any specification of an Application to ApplicationSets as I believe this is easier to use, especially when the Chart is rendering it for you. However, it is still possible to define Applications as well.

The following example defines such an Application. The configuration differs compared to the ApplicationSet, however, the main idea stays the same:

applications: (1)
node-labelling: (2)
enabled: true
description: "Deploy Node Labels"
labels:
category: configuration
namespace:
name: default
create: false
server: *mgmtcluster (3)
project: default
syncOptions: (4)
- name: ServerSideApply
value: true
- name: Validate
value: false
source: (5)
path: clusters/management-cluster/node-labels
helm:
valuesfiles:
- name: values.yaml
repourl: *repourl
targetrevision: *branch

1	Defining Applications
2	Name of the Application. Be sure that it is unique since this time no prefix will be added
3	The target cluster for this Application
4	Different options for the synchronization
5	The specification of the source. This contains the path, URL and branch of the repository and (in this case) the definition of a Helm values file.

Summary

I hope I was able to explain the usage of my chart helper-argocd and how I configure it. You can also verify the README to find additional possible settings and the example values.file that I use for all my clusters when I to a demo.

[Ep.3] Setup OpenShift GitOps/Argo CD

Fri, 02 Feb 2024 00:00:00 +0000

„If it is not in GitOps, it does not exist“ - is a mantra I hear quite often and also try to practice at customer engagements. The idea is to have Git as the only source of truth on what happens inside the environment. That said, Everything as Code is a practice that treats every aspect of the system as a code. Storing this code in Git provides a shared understanding, traceability and repeatability of changes.

While there are many articles about how to get GitOps into the deployment process of applications, this one rather sets the focus on the cluster configuration and tasks system administrators usually have to do.

Also check out the article GitOps Argo CD

Prerequisites

It all begins with an OpenShift cluster. Such a cluster must be installed and while we will not discuss a bootstrap of the whole cluster … yes, it is possible to even automate the cluster deployment using Advanced Cluster Management as an example, we will simply assume that one cluster is up and running.

For our setup, an OpenShift cluster 4.14 is deployed and we will use the repository OpenShift Cluster Configuration using GitOps to deploy our configuration onto this cluster. This repository shall act as the source of truth for any configuration. In the article Choosing the right Git repository structure I have explained the folder structure I am usually using. As tool I am usually using Helm Charts.

The openshift-clusterconfig-gitops repository heavily uses the Helm Repository found at https://charts.stderr.at/

Deploy OpenShift-GitOps

The first thing we need to do is to deploy OpenShift-GitOps, which is based on the Argo CD project. OpenShift-GitOps comes as an Operator and is available to all OpenShift customers. The Operator will deploy and configure Argo CD and provide several custom resources to configure Argo CD Applications or ApplicationSets for example.

To automate the operator deployment the following shell script can be used: init_GitOps.sh.

This Shell script is the only script that is executed manually. It installs and configures Argo CD. Any other operation on the cluster must then be done using GitOps processes. I am using this to quickly install a new Demo-cluster. There are alternatives and maybe better way, but for my purpose it works pretty well.

Clone the repository to your local machine

git clone https://github.com/tjungbauer/openshift-clusterconfig-gitops.git

Be sure that you are logged in the the required cluster
```
oc whoami --show-server
```
Execute the init_GitOps.sh
```
./init_GitOps.sh
```

The script will deploy the operator and configure/patch the Argo CD instance. In addition, it will create the so-called Application of Applications, which acts as an umbrella Application, that automatically creates all other Argo CD Application(Sets). For now, the App of Apps is the only Argo CD Application that automatically synchronizes all changes found in Git. This is for security, purposes so you can test the cluster configuration one after another.

Of course, it is up to you if you want to use the shell script. The Operator can also be installed manually, using Advanced Cluster Manager, or using Platform Operators and installing the Operating during the cluster installation (However, this feature is currently (v4.15) TechPreview)

What will this script do?

I will not de-assemble the script line by line, but in general, the following will happen:

Adding repository https://charts.stderr.at/ and install the Chart openshift-gitops

This FIRST OpenShift-GitOps will be deployed with cluster-admin privileges since we want to manage the whole cluster configuration. This Argo CD instance should not be used for application deployment. For that, deploy additional instances of GitOps.

Waiting for Deployments to become ready
Deploy the Application of Applications that is responsible for automatically deploying a set of Applications or ApplicationSets (see [The Argo CD Object Manager Application])

The following shows the output of the command:

Expand me...

❯ ./init_GitOps.sh
Starting Deployment
Deploying OpenShift GitOps Operator
Adding Helm Repo https://charts.stderr.at/
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /Users/tjungbau/openshift-aws/aws/auth/kubeconfig
"tjungbauer" has been added to your repositories
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /Users/tjungbau/openshift-aws/aws/auth/kubeconfig
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "sealed-secrets" chart repository
...Successfully got an update from the "tjungbauer" chart repository
...Successfully got an update from the "apache-airflow" chart repository
...Successfully got an update from the "hashicorp" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /Users/tjungbau/openshift-aws/aws/auth/kubeconfig
Release "openshift-gitops-operator" has been upgraded. Happy Helming!
NAME: openshift-gitops-operator
LAST DEPLOYED: Mon Sep 26 13:22:33 2022
NAMESPACE: openshift-operators
STATUS: deployed
REVISION: 2
TEST SUITE: None
Give the gitops-operator some time to be installed. Waiting for 45 seconds...
Waiting for operator to start. Chcking every 10 seconds.
NAME READY UP-TO-DATE AVAILABLE AGE
gitops-operator-controller-manager 1/1 1 1 4d4h
Waiting for openshift-gitops namespace to be created. Checking every 10 seconds.
NAME STATUS AGE
openshift-gitops Active 4d4h
Waiting for deployments to start. Checking every 10 seconds.
NAME READY UP-TO-DATE AVAILABLE AGE
cluster 1/1 1 1 4d4h
Waiting for all pods to be created
Waiting for deployment cluster
deployment "cluster" successfully rolled out
Waiting for deployment kam
deployment "kam" successfully rolled out
Waiting for deployment openshift-gitops-applicationset-controller
deployment "openshift-gitops-applicationset-controller" successfully rolled out
Waiting for deployment openshift-gitops-redis
deployment "openshift-gitops-redis" successfully rolled out
Waiting for deployment openshift-gitops-repo-server
deployment "openshift-gitops-repo-server" successfully rolled out
Waiting for deployment openshift-gitops-server
deployment "openshift-gitops-server" successfully rolled out
GitOps Operator ready
Lets use our patched Argo CD CRD
argocd.argoproj.io/openshift-gitops unchanged
clusterrolebinding.rbac.authorization.k8s.io/cluster-admin-0 unchanged
Waiting for deployment cluster
deployment "cluster" successfully rolled out
Waiting for deployment kam
deployment "kam" successfully rolled out
Waiting for deployment openshift-gitops-applicationset-controller
deployment "openshift-gitops-applicationset-controller" successfully rolled out
Waiting for deployment openshift-gitops-redis
deployment "openshift-gitops-redis" successfully rolled out
Waiting for deployment openshift-gitops-repo-server
deployment "openshift-gitops-repo-server" successfully rolled out
Waiting for deployment openshift-gitops-server
deployment "openshift-gitops-server" successfully rolled out
GitOps Operator ready... again
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /Users/tjungbau/openshift-aws/aws/auth/kubeconfig
Release "app-of-apps" has been upgraded. Happy Helming!
NAME: app-of-apps
LAST DEPLOYED: Mon Sep 26 13:23:59 2022
NAMESPACE: openshift-gitops
STATUS: deployed
REVISION: 2
TEST SUITE: None

Logging into Argo CD

At this point, we have GitOps and the "App of Apps" deployed. Argo CD comes with a WebUI and a command line tool. The latter must installed to your local environment. In this article, we will use the WebUI.

To access the WebUI use the applications menu of the top right corner in Openshift.

Figure 1. Argo CD: WebUI Link

Use the button "Login via OpenShift".

Figure 2. Argo CD: Authentication

The Argo CD Resources Manager Application

The Application of Applications (short App of Apps) is called Argo CD Resources Manager and it is the only Argo CD application that is deployed using the init script. This single Argo CD Application has the sole purpose of deploying other Argo CD objects, such as Applications, ApplicationSets and AppProjects.

Figure 3. Argo CD: App of Apps

It synchronizes everything that is found in the repository in the path: base/argocd-resources-manager (main branch)

Whenever you would like to create a new Argo CD application(set) it is supposed to be done using this App-of-Apps or to be more exact: in the path mentioned above.

The App-of-Apps is the only Argo CD Application (at this moment) that has automatic synchronization enabled. Thus any changes in the App-of-Apps will be propagated automatically as soon as GitOps syncs with Git.

The current Applications or ApplicationSets that come with the bootstrap repository are for example:

Deployment of Advanced Cluster Security (RHACS)
Deployment of Advanced Cluster Management (RHACM)
Deployment of basic cluster configuration (i.e. etcd encryption, some UI tweaks …)
Deployment of Compliance Operator
and many more.

Check out the deployed Argo CD objects or the openshift-clusterconfig-gitops repository.

A deep dive into the argocd-resources-manager will be topic of a different episode of this serie.

[Ep.2] Choosing the right Git repository structure

Thu, 28 Dec 2023 00:00:00 +0000

One of the most popular questions asked before adopting the GitOps approach is how to deploy an application to different environments (Test, Dev, Production, etc.) in a safe and repeatable way.

Each organisation has different requirements, and the choice will depend on a multitude of factors that also include non-technical aspects.

Therefore, it is important to state: "There is no unique “right” way, there are common practices".

In this series, I will focus on cluster configuration.

Git Repository Strategy - Options

As written in the introduction: There is no unique “right” way, there are common practices. But how shall the Git repository structure look like? How shall the folder structure look like? Multiple options might be considered. Each has advantages and disadvantages, some I would recommend, some I would not recommend.

It is important to understand that:

The Git repository structure will depend heavily on how the organisation is laid out.
The final repo and directory structure is unique for every organisation, as such the right one will be a discovery process within the organisation and the teams involved in the GitOps engineering process.

Before I describe what I usually try to leverage, let’s see the different options.

Environment-per-branch

In this case, there is a Git branch for each environment. A “Dev” branch holds the configuration for the DEV environments, a “production” branch for production and so on. This approach is very popular and will be familiar to people who have adopted git flow in the past. However, it is focused on application source code and not environment configuration and is best used when you need to support multiple versions of your application in production. I do not recommend this approach for GitOps, the main reasons are that pull requests and merges will be very complex and promotions between environments are a hurdle. The whole life cycle of a cluster configuration will be very complex.

Environment-per-folder - Monorepo

In this case, all environments are in a single Git repository, and all are in the same branch. The filesystem has different folders that hold configuration files for each environment. The configuration of the “DEV” environment is described by a “DEV” folder, the “production” environment is found in a “production” folder and so on.

Figure 1. GitOps Monorepo Approach

This is the approach I usually recommend, especially when someone is new to the whole GitOps workflow and because of the simplicity of setting up such a repository.

The following advantages and disadvantages should be considered:

Pros

Provides a central location for configuration changes.
This simplicity enabled straightforward Git workflows that will be centrally visible to the entire organisation, allowing a smoother and clearer approval process and merging.
Better suitable for small teams that are managing the cluster and easy to read and understand
Easy to debug problems.

Cons

Scalability >> Increase complexity >> Management
Performance for huge repositories
Challenging to control access permissions on a single repository

Environment-per-repository - Multirepo

In this case, each environment is on its separate git repository. So, the DEV environment is in a git repository called “DEV”, the “production” environment is in a “production” git repository and so on. The GitOps agent (OpenShift GitOps) connects to multiple repositories and takes care to apply the correct configuration to the correct target cluster.

Figure 2. GitOps Multirepo Approach

Like the Monorepo approach, Multirepo comes with some advantages and disadvantages:

Pros

Allows separating concerns between different departments of organisations (a repository for the security team, a repository for the operations team, etc.)

Cons

More complex to manage
Harder to understand and read the configuration (what is coming from where)
Argo CD Application dependencies might not be solved (i.e., Security tries to manage the same object as the operating team. Who is the leader?)

Example Setup

The Approach I choose

I usually use and recommend Monorepo approach. The environment-per-folder approach is a very good way to organise your GitOps applications. Not only is it very simple to implement and maintain, but it is also the optimal method for promoting releases between different GitOps environments. This approach can also work for any number of environments without any additional effort. Cluster configurations (for multiple clusters) are typically done by one team, therefore controlling access permissions in Git is not a big issue.

Example Folder Structure

Over time the following folder structure evolved or my repository:

├── base (1)
│   ├── argocd-resources-manager (2)
│   └── init_app_of_apps (3)
├── charts (4)
├── clusters (5)
│   ├── all (6)
│   │   ├── base-operators
│   │   ├── etcd-encryption
│   ├── management-cluster (7)
│   │   ├── branding
│   │   ├── generic-cluster-config
│   │   ├── management-gitops
│   │   ├── node-labels
│   │   ├── openshift-data-foundation
│   │   ├── setup-acm
│   │   ├── setup-acs
│   │   ├── setup-compliance-oeprator
│   │   ├── setup-openshift-logging
│   │   └── setup-quay
│   └── production-cluster (8)
│   │   ├── branding
│   │   ├── generic-cluster-config
│   │   ├── node-labels
│   │   ├── openshift-data-foundation
│   │   ├── setup-acs
│   │   ├── setup-compliance-oeprator
│   │   └── setup-openshift-logging
├── init_GitOps.sh (9)
├── scripts (10)
│   ├── example_htpasswd
│   ├── sealed_secrets
├── tenant-projects (11)
   ├── my-main-app
   └── my-second-app

1	The `base` folder contains basic configurations or Argo CD itself.
2	The `argocd-resources-manager` is a Helm Chart that configures Applications and ApplicationSets or Argo CD using a single configuration file.
3	The `init_app_of_apps` is used during the initial installation of OpenShift GitOps and installs the App-of-Apps that manages other Applications or Argo CD. This Application automatically synchronises and watches for changes in the folder `argocd-resources-manager`.
4	The `charts` folder is optional and can store local Helm Charts. Usually, it is better to release the Charts in a Helm repository, where they can be managed independently to the cluster configuration repository.
5	The folder for the different clusters.
6	Configurations that are equal for all clusters and simple to achieve without any deeper configuration. Currently, for example, the activation of the etcd encryption and the deployment of base Operators that every cluster will require. In this case, the Operators are installed only, without further configuration.
7	Configuration for the `management-cluster`. For example, deploying ACM, ACS, Quay or any generic cluster configuration. Here we see immediately what is deployed and where I can modify the configuration for that cluster.
8	Configuration for the `production-clusters`
9	The deployment script to install and configure the OpenShift GitOps Operator. This might be replaced or at least modified in the future once PlatformOperators are generally available and not in a technology preview state anymore.
10	The `scripts` folder simply contains some shell scripts that might be useful. For example, to backup a Sealed Secrets key or generate a htpasswd file.
11	The `tenant-projects` folder is a special folder to store the configuration or projects. Any project onboarding is configured here, such as Quota, LimitRanges, NetworkPolicies etc.

Why the repeating folders?

Some may argue why certain folders are equal for management and production clusters, for example, "setup-compliance-operator", when this could be done more easily by defining such folder only once and using different overlays (using Kustomize) or different values-files (using Helm Charts). However, while this is a very valid question, I personally, like to see immediately what is configured on each cluster. I see, based on the folders, what is configured on the management cluster and where I could modify the configuration.

Using Kustomize overlays, for example, would mean recreating the overlays for each configuration (if you want to have a clean separation and not combine all manifests into one overlay). Using different values-files is again a valid option, but (also again), you do not see what is configured on which cluster with one look.

Therefore, I like this folder structure, even if it may look weird (especially if you are used to Kustomize overlays). However, everyone is invited to define their very own structure :)

Managing Kubernetes Manifests

The Kubernetes manifests (the yaml files) must be managed in a way Argo CD can read and synchronise them.

Three main options are commonly used:

Helm: Helm uses a packaging format called charts. A chart is a collection of files that describe a related set of Kubernetes resources.
Kustomize: A Template-free way to customise application configuration that simplifies the use of off-the-shelf applications.
Plain Text: Plain text Kubernetes objects provided in YAML of JSON format.

Argo CD also understands jsonnet or even custom plugins. However, I had no customer up until now, who wanted to use something else than Kustomize or Helm.

The different tools are not explained in detail in this article, but the choice of the tool highly depends on the existing knowledge and individual preferences inside the company. Every option has advantages and disadvantages that will become visible when they are used.

I have seen companies tend to use Helm Charts or Plain Text, especially when they are new to the tools. However, no tool is better than the other. Instead, the tools can be combined which might be useful for some use cases.

Kustomize and Helm do not exclude each other and can be combined. However, for the start, a single tool should be selected.

[Ep.1] Introducing the GitOps Approach

Fri, 01 Dec 2023 00:00:00 +0000

When managing one or more clusters, the question arises as to how cluster configurations and applications can be installed securely, regularly, and in the same way. This is where the so-called GitOps approach helps, according to the mantra: "If it is not in Git, it does not exist".

The idea is to have Git as the only source of truth on what happens inside the environment. While there are many articles about how to get GitOps into the deployment process of applications, this series of articles tries to set the focus on the cluster configuration and tasks system administrators usually have to do, for example: Setup an Operator.

The GitOps Approach

This series includes the following articles:

In this series, I will focus on cluster configuration.

The GitOps approach is a very common practice and the-facto "standard" as of today.

When I write standard, then be assured, that the approach itself should be followed, but HOW this is done can be a topic of many tough discussions.

But what is it and why should a company invest time to follow this approach?

_GitOps is a declarative way to implement continuous deployment for cloud-native applications. It should be a repeatable process to manage multiple clusters.

GitOps adds the following features to company processes:

Everything as code: The entire state of the application, infrastructure and configuration is declaratively defined as code.
Git and the single source of truth: Every setting and every manifest is stored and versioned in Git. Any change must first be saved to Git.
Operations via Git workflows: Standard Git procedures, such as pull or merge requests, should be used to track any changes to the applications or cluster configurations.

It is important that not only the manifests of the applications but also the cluster configuration is stored in Git. The goal should be to ensure that no manual changes are made directly to the cluster.

Benefits/Challenges

Deploying new versions of applications or cluster configurations with a high degree of confidence is a desirable goal as getting features reliably to production is one of the most important characteristics of fast-moving organisations. GitOps is a set of common practices where the entire code delivery process is controlled via Git, including infrastructure and application definition as code and automation to complete updates and rollbacks.

GitOps constantly watches for changes in Git repositories and compares them with the current state of the cluster. If there is a drift it will either automatically synchronise to the wanted state or warn accordingly (manual sync must then be performed).

The key GitOps advantages are:

Cluster and application configuration versioned in Git
Visualisation of desired system state
Automatically syncs configuration from Git to clusters (if enabled)
Drift detection, visualisation, and correction
Rollback and roll-forward to any Git commit.
Manifest templating support (Helm, Kustomize, etc.)
Visual insight into sync status and history.
Role-Based access support
Pipeline integration

Adopting GitOps has enormous benefits but does pose some challenges. Many teams will have to adjust their culture and way of working to support using Git as the single source of truth. Strictly adhering to GitOps processes will mean all changes will be committed. This may present a challenge when it comes to debugging a live environment. There may be times when that is necessary and will require suspending GitOps in some way.

Some other prerequisites for adopting GitOps include

Good testing and CI processes are in place.
A strategy for dealing with promotions between environments.
Strategy for Secrets management.

Used Tools

The following list of tools (or specifications) are used for our GitOps Approach.

Used Repositories

The following two Git repositories are used throughout the series: