YAUB Yet Another Useless Blog on TechBlog about OpenShift/Ansible/Satellite and much more

Onboarding to Ansible Automation Platform with Configuration as Code

Wed, 04 Mar 2026 00:00:00 +0000

We had the honor of presenting at the 2nd Ansible Anwendertreffen in Austria. The topic was the onboarding of application teams to the Ansible Automation Platform.

We created an extensive demo that:

Onboards a new tenant into an AAP organization.
Each tenant gets a configuration as code repository, cloned from a template.
A push event to the config-as-code repository triggers an update of AAP objects.
Provides an example repository for each tenant to get started.
The example repository provides a webhook that creates an OpenShift Virtualization VM for testing code changes when a feature branch is created.
Contains a playbook to display objects that are not currently managed by the tenant’s configuration-as-code repository.

The source code for the demo is available on GitHub.

The README contains more details about the implementation, and the slide deck is also available on GitHub.

OKD Single node installation in a disconnected environment

Mon, 09 Feb 2026 00:00:00 +0000

Because of reasons we had to set up a disconnected single node OKD "cluster". This is our brain dump as the OKD documentation sometimes refers to OpenShift image locations, we had to read multiple sections to get it working and we also consulted the OpenShift documentation at https://docs.redhat.com.

Updated on 2026-03-03: Fixed install-config.yaml and create manifest command

First we need to download the oc command in the version we would like to install. OKD provides it on its release page:

https://github.com/okd-project/okd/releases/download/4.21.0-okd-scos.3/openshift-client-linux-amd64-rhel9-4.21.0-okd-scos.3.tar.gz

To get an overview of releases available for OKD see https://amd64.origin.releases.ci.openshift.org/.

Another option is to extract the oc command from the release image, if you have a version of oc already installed:

oc adm release extract --tools quay.io/okd/scos-release:4.21.0-okd-scos.3

This will also extract the openshift-install tar.gz which we need later to generate the installation manifests and ignition configs.

Next we need the oc-mirror plugin, which is used to mirror the release content to our local registry. We were only able to find the plugin on console.redhat.com. Another option might be to compile the plugin from source code (https://github.com/openshift/oc-mirror/).

We copied the oc command and the oc-mirror plugin to /usr/local/bin/ and made them executable. After that we ran oc mirror --v2 --help to test the installation.

The upstream oc-mirror plugin documentation was also helpful.

As our private registry required authentication, we created a .dockerconfigjson file with the registry credentials:

{
"auths": {
"internal.registry": {
"auth": "<credentials base64 encoded>",
"email": "you@example.com"
}
}
}

You can encode the credentials with:

echo -n "username:password" | base64 -w0

For mirroring all required OKD images to our private registry we created an ImageSetConfiguration:

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
platform:
channels:
- name: 4-scos-stable
minVersion: 4.21.0-okd-scos.3
maxVersion: 4.21.0-okd-scos.3
graph: false
operators:
- registry: quay.io/okderators/catalog-index:testing-4.20 (1)
packages:
- name: aws-load-balancer-operator
- name: 3scale-operator
- name: node-observability-operator
additionalImages:
- name: registry.redhat.io/ubi8/ubi:latest
- name: registry.redhat.io/ubi9/ubi@sha256:20f695d2a91352d4eaa25107535126727b5945bff38ed36a3e59590f495046f0

We used a minimal configuration without operators and additional images, because downloading additional images did not work for us:

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
platform:
channels:
- name: 4-scos-stable (1)
type: okd
minVersion: 4.21.0-okd-scos.3 (2)
maxVersion: 4.21.0-okd-scos.3
graph: false
operators: []
additionalImages: []

The hard part for us was finding the right values for channels and the min/max versions. Once again the OKD release status page was helpful.

Our private registry is based on Harbor for our experiments, but any Docker v2 compatible registry should work. The only thing required is a project within Harbor called openshift and a user/password for pulling/pushing images to this project. The project name might be configurable with oc mirror, but this requires further investigation.

oc mirror wants to verify signatures so we had to download a public key and provide the OCP_SIGNATURE_URL:

curl -LO https://raw.githubusercontent.com/openshift/cluster-update-keys/master/keys/verifier-public-key-openshift-ci-4
export OCP_SIGNATURE_URL="https://storage.googleapis.com/openshift-ci-release/releases/signatures/openshift/release/"
export OCP_SIGNATURE_VERIFICATION_PK="verifier-public-key-openshift-ci-4"
# for debugging add --log-level debug, but we had intermittent errors with this option.
# after removing --log-level debug oc mirror ran without problems
oc mirror -c ImageSetConfiguration.yaml --authfile docker-auth.json --workspace file:///workspace docker://internal.registry --v2

This will create the ImageDigestMirrorSource in /workspace/working-dir/cluster-resources/idms-oc-mirror.yaml and ImageTagMirrorSource custom resources in /workspace/working-dir/cluster-resources/itms-oc-mirror.yaml.

The content of ImageDigestMirrorSource will be reused in the install-config.yaml. This is required to redirect image pulls done by the node from quay.io to our private registry.

Next we need DNS records for our cluster:

api.sno.internal
api-int.sno.internal
*.apps.sno.internal (wildcard DNS entry for applications deployed on the cluster)

Now we prepared the required install-config.yaml file to trigger the openshift-install command. We basically followed Installing a Single Node OpenShift Cluster.

apiVersion: v1
baseDomain: internal
compute:
- hyperthreading: Enabled
name: worker
replicas: 0 (1)
controlPlane:
hyperthreading: Enabled
name: master
replicas: 1 (2)
metadata:
name: okd-sno
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
networkType: OVNKubernetes
serviceNetwork:
- 172.30.0.0/16
platform:
none: {}
fips: false
sshKey: 'the key'
bootstrapInPlace: (3)
installationDisk: /dev/vda
pullSecret: |
{
"auths": {
"registry.internal": {
"auth": "<username:password base64 encoded>",
"email": "some@email"
}
}
}
additionalTrustBundle: | (4)
-----BEGIN CERTIFICATE-----
ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ
-----END CERTIFICATE-----
ImageDigestSources: (5)
- mirrors:
- registry.internal/openshift/release
source: quay.io/okd/scos-content
- mirrors:
- registry.internal/openshift/release-images
source: quay.io/okd/scos-release

1	We set the number of workers to zero (0). Because this is a SNO installation. Worker nodes could be added later, even for a SNO cluster
2	We set the number of master nodes to one (1)
3	This tells the installer that we do not have a bootstrap node and it should create one large ignition config file
4	If required the CA certificate of our registry. This is required if the registry uses an internal certificate
5	ImageDigestSources configures CRIO to redirect requests to quay.io to our private registry.

It’s time to create the installation manifests and ignition configs. We created a directory install, copied the install-config.yaml into this directory and triggered the openshift-install command:

$ mkdir install
$ cp install-config.yaml install/
$ openshift-install --dir=install create single-node-ignition-config

We are now ready to download the installation ISO and to embed our ignition config into it. We also want to set kernel boot arguments to configure the network interface with a static IP address.

curl -L $( ./openshift-install coreos print-stream-json |jq -r ".architectures.x86_64.artifacts.metal.formats.iso.disk.location" ) -o fhcos-live.iso

The next step is to modify the ISO. We basically followed the instructions in the OKD documentation: https://docs.okd.io/4.14/installing/installing_sno/install-sno-installing-sno.html#generating-the-install-iso-manually_install-sno-installing-sno-with-the-assisted-installer

alias coreos-installer='podman run --privileged --pull always --rm -v /dev:/dev -v /run/udev:/run/udev -v $PWD:/data -w /data quay.io/coreos/coreos-installer:release'
coreos-installer iso ignition embed -fi install/bootstrap-in-place-for-live-iso.ign fcos-live.iso
coreos-installer iso kargs modify --append 'ip=10.0.0.99::10.0.0.1:255.255.255.0:sno.internal:ens33:none:10.0.0.255' fcos-live.iso

We can take a look at the ISO bootstrap config and the kernel arguments with:

coreos-installer iso ignition show fcos-live.iso
coreos-installer iso kargs show fcos-live.iso

For understanding the kernel boot arguments see https://access.redhat.com/solutions/5499911.

The final step is to boot the modified ISO on the target machine and wait for the installation to complete.

OpenShift Virtualization Networking - The Overview

Mon, 29 Dec 2025 00:00:00 +0000

It’s time to dig into OpenShift Virtualization. You read that right, OpenShift Virtualization, based on kubevirt allows you to run Virtual Machines on top of OpenShift, next to Pods. If you come from a pure Kubernetes background, OpenShift Virtualization can feel like stumbling into a different dimension. In the world of Pods, we rarely care about Layer 2, MAC addresses, or VLANs. The SDN (Software Defined Network) handles the magic and we are happy.

But Virtual Machines are different….

They are needy creatures. They often refuse to live in the bubble of the default Pod network. They want to talk to the external Oracle database on the bare metal server next door, they need a static IP from the corporate range, or they need to be reachable via a specific VLAN. Networking is a key part of the Virtualization story and we need to master it.

To achieve this, we need to understand two tools: NMState Operator and Multus.

In this guide, I will try to discuss some of the basics for "Multihomed Nodes" and look at the three main ways to connect your VMs to the outside world.

Who does what: NMState vs. Multus

Before we look at YAML, we need to clear up a common confusion. Who does what?

Think of your OpenShift Node as a house.

NMState (The Electrician): This operator works on the Node (Host) level. It drills holes in the walls, lays the physical cables, and installs the wall sockets. In technical terms: It configures Bridges, Bonds, and VLANs on the Linux OS of the worker nodes.
Multus (The Extension Cord): This CNI plugin works on the VM/Pod level. It plugs the VM into the socket that NMState created. It allows a VM to have more than one network interface (eth0, eth1, etc.).

You usually need both. First, NMState builds the bridge. Then, Multus connects the VM to it.

NMState Operator Installation

While Multus is installed by default, NMState is not. You need to install the Operator first.

To install the NMState Operator, you can use the Web UI or the CLI. For the Web UI, you can use the following steps:

In the OpenShift Web Console, navigate to Operators → OperatorHub (or Ecosystem → Software Catalog in version 4.20+).
In the OperatorHub, search for NMState and select the Operator from the list.
Click on the NMState Operator and click on the Install button.
Select the Install button to install the Operator.
After the Operator is installed, you can check the status of the Operator in the Installed Operators view.

Once done you will need to create an instance of the NMState Operator. Simply create the following resource:

apiVersion: nmstate.io/v1
kind: NMState
metadata:
name: nmstate
spec:
probeConfiguration:
dns:
host: root-servers.net

NMState Operator Resource Types

The NMState Operator provides the three custom resource types to manage the node networking:

NodeNetworkState (NNS) → The "As-Is" State:
- This is an automatically generated report of the current status. It effectively tells you: "On Server 1, I currently see Network Card A and B, and IP address X is configured."
- Purpose: To verify the current reality on the ground before you make changes.
NodeNetworkConfigurationPolicy (NNCP) → The "To-Be" State (The Blueprint):
- This is the most critical resource. This resource is used to configure the network on the node. Here you tell the cluster that "I want a bridge named br1 to exist on all servers, and it must be connected to port ens4."
- The operator will try to configure this configuration across your nodes.
NodeNetworkConfigurationEnactment (NNCE) → The Result report:
- This resource is created autoamtically after the NNCP (the blueprint) has been created.
- It resports back: This resource is used to enact the network configuration on the node.
- It reports back on the execution: "Success! The bridge has been built," or "Failure! Cable not found."

With NMState in place, we can now start to configure the network on the node.

Part 1: Building the Bridge (NMState)

By default, Virtual Machines (VMs) are connected to the default pod network of the cluster as any other Pod. Using this network, the VM can communicate with other resources inside the cluster, or any resources that are reachable through the node network. Sometimes the VM needs a connection to a different network. In such a case you must connect the VM to an additional network. The VM will be configured with an additional network interface and is considered as multihomed VM.

To connect a VM to the physical network, we first need a bridge on the worker nodes. A bridge forwards packets between connected interfaces, similar to a network switch. We have two choices: Linux Bridge or OVS (Open vSwitch) Bridge.

Linux Bridge: The sturdy, wooden bridge. Simple, robust, standard Linux kernel tech. Use this for 90% of your use cases (Static IPs, simple VLAN access).
OVS Bridge: The magical, floating bridge. Complex, programmable, supports SDN logic. Use this only if you need advanced tunneling or integration with OVN policies on the physical interface.

Let’s create a standard Linux Bridge called br1 on all our worker nodes, attached to physical interface ens4.

We do this by applying a NodeNetworkConfigurationPolicy (NNCP).

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: br1-policy
spec:
nodeSelector:
node-role.kubernetes.io/worker: "" (1)
desiredState:
interfaces:
- name: br1 (2)
description: Linux bridge linked to ens4 (3)
type: linux-bridge (4)
state: up (5)
ipv4:
dhcp: true (6)
enabled: true
bridge: (7)
options:
stp:
enabled: false
port: (8)
- name: ens4

1	Apply to all worker nodes. If omitted, the policy will be applied to all nodes.
2	Name of the bridge
3	Description of the bridge
4	Type of the bridge (linux-bridge)
5	State of the bridge (up)
6	DHCP is enabled for the bridge (true)
7	Options for the bridge
8	Port of the bridge (ens4)

Setting an interface to absent or deleting an NNCP resource does not restore the previous configuration. A cluster administrator must define a policy with the previous configuration to restore settings.

Part 2: Defining the NetworkAttachmentDefinition (NAD)

Now that the bridge br1 exists on the nodes, we need to tell OpenShift how to use it. We do this with a NetworkAttachmentDefinition (NAD).

There are three distinct "Types" of networks you can define in a NAD.

1. The Linux Bridge (CNI-Plugin)

Linux bridges and OVS bridges both connect VMs to additional networks. A Linux bridge offers a simple, stable and sturdy solution and is well established. The OVS bridge on the other hand provides an advanced feature set for software-defined networks and is more challenging to troubleshoot.

Use Case: Direct Layer 2 connection to the physical network.

Behavior: The VM is practically "on the wire" of your datacenter. It can do DHCP with your corporate router.

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: bridge-network
namespace: my-vm-namespace
spec:
config: '{
"cniVersion": "0.3.1",
"name": "bridge-network",
"type": "bridge", (1)
"bridge": "br1", (2)
"ipam": {
"type": "dhcp"
}
}'

1	Type of the network (bridge)
2	Name of the bridge (br1)

type: "bridge" refers to the CNI plugin that utilizes the Linux Bridge we created earlier.

A note on IPAM (IP Address Management)

You might have noticed the "ipam" section in the NAD examples. You have three main choices there:

dhcp: Passes DHCP requests to the physical network (requires an external DHCP server).
static: You manually define IPs in the VM config (hard work).
whereabouts: A "Cluster-wide DHCP" for private networks. Perfect for the OVN L2 Overlay scenario where no external DHCP exists.

2. OVN Kubernetes L2 Overlay

A switched (layer 2) topology network interconnects workloads through a cluster-wide logical switch. This configuration allows East-West traffic only (packets between Pods within a cluster).

Use Case: You need a private network between your VMs (East-West traffic only), but you don’t want them to talk to the physical network.

Behavior: It creates a Geneve tunnel (like VXLAN) over the network.

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: private-overlay
namespace: my-vm-namespace
spec:
config: '{
"cniVersion": "0.4.0",
"name": "private-overlay",
"type": "ovn-k8s-cni-overlay",
"topology": "layer2", (1)
"netAttachDefName": "my-vm-namespace/private-overlay" (2)
}'

1	Topology of the network (layer2)
2	NetworkAttachmentDefinition name (my-vm-namespace/private-overlay)

A pod with the annotation k8s.v1.cni.cncf.io/networks: private-overlay will be connected to the private-overlay network.

3. OVN Kubernetes Secondary Localnet

OVN-Kubernetes also supports a localnet topology for secondary networks. This type creates a connection between a secondary network and a physical network, allowing VMs to communicate with destinations that are outside of the cluster. You must map the secondary network to the OVS bridge on the node. This is done by using a NodeNetworkConfigurationPolicy resource.

Use Case: You want to connect to the physical network (like the Linux Bridge), but you want to leverage OVN features (like NetworkPolicies) on that traffic.

Behavior: Uses Open vSwitch mapping to connect the OVN logic to the physical port.

Let’s create the bridge mapping by applying the following NodeNetworkConfigurationPolicy resource:

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: br0-ovs
spec:
nodeSelector:
node-role.kubernetes.io/worker: "" (1)
desiredState:
interfaces:
- name: br0-ovs (2)
description: OVS bridge with ens4 as a port
type: ovs-bridge (3)
state: up
ipv4:
dhcp: true
enabled: true
bridge:
options:
stp: true
port:
- name: ens4 (4)
ovn:
bridge-mappings:
- localnet: br0-network (5)
bridge: br0-ovs (6)
state: present

1	Apply to all worker nodes. If omitted, the policy will be applied to all nodes.
2	Name of the bridge
3	Type of the bridge (ovs-bridge)
4	Port of the bridge (ens4)
5	Localnet network name (br0-network)
6	OVS bridge name (br0-ovs)

Now let’s create the NetworkAttachmentDefinition for the localnet network so that VMs can connect to it. This NAD is created in the namespace of the VM.

If the NAD is created in the default namespace, the configuration will be available to all namespaces.

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: ovn-localnet
namespace: my-vm-namespace
spec:
config: '{
"cniVersion": "0.4.0",
"name": "ovn-localnet", (1)
"type": "ovn-k8s-cni-overlay", (2)
"topology": "localnet", (3)
"netAttachDefName": "my-vm-namespace/br0-network" (4)
}'

1	Name of the network (ovn-localnet)
2	Type of the network (ovn-k8s-cni-overlay)
3	Topology of the network (localnet)
4	Reference to the localnet mapping name defined in the NNCP (my-vm-namespace/br0-network)

Part 3: Connecting the VM

Finally, we have our "socket" (the NAD). Let’s plug the VM in. In your VirtualMachine manifest, you add the network to the spec.

This can be done via the Web Console or via the CLI.

Via the Web Console:

In the OpenShift Web Console, navigate to Virtualization → VirtualMachines.
Click on the VM you want to connect to the network.
Select the "Configuration" tab
Select "Network"

Click "Add network interface"
In the Popup enter a name for the interface
Model keep as "virtio"
On the "Network" select the NAD you created in the previous step.

Click "Save"

The VM will now have a Pending state. This is because the VM is waiting for a migration to activate the new network interface.

In the action menu you can select "Migrate" > "Compute" to start the migration process. After a while the new NIC will be active.

Via the CLI:

Here is a VM connected to the Linux Bridge NAD we created in step 1:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: helloworld-vm
namespace: my-vm-namespace
spec:
template:
spec:
domain:
devices:
interfaces:
- name: default
masquerade: {} (1)
- name: br0-network
bridge: {} (2)
networks:
- name: default
pod: {}
- name: br0-network (3)
multus:
networkName: br0-network (4)

1	The standard Pod Network
2	Our new connection
3	The NAD name
4	The NAD name

GitOps Catalog

Mon, 22 Dec 2025 00:00:00 +0000

The GitOps Catalog page provides an interactive visualization of all available ArgoCD applications from the openshift-clusterconfig-gitops repository. Check out the page GitOps Catalog for more details.

The Guide to OpenBao - Authentication Methods - Part 7

Mon, 09 Mar 2026 00:00:00 +0000

With OpenBao deployed and running, the next critical step is configuring authentication. Ultimately, you want to limit access to only authorised people. This article covers two common authentication methods: Kubernetes for pods and LDAP for enterprise directories (in a simplified example). There are many more methods, but we cannot cover them all in this article.

Introduction

Authentication in OpenBao verifies the identity of clients before granting access to secrets. OpenBao supports multiple authentication methods, each suited for different use cases. Among others it supports:

Method	Best For	Key Features
Kubernetes	Pods running in K8s/OpenShift	Service account tokens, automatic
OIDC	Human users, SSO	OpenShift OAuth, Keycloak, Azure AD
LDAP	Enterprise directories	Active Directory, OpenLDAP
AppRole	CI/CD, automation	Role ID + Secret ID
Token	Direct access, bootstrap	Simple but less secure

Method

Best For

Key Features

Kubernetes

Pods running in K8s/OpenShift

Service account tokens, automatic

OIDC

Human users, SSO

OpenShift OAuth, Keycloak, Azure AD

LDAP

Enterprise directories

Active Directory, OpenLDAP

AppRole

CI/CD, automation

Role ID + Secret ID

Token

Direct access, bootstrap

Simple but less secure

The full documentation of the authentication methods and the list of available methods (for example Radius) can be found here: OpenBao Authentication Methods.

Authentication Workflow

The authentication workflow in OpenBao from a user perspective is as follows:

A client wants to authenticate to OpenBao and provides credentials.
OpenBao validates the credentials against the authentication method. For example: LDAP
If the authentication method (e.g. LDAP) is successful, it will return the required information about the client.
OpenBao maps this result to policies that are mapped to the authentication method.
OpenBao generates a token that is associated with the policies and returns it to the client.
The client can then use this token for further operations.

Prerequisites for Authentication Methods

Before configuring authentication:

OpenBao is deployed and unsealed (Parts 2-6)
You have admin access to OpenBao (root token)
Access to the required authentication backend:
- For Kubernetes auth: Access to the OpenShift cluster
- For LDAP: Access to the LDAP server
- etc.

Set up your environment:

You can set up your environment by using either the CLI or the UI.

Using the CLI

You can use the CLI to interact with OpenBao. In this example we will forward the port locally to the OpenBao server and set the environment variable BAO_ADDR to the local address. With boa login and the root token you can authenticate against OpenBao with full access.

# Port forward (if needed)
oc port-forward svc/openbao 8200:8200 -n openbao &
# Set environment
export BAO_ADDR='http://127.0.0.1:8200'
# You may need the SSL CA too.
# export BAO_CACERT="$PWD/openbao-ca.crt"
# Login with root token
bao login
# List all authentication methods
bao auth list

The last command returns a list of the available authentication methods. Currently there is only one authentication method enabled: token.

Path Type Accessor Description Version
---- ---- -------- ----------- -------
token/ token auth_token_1020fa7b token based credentials n/a

Using the UI

The UI, if enabled, is accessible via the OpenShift Route (if running on OpenShift and if it was created in Part 3). The password/token is the root token from the initialisation.

Figure 1. OpenBao UI Login Form

The UI has the advantage that it offers more visibility into the configuration and possibilities. For example, at the beginning there is one authentication method token enabled. This is required for the root token authentication.

Figure 2. OpenBao UI Authentication Methods

We can enable the other authentication methods by clicking on the Enable new method link to see what options are available.

Figure 3. OpenBao UI Configuring New Authentication Method

Understanding Policies

Before you can actually use an authentication method, it is important to understand what policies do. A policy is a way to declaratively define what (which paths) authenticated users can access or not. All paths are denied by default. A policy is mapped to an authentication method. For example, when we look at LDAP, we could configure something like this:

Every member of group "dev" is mapped to a policy named "dev-policy". The policy then allows the user to read the secrets under the path "secret/data/dev/*".

Basic Policy Structure

A policy is a set of path rules. Each rule says: “for this path (or path prefix), the bearer of this policy may use these capabilities.” Paths are tied to OpenBao’s internal API: for example, secrets in the KV v2 engine live under secret/data/<mount-path>/<key>, so a path like secret/data/myapp/* means “any key under the myapp prefix in the KV v2 store mounted at secret/.”

The example below does two things that are typical for an application policy:

Secrets access: It allows read and list on secret/data/myapp/*.
- read lets the client fetch the value of a secret at a path (e.g. secret/data/myapp/database).
- list is required to list keys under a path (e.g. to discover myapp/database, myapp/api-key); without it, the client would need to know every path in advance.
Token renewal: It allows update on auth/token/renew-self. Tokens often have a limited lifetime; this path lets the holder extend their own token without needing the root token or extra permissions. Including it avoids applications losing access when the token expires.

Policies can be written in JSON or HCL.

# Example policy: myapp-read-policy.hcl
# Allow reading secrets from specific path (KV v2 engine at mount "secret/")
path "secret/data/myapp/*" {
capabilities = ["read", "list"]
}
# Allow renewing own token so the app can extend its token before expiry
path "auth/token/renew-self" {
capabilities = ["update"]
}

The path secret/data/myapp/* assumes you have a KV v2 secrets engine mounted at secret/ and will store application secrets under keys like myapp/database, myapp/api-key, etc. Adjust the mount path and prefix to match your setup.

The use of globs (*) may result in surprising or unexpected behaviour. Use them with caution.

Policy Capabilities

Each path defined in a policy must have at least one capability. This provides control over the allowed or denied operations a user may perform. There are several capabilities available:

create - Create new data at the given path
read - Read data
update - Update existing data
patch - Patch existing data (Partial update)
delete - Delete data
list - List keys at a path
scan - Scan or browse the path for keys
sudo - Allows access to paths that are root-protected.
deny - Explicitly deny (overrides other grants)

Default Policies

When OpenBao is deployed the first time, there are no additional authentication methods enabled, except for token. This is required for the root access itself. To allow the administrator to access using the root token, there is one default policy called root. This policy is somewhat of a catch-all policy; it allows the administrator to access everything.

In addition to the root policy, there is one other default policy called default. This policy is used for all authenticated users. It allows them, for example, to look up their own properties and to renew or revoke their own token.

Let’s look at the default policies in the CLI:

A quick reminder of how to log in to OpenBao using the CLI when running it against Kubernetes: first open a port-forward, set the variable BAO_ADDR to the local address; if you use HTTPS you may need to set the SSL CA certificate, then log in with the root token.

You need to have the CA certificate in the current directory, which you might fetch with:

oc get secret openbao-ca-secret -n openbao -o jsonpath='{.data.ca\.crt}' | base64 -d > openbao-ca.crt

Port forward, set the environment variables and login with the root token:

# Port forward (if needed)
oc port-forward svc/openbao 8200:8200 -n openbao &
# Set environment
export BAO_ADDR='https://127.0.0.1:8200'
# You may need the SSL CA too.
export BAO_CACERT="$PWD/openbao-ca.crt"
# Login with root token
bao login

Now to list and fetch the current policies you can use the following commands:

# List policies
bao policy list
# Returns something like this:
#default
#root
# Read a policy
bao policy read default

The second command returns the policy content:

# Allow tokens to look up their own properties
path "auth/token/lookup-self" {
capabilities = ["read"]
}
# Allow tokens to renew themselves
path "auth/token/renew-self" {
capabilities = ["update"]
}
# Allow tokens to revoke themselves
path "auth/token/revoke-self" {
capabilities = ["update"]
}
# Allow a token to look up its own capabilities on a path
path "sys/capabilities-self" {
[...]

Again you will see the same policies in the UI. You can also manage them or create new ones using the UI.

Figure 4. OpenBao UI Listing Policies

Creating Policies using the CLI

Creating a policy, for example using the CLI, is straightforward. You can create a policy from a file or inline. The syntax is either JSON or HCL. The example below creates a policy called "myapp-read" from a file called "myapp-read-policy.hcl". The policy allows reading and listing secrets under the path "secret/data/myapp/*".

# Create a policy from file
bao policy write myapp-read myapp-read-policy.hcl
# Or inline
bao policy write myapp-read - <<EOF
path "secret/data/myapp/*" {
capabilities = ["read", "list"]
}
EOF
# List policies
bao policy list
# Read a policy
bao policy read myapp-read

Kubernetes Authentication Method

As we work a lot with OpenShift/Kubernetes, this method of authentication will be the first to try. The Kubernetes auth method can be used to authenticate against OpenBao using Kubernetes Service Account tokens. It is essential for pods to access secrets.

Let us try a simple example.

Step 1: Enable Kubernetes Auth

Assuming you are still logged into OpenBao, you can enable the kubernetes auth method with the following command:

bao auth enable --description="Authentication method for Kubernetes/OpenShift cluster" kubernetes

This simply enabled the authentication method and set a description. However, we need to configure it further.

Step 2: Configure Kubernetes Auth

To configure the kubernetes auth method, you need to get the Kubernetes API server address. You can get it with the following command (assuming you are logged into the OpenShift cluster):

The command oc can be replaced by kubectl if you are not using an OpenShift cluster.

# Get the Kubernetes API server address
KUBERNETES_HOST=$(oc whoami --show-server) (1)
# Configure the auth method
bao write auth/kubernetes/config \ (2)
kubernetes_host="$KUBERNETES_HOST"

1	Get the Kubernetes API server address. This will use something like https://api.cluster.example.com:6443
2	Configure the auth method with the Kubernetes API server address.

When running inside Kubernetes, OpenBao automatically uses the pod’s service account to communicate with the Kubernetes API.

Step 3: Create a Policy for the Application

Now we need to associate a policy to the authentication method. We could also assign the default policy but we want to provide access to the path secret/data/expense/database and secret/metadata/expense/config for the application.

# Create policy for an example application
bao policy write expense-app - <<EOF
# Read database credentials
path "secret/data/expense/database" {
capabilities = ["read"]
}
# Read application config
path "secret/data/expense/config" {
capabilities = ["read", "list"]
}
# Allow token renewal
path "auth/token/renew-self" {
capabilities = ["update"]
}
EOF

Step 4: Create a Kubernetes Auth Role

As the next step we map the policy to the authentication method. We will further limit the access to the namespace expense and the service account expense-app.

# Create a role that maps Kubernetes service accounts to OpenBao policies
bao write auth/kubernetes/role/expense-app \
bound_service_account_names=expense-app \ (1)
bound_service_account_namespaces=expense \ (2)
policies=expense-app \ (3)
ttl=1h \ (4)
max_ttl=24h (5)

1	`bound_service_account_names`: K8s service account name(s)
2	`bound_service_account_namespaces`: K8s namespace(s)
3	`policies`: OpenBao policies to attach
4	`ttl`: Token time-to-live
5	`max_ttl`: Maximum token lifetime

All of the configuration above, created using the CLI, can also be done using the UI of OpenBao.

Ensure the KV v2 secrets engine is mounted at `secret/`

We will discuss secret engines in more detail in the next part of the guide.

The policy and demo commands in this guide use the path secret/data/expense/database, which assumes a KV v2 secrets engine mounted at secret/. To check and enable it:

Check existing mounts: List enabled secrets engines and look for secret/ with type kv (version 2):
```
# List all secrets engine mounts (requires a token with read on sys/mounts)
bao secrets list
```
Look for an entry like secret/ with type kv or kv-v2. If you see secret/ and it is KV v2, you can skip to Step 5.
Enable KV v2 at secret/ if missing: If secret/ is not listed, or you want to use a fresh mount, enable the KV v2 engine at the path secret:
```
# Enable KV v2 secrets engine at path "secret" (requires root or policy with sys/mounts capability)
bao secrets enable -path=secret -version=2 kv
```
If secret/ already exists as KV v1, you cannot change it in place to v2; use a different path (e.g. secretv2/) and update the policy and demo paths accordingly (secretv2/data/expense/database, etc.).

Step 5: Store the database secret in OpenBao (one-time)

Before the application can read it, store the database password (and any other keys) at the path the policy allows. Run this once from a machine that has OpenBao access (e.g. bao CLI and a token).

You must use a token that has create and update capability on the KV path. The expense-app policy is read-only (for the application). Use the root token from OpenBao initialisation, or log in with bao login and use a token that has a policy granting create and update on secret/data/expense/*. If you use a read-only token (e.g. one that only has the expense-app policy), you will get a 403 "preflight capability check returned 403".

# Store the database secret at secret/data/expense/database (KV v2 path)
bao kv put secret/expense/database password="my-super-secret-db-password" username="expense_db_user"

This creates (or overwrites) the secret at secret/data/expense/database. The policy grants the expense-app role read on this path only; the application cannot create or update it.

Step 6: Create the expense namespace and demo application

Apply the following manifests to create the namespace, service account, optional long-lived token (Kubernetes 1.24+), and a demo deployment that authenticates to OpenBao and fetches the database secret. Adjust the OpenBao URL if your OpenBao is in another namespace or uses HTTPS (e.g. https://openbao.openbao.svc:8200).

Remember: We have created a policy that allows the service account with the name expense-app in the namespace expense to read the secret at secret/data/expense/database. If you want to use different paths or namespaces, you need to adjust the policy accordingly.

# full-demo-expense-app.yaml
# 1. Namespace
apiVersion: v1
kind: Namespace (1)
metadata:
name: expense
---
# 2. Service account used by the demo app to authenticate to OpenBao
apiVersion: v1
kind: ServiceAccount (2)
metadata:
name: expense-app
namespace: expense
---
# 3. Long-lived token for Kubernetes 1.24+ (optional; allows pods to use the SA token)
apiVersion: v1
kind: Secret (3)
metadata:
name: expense-app-token
namespace: expense
annotations:
kubernetes.io/service-account.name: expense-app
type: kubernetes.io/service-account-token
---
# 4. Demo deployment: authenticates to OpenBao and fetches secret/data/expense/database
apiVersion: apps/v1
kind: Deployment
metadata:
name: expense-demo
namespace: expense
labels:
app: expense-demo
spec:
replicas: 1
selector:
matchLabels:
app: expense-demo
template:
metadata:
labels:
app: expense-demo
spec:
serviceAccountName: expense-app
containers:
- name: demo
image: registry.access.redhat.com/ubi9/ubi-minimal:latest
command:
- /bin/sh
- -c
- |
echo "=== Installing curl ==="
microdnf install -y curl 2>/dev/null || true
echo "=== Fetching database secret from OpenBao ==="
JWT=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
OPENBAO_URL="${OPENBAO_URL:-http://openbao.openbao:8200}"
RESP=$(curl -s -k -X POST -H "Content-Type: application/json" \ (4)
-d "{\"jwt\": \"$JWT\", \"role\": \"expense-app\"}" \
"$OPENBAO_URL/v1/auth/kubernetes/login")
TOKEN=$(echo "$RESP" | sed -n 's/.*"client_token":"\([^"]*\)".*/\1/p')
if [ -z "$TOKEN" ]; then
echo "ERROR: Failed to get OpenBao token. Check OpenBao URL ($OPENBAO_URL), role expense-app, and network."
echo "Response: $RESP"
exit 1
fi
echo "OpenBao token obtained. Reading secret/data/expense/database ..."
SECRET=$(curl -s -k -H "X-Vault-Token: $TOKEN" "$OPENBAO_URL/v1/secret/data/expense/database") (5)
echo "$SECRET"
PASSWORD=$(echo "$SECRET" | sed -n 's/.*"password":"\([^"]*\)".*/\1/p')
echo "Database password (data.data.password): ${PASSWORD:-<not set>}"
echo "=== Demo complete. Pod stays running; exec in to run more curl commands. ==="
exec sleep infinity
env:
# Override if OpenBao is in another namespace or uses HTTPS
- name: OPENBAO_URL
value: "https://openbao.openbao:8200"
resources:
requests:
memory: "64Mi"
cpu: "10m"

1	Create the namespace
2	Create the service account
3	Create the secret (long-lived token for Kubernetes 1.24+)
4	Authenticate to OpenBao and get a client token (using the Kubernetes JWT)
5	Fetch the database secret

The examples above use curl -k to bypass certificate verification. For production, you should adjust this and mount the CA certificate in the pod (see the note earlier in this section on using the openbao-ca Secret).

Apply the manifests and wait for the pod to be ready:

oc apply -f full-demo-expense-app.yaml
# Wait for the demo pod to be running
oc -n expense rollout status deployment/expense-demo
# View the log: you should see the OpenBao token response and the fetched secret (including the password)
oc -n expense logs -l app=expense-demo -f

The log output shows the application authenticating with the Kubernetes JWT, receiving an OpenBao token, and reading the secret at secret/data/expense/database. The field data.data.password is the database password you stored in Step 5.

Step 7: Test Kubernetes authentication manually

From any pod that uses the same service account (e.g. the demo pod above), you can run the same steps by hand:

# From within a pod using the service account (e.g. oc exec -it deployment/expense-demo -n expense -- /bin/sh)
JWT=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
# 1. Authenticate to OpenBao and get a client token (sed works on UBI minimal; use jq -r .auth.client_token if jq is available)
LOGIN=$(curl -s -k --request POST \
--data "{\"jwt\": \"$JWT\", \"role\": \"expense-app\"}" \
https://openbao.openbao:8200/v1/auth/kubernetes/login)
TOKEN=$(echo "$LOGIN" | sed -n 's/.*"client_token":"\([^"]*\)".*/\1/p')
# 2. Fetch the database secret (same path the policy allows)
curl -s -k -H "X-Vault-Token: $TOKEN" https://openbao.openbao:8200/v1/secret/data/expense/database

Summary: Kubernetes auth method

A lot of steps were involved to configure the Kubernetes auth method. Let us summarize them:

In OpenBao: Enabled the Kubernetes auth method, configured it with the cluster API server address, created the expense-app policy (read on secret/data/expense/database and secret/data/expense/config, update on auth/token/renew-self), and created the expense-app role that binds the Kubernetes service account expense-app in namespace expense to that policy.
Secrets engine: Ensured the KV v2 engine is mounted at secret/ and stored the database secret at secret/expense/database (one-time, using the root token).
In the cluster: Created the expense namespace, the expense-app service account, an optional long-lived token Secret (for Kubernetes 1.24+), and the expense-demo deployment that uses that service account to authenticate to OpenBao and fetch the secret. For HTTPS, created the openbao-ca Secret in expense with the OpenBao server CA.
Verification: Applied the manifests, confirmed the demo pod starts and logs show a successful login and the fetched secret (including the database password). Optionally ran the same login and read steps manually via oc exec into the pod.

LDAP Authentication Method

LDAP auth integrates with enterprise directories like Active Directory.

This is a very simplified example. As a prerequisite you need to have an LDAP server and a service account with the necessary permissions to read the users and groups. This is not part of this guide.

Step 1: Enable LDAP Auth

To enable LDAP authentication, you need to use the following command:

bao auth enable ldap

Step 2: Configure LDAP (Active Directory Example)

In this step you need to configure the LDAP server address, the user and group attributes, the bind DN and the bind password. The example below configures the LDAP server address to ldaps://ad.example.com:636, the user attribute to sAMAccountName, the user DN to OU=Users,DC=example,DC=com, the group DN to OU=Groups,DC=example,DC=com, the group attribute to cn, the group filter to (&(objectClass=group)(member:1.2.840.113556.1.4.1941:={{.UserDN}})), the bind DN to CN=openbao-svc,OU=ServiceAccounts,DC=example,DC=com and the bind password to service-account-password.

bao write auth/ldap/config \
url="ldaps://ad.example.com:636" \
userattr="sAMAccountName" \
userdn="OU=Users,DC=example,DC=com" \
groupdn="OU=Groups,DC=example,DC=com" \
groupattr="cn" \
groupfilter="(&(objectClass=group)(member:1.2.840.113556.1.4.1941:={{.UserDN}}))" \
binddn="CN=openbao-svc,OU=ServiceAccounts,DC=example,DC=com" \
bindpass="service-account-password" \
certificate=@/path/to/ldap-ca.pem \
insecure_tls=false \
starttls=false

Step 3: Create the policies used by the group mapping

Before mapping LDAP groups to policy names, create those policies. Below are example definitions: admin-policy grants read/list/update on secrets (e.g. for operators), and user-policy grants read-only access to a limited path.

admin-policy — for members of openbao-admins (e.g. operators who may read and update secrets)

user-policy — for members of openbao-users (e.g. developers with read-only access to a subset of secrets)

Create them with the CLI (run before the group mapping in Step 4):

# Create admin-policy (e.g. from file admin-policy.hcl)
bao policy write admin-policy - <<EOF
path "secret/data/*" {
capabilities = ["read", "list", "update", "create", "delete"]
}
path "secret/metadata/*" {
capabilities = ["read", "list"]
}
path "auth/token/renew-self" {
capabilities = ["update"]
}
EOF
# Create user-policy
bao policy write user-policy - <<EOF
path "secret/data/myapp/*" {
capabilities = ["read", "list"]
}
path "auth/token/renew-self" {
capabilities = ["update"]
}
EOF

Adjust paths (e.g. secret/data/myapp/) to match your KV v2 mount and the paths you want each role to access. The default policy is built-in and grants token lookup/renew/revoke-self. Attaching it in addition to admin-policy or user-policy is a typical pattern.

Step 4: Map LDAP groups to policies

Now map the LDAP groups to the policy names. The example below maps the LDAP group openbao-admins to the policies admin-policy and default and the LDAP group openbao-users to the policies user-policy and default.

# Map an LDAP group to policies
bao write auth/ldap/groups/openbao-admins \
policies="admin-policy,default"
bao write auth/ldap/groups/openbao-users \
policies="user-policy,default"

Step 5: Test LDAP authentication

Assuming you have a user jdoe in the LDAP group openbao-users, you can test the LDAP authentication with the following commands:

# Login with LDAP credentials
bao login -method=ldap username=jdoe
# Or via API
curl --request POST \
--data '{"password": "user-password"}' \
$BAO_ADDR/v1/auth/ldap/login/jdoe

What jdoe can and cannot do (user-policy)

jdoe holds user-policy (and default), so their token has only the capabilities defined there.

What jdoe can do:

Get a secret under secret/data/myapp/*: after logging in, jdoe can read and list secrets in that path. For example:

# After bao login -method=ldap username=jdoe (and entering the password)
bao kv get secret/myapp/database
# Or via API (use the client_token from the login response)
curl -s -H "X-Vault-Token: $TOKEN" $BAO_ADDR/v1/secret/data/myapp/database

List keys under secret/data/myapp/ to discover available secrets.
Renew or revoke their own token (from the default policy) and look up their own token properties.

What jdoe cannot do:

Read secrets outside myapp — e.g. secret/data/expense/database or secret/data/other-app/* returns permission denied.
Create, update, or delete any secret — user-policy grants only read and list on secret/data/myapp/*, so write operations are denied.
Access admin or system paths — no access to sys/, other auth methods, or policies. Only the paths explicitly granted in *user-policy and default are allowed.

Common Practices for Authentication

Use the principle of least privilege - Do not grant more permissions than necessary.

# Bad: Too broad
path "secret/*" {
capabilities = ["read", "list"]
}
# Good: Specific paths
path "secret/data/myapp/database" {
capabilities = ["read"]
}

Set Appropriate Token TTLs - Set the appropriate TTLs for different use cases.

# Short-lived for automated systems
bao write auth/kubernetes/role/batch-job \
ttl=5m \
max_ttl=15m
# Longer for interactive users
bao write auth/oidc/role/human-user \
ttl=8h \
max_ttl=24h

Enable Audit Logging - Enable audit logging to track authentication attempts and other activities.

# Enable file audit
bao audit enable file file_path=/var/log/openbao/audit.log
# All authentication attempts are logged

Regularly Rotate Credentials - Rotate the credentials periodically to reduce the risk of compromise.

# Rotate AppRole Secret IDs periodically
bao write -f auth/approle/role/cicd-role/secret-id
# Revoke old tokens
bao token revoke <old-token>

Common Authentication Patterns

Pattern 1: Application Pods

Pod → Service Account Token → Kubernetes Auth → Policy → Secret

Pattern 2: Human Users

User → LDAP/OIDC Login → Group Mapping → Policy → Secret

Pattern 3: CI/CD Pipeline

Pipeline → AppRole (Role ID + Secret ID) → Policy → Secret

What is Coming Next?

In Part 8, we will explore secrets engines, like:

KV (Key-Value) for static secrets

In upcoming parts, I will try to cover common practices for OpenBao operation and maintenance. If there is time and an opportunity to test other authentication methods, I will do so.

Conclusion

Proper authentication configuration is crucial for OpenBao security. Key takeaways:

Use Kubernetes auth for pods
Use LDAP/OIDC for human users (SSO)
Use AppRole for automation
Create specific policies with least privilege
Set appropriate TTLs for different use cases
Enable audit logging for compliance

Resources

The Guide to OpenBao - Initialisation, Unsealing, and Auto-Unseal - Part 6

Thu, 05 Mar 2026 00:00:00 +0000

After deploying OpenBao via GitOps (Part 5), OpenBao must be initialised and then unsealed before it becomes functional. You usually do not want to do this unsealing manually, since this is not scalable especially in bigger, productive environments. This article explains how to handle initialisation and unsealing, and possible options to configure an auto-unseal process so that OpenBao unseals itself on every restart without manual key entry.

What is happening?

OpenBao starts in a sealed state. You must:

Initialise (once): run bao operator init to generate unseal keys (or recovery keys when using auto-unseal).
Unseal (after each restart): provide a threshold of unseal keys so each node can decrypt the root key. In the previous articles, we used a threshold of 3 (out of 5). This means that you need to run bao operator unseal three times with different keys on every node after every start before the service becomes available.

Unseal workflow (assuming a threshold of 3 and 5 nodes and initialisation already happened):

---
title: "Unseal Workflow"
config:
theme: 'dark'
---
flowchart LR
A["openbao-0 starts → 3× unseal using different keys"]
A --> B["openbao-1 starts → 3× unseal using different keys"]
B --> C["openbao-2 starts → 3× unseal using different keys"]
C --> D[OpenBao is ready]

This is a chicken-egg problem: you need to start somewhere, but you cannot start without the keys.

Manual unsealing does not scale very well and for production, auto-unseal is recommended.

This article covers:

Practical approaches to initialisation and unsealing (manual, Jobs, init containers).
Potential auto-unseal options: with AWS KMS, static key, and transit seal.

Handling Initialisation and Unsealing

The challenge with GitOps is that initialisation and unsealing are one-off or stateful operations. Below are several approaches. The problem is always the same: where do you start? You must have the unseal keys available to start the process. Do you keep them locally, do you keep them in a Kubernetes Secret, do you use an external Key Management System (KMS)? The most robust for production in my opinion is auto-unseal using a KMS (see [_approach_4_auto_unseal_with_kms_recommended_for_production] and the following sections). It uses an external Key Management System (KMS) to store the keys. This way, the keys are not stored in the cluster and are not exposed to the risk of being compromised.

Let’s have a look at the different options.

Approach 1: Manual Initialisation (simplest)

While manual initialisation and unsealing is the simplest approach, it is not recommended for production. Still, I would like to mention it here for completeness and at least provide a simple for loop to unseal the OpenBao service (on Kubernetes/OpenShift).

Manual unsealing is still a valid option when you have a small cluster and assume that it is not required often.

After Argo CD deploys OpenBao, the first pod, openbao-0, will not become ready until it is initialised and unsealed. Once done, the next pod, openbao-1, will try to start and wait until it is unsealed, and then the third pod, and so on.

The following, simplified script will go through the pods and unseal them one by one. It initialises the first pod and stores the unseal keys in the file openbao-init.json locally on your machine. From there it takes three different keys and unseals the OpenBao service. Since it takes a while until other pods are pulling the image and starting, we will use a sleep timer of 30 seconds between each pod. This may or may not be enough depending on your network and cluster speed. As said, this is a very simplified script:

The CLI tool oc can be replaced by kubectl in case you are using a different cluster. In addition, you will need the jq command line tool to parse the JSON file.

If your OpenBao was initialised previously, you can remove the first three commands and start with the unseal commands.

# Initialise the first pod, assuming the Pod openbao-0 is already running.
# You can skip this if OpenBao was initialised previously.
echo "Initialising OpenBao on openbao-0 (keys will be saved to openbao-init.json)..."
oc exec -it openbao-0 -n openbao -- bao operator init \
-key-shares=5 -key-threshold=3 -format=json > openbao-init.json (1)
echo "Init complete. Unseal keys and root token are in openbao-init.json."
# Unseal each pod (three times each with different keys)
echo "Unsealing pods openbao-0, openbao-1, openbao-2 (3 key steps each)..."
for i in 0 1 2; do
sleep_timer=30
SLEEPER_TMP=1
echo "Unsealing openbao-$i..."
for j in 1 2 3; do (2)
KEY=$(cat openbao-init.json | jq -r ".unseal_keys_b64[$((j-1))]")
oc exec -it openbao-$i -n openbao -- bao operator unseal $KEY
done
echo "openbao-$i unsealed. Waiting ${sleep_timer}s for next pod to be ready..."
while [[ $SLEEPER_TMP -le "$sleep_timer" ]]; do (3)
if (( SLEEPER_TMP % 10 == 0 )); then
echo -n "$SLEEPER_TMP"
else
echo -n "."
fi
sleep 1
SLEEPER_TMP=$((SLEEPER_TMP + 1))
done
echo ""
done
echo "All pods unsealed. Store init data securely (e.g. in a password manager)."
# Store init data securely (e.g. in a password manager)

1	Initialise the first pod, assuming the Pod openbao-0 is already running. This will store the keys (including the root token) in the file openbao-init.json.
2	Unseal each pod (three times each with different keys). The keys are taken from the openbao-init.json file.
3	Wait 30 seconds between each pod to give the other pods time to start and pull the image … super-fancy sleep timer.

The unseal keys as well as the root token are stored in the openbao-init.json file. This file is stored locally on your machine. Make sure to store it securely and do not share it with anyone.

Approach 2: Kubernetes Job for initialisation

A Job can run after OpenBao is deployed, initialise it if needed, and store the init output (unseal keys and root token) in a Kubernetes Secret. You can then unseal manually or with a second Job that reads from that Secret. This section assumes the init output is stored in a Kubernetes Secret and explains how to create that Secret from the Job and how to use it.

How the Secret is created

The init Job runs bao operator init and then creates the Secret using oc (or kubectl). The Secret name and key used here are openbao-init-data and init.json (the file contains the JSON output of bao operator init, including unseal_keys_b64 and root_token). One advantage of this approach is that you can integrate it into your GitOps pipeline. However, we can argue whether it is a good idea to store the unseal keys and the root token as a Secret in the same cluster where OpenBao is running.

Example structure of init.json:

{
"unseal_keys_b64": [
"key1",
"key2",
"key3",
"key4",
"key5"
],
"unseal_keys_hex": [
"hex1",
"hex2",
"hex3",
"hex4",
"hex5"
],
"unseal_shares": 5,
"unseal_threshold": 3,
"recovery_keys_b64": null,
"recovery_keys_hex": null,
"recovery_keys_shares": 0,
"recovery_keys_threshold": 0,
"root_token": "root.token"
}

You need at least unseal_threshold keys (e.g. 3) from unseal_keys_b64 to unseal each node. Store this file securely; anyone with access can unseal OpenBao and the root_token has full admin access.

RBAC Configuration

The Job’s service account (openbao-init) must be allowed to create (and optionally get/update) Secrets in the openbao namespace. Create a Role and RoleBinding:

apiVersion: v1
kind: ServiceAccount
metadata:
name: openbao-init (1)
namespace: openbao
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: openbao-init-secret-writer
namespace: openbao
rules: (2)
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create", "get", "update", "patch"]
- apiGroups: [""] (3)
resources: ["pods"]
verbs: ["get", "list"]
- apiGroups: [""] (4)
resources: ["pods/exec"]
verbs: ["create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding (5)
metadata:
name: openbao-init-secret-writer
namespace: openbao
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: openbao-init-secret-writer
subjects:
- kind: ServiceAccount
name: openbao-init
namespace: openbao

1	The ServiceAccount is used by the Job(s) to create the Secret and unseal the OpenBao pods.
2	The Role is used by the Job to create the Secret. The rules might be extended to include the permission to execute into the pods to unseal the OpenBao service (see below).
3	Permissions to get and list the Pods, required for the unseal process later
4	Permissions to execute into the Pods, required for the unseal process later
5	The RoleBinding is used by the Job to create the Secret.

The Job must also be able to reach the OpenBao API (e.g. openbao.openbao.svc:8200). This is usually possible when running in the same namespace as the OpenBao service.

The example below shows a working Job manifest that will perform the initialisation and the creation of the Secret in the openbao namespace. The manifest is already prepared for Argo CD with useful annotations. It runs an init container to initialise the OpenBao service and a main container to create the Secret. The init container writes (using the OpenBao CLI) the init.json file to a shared volume and the main container (using the kubectl command) creates the Secret from that file. In addition, the Secret containing the CA certificate is mounted as a volume and the BAO_CACERT environment variable is set to the path of the CA certificate.

Init Job that creates the Secret

apiVersion: batch/v1
kind: Job
metadata:
name: openbao-init
namespace: openbao
annotations: (1)
argocd.argoproj.io/sync-wave: "30"
argocd.argoproj.io/hook: PostSync
argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
template:
spec:
serviceAccountName: openbao-init (2)
initContainers: (3)
- name: openbao-init
image: ghcr.io/openbao/openbao:latest
command: (4)
- /bin/sh
- -c
- |
if [ -f /etc/openbao-ca/ca.crt ]; then
export BAO_CACERT=/etc/openbao-ca/ca.crt
export BAO_ADDR=https://openbao:8200
echo "Using TLS CA from /etc/openbao-ca/ca.crt"
else
export BAO_ADDR=http://openbao:8200
echo "Using plain HTTP"
fi
until bao status 2>&1 | grep -q "Initialized"; do
echo "Waiting for OpenBao..."
sleep 5
done
if bao status | grep -q "Initialized.*false"; then
echo "Initialising OpenBao..."
bao operator init -key-shares=5 -key-threshold=3 \
-format=json > /shared/init.json
echo "Init complete; init.json written to shared volume."
else
echo "OpenBao already initialised (skipping init)."
fi
volumeMounts:
- name: openbao-ca
mountPath: /etc/openbao-ca
readOnly: true
- name: shared
mountPath: /shared
containers: (5)
- name: create-secret
image: bitnami/kubectl:latest
command:
- /bin/sh
- -c
- |
if [ -f /shared/init.json ]; then
echo "Creating Secret openbao-init-data from init.json..."
kubectl create secret generic openbao-init-data \
--from-file=init.json=/shared/init.json \
-n openbao --dry-run=client -o yaml | kubectl apply -f -
echo "Initialisation complete!"
else
echo "No init.json (OpenBao already initialised or init skipped)."
fi
volumeMounts:
- name: shared
mountPath: /shared
readOnly: true
volumes:
- name: openbao-ca
secret:
secretName: openbao-ca-secret
optional: true
- name: shared (6)
emptyDir: {}
restartPolicy: Never
backoffLimit: 3

1	The annotations are used by Argo CD to trigger the Job after the OpenBao deployment.
2	The ServiceAccount is used by the Job to initialise the OpenBao service.
3	The init container runs the OpenBao CLI (wait for API, then operator init) and writes init.json to a shared volume.
4	The command is used to initialise the OpenBao service.
5	The main container runs kubectl to create the Secret from that file. This avoids needing a single image that bundles both `bao` and `kubectl`.
6	The shared volume is used to store the init.json file.

As a result the secret openbao-init-data is created in the openbao namespace. It contains the unseal keys and the root token.

Using the Secret in a Job (unseal)

A second Job can use the Secret openbao-init-data that was created by the init Job to unseal each OpenBao pod. A lot is happening here:

The init container extracts the unseal keys from the Secret using jq and writes them to a shared volume.
The main container is reading the unseal keys from the shared volume and unseals the first pod
Then the process is waiting for 30 seconds, to give the next pod time to pull the image and start
The process is repeated for the next pod and so on until all pods are unsealed.

First of all, the unseal Job needs permission to exec into the OpenBao pods. Add a Role that grants the Job’s service account create on pods/exec in the openbao namespace. To make it easier, we can extend the Role openbao-init-secret-writer that was created in the init Job.

These rules were already applied with the configuration for the init Job, but we will repeat them here for completeness.

rules: (1)
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"]

1	The role `openbao-init-secret-writer` must be extended to include the permission to execute into the pods to unseal the OpenBao service.

apiVersion: batch/v1
kind: Job
metadata:
name: openbao-unseal
namespace: openbao
annotations:
argocd.argoproj.io/sync-wave: "35"
argocd.argoproj.io/hook: PostSync
argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
template:
spec:
serviceAccountName: openbao-init (1)
initContainers:
- name: extract-keys
image: quay.io/codefreshplugins/curl-jq
command:
- /bin/sh
- -c
- |
if [ ! -f /secrets/init.json ]; then
echo "Secret not found; run init Job first."
exit 1
fi
for i in 0 1 2; do
jq -r ".unseal_keys_b64[$i]" /secrets/init.json | tr -d '\n' > "/shared/key$i" (2)
done
echo "Unseal keys extracted to shared volume."
volumeMounts:
- name: init-secret
mountPath: /secrets
readOnly: true
- name: shared
mountPath: /shared
containers: (3)
- name: unseal
image: bitnami/kubectl:latest
command: (4)
- /bin/sh
- -c
- |
if [ -f /etc/openbao-ca/ca.crt ]; then
export BAO_CACERT=/etc/openbao-ca/ca.crt
export BAO_ADDR=https://openbao:8200
echo "Using TLS CA from /etc/openbao-ca/ca.crt"
else
export BAO_ADDR=http://openbao:8200
echo "Using plain HTTP"
fi
sleep_timer=30
if [ ! -f /shared/key0 ]; then
echo "Keys not found; extract-keys init container may have failed."
exit 1
fi
echo "Unsealing pods openbao-0, openbao-1, openbao-2 (3 key steps each), ${sleep_timer}s delay between pods..."
for pod in openbao-0 openbao-1 openbao-2; do
echo "Unsealing $pod..."
for i in 0 1 2; do
key=$(cat "/shared/key$i" | tr -d '\n')
kubectl exec -n openbao "$pod" -- sh -c 'bao operator unseal "$1"' _ "$key" || true
done
echo "$pod unsealed."
if [ "$pod" != "openbao-2" ]; then
echo "Waiting ${sleep_timer}s for next pod to be ready..."
SLEEPER_TMP=1
while [ "$SLEEPER_TMP" -le "$sleep_timer" ]; do
if [ $(( SLEEPER_TMP % 10 )) -eq 0 ]; then
echo -n "$SLEEPER_TMP"
else
echo -n "."
fi
sleep 1
SLEEPER_TMP=$(( SLEEPER_TMP + 1 ))
done
echo ""
fi
done
echo "Unseal complete."
volumeMounts:
- name: shared
mountPath: /shared
readOnly: true
- name: openbao-ca
mountPath: /etc/openbao-ca
readOnly: true
volumes: (5)
- name: init-secret
secret:
secretName: openbao-init-data
- name: shared
emptyDir: {}
- name: openbao-ca
secret:
secretName: openbao-ca-secret
optional: true
restartPolicy: Never
backoffLimit: 2

1	The ServiceAccount is used by the Job to unseal the OpenBao pods. Here we are using the same ServiceAccount as the one used to initialise the OpenBao service.
2	The init container extracts unseal keys from init.json using jq and writes them to a shared volume.
3	The main container runs kubectl to exec into each OpenBao pod and run `bao operator unseal` (the OpenBao image is used inside the target pods).
4	The command is used to unseal the OpenBao service.
5	The volumes that are mounted: init-secret (contains the unseal keys), shared (contains the extracted unseal keys, shared between containers), openbao-ca (contains the CA certificate).

Both Jobs are using the same ServiceAccount and therefore the same RBAC rules. It is also possible to use a different ServiceAccount for the unseal Job.

Store init data securely. Keeping unseal keys in a cluster Secret is convenient but less secure than auto-unseal or an external secrets manager. Restrict access to the openbao-init-data Secret (e.g. RBAC and network policies) and consider rotating or moving keys to a more secure store after bootstrap.

Approach 3: Auto-unseal with Key Management Service KMS (recommended for production)

Configure a KMS (e.g. AWS KMS, Azure Key Vault, GCP Cloud KMS, Transit etc.) so that OpenBao unseals itself on every restart. You initialise once and thereafter no manual unseal step is needed.

The rest of this article will mention the different options. However, I can only demonstrate a real example using the AWS KMS option.

Auto-unseal options overview

Without auto-unseal, you must run bao operator unseal with a threshold of keys after each restart. With auto-unseal, OpenBao uses a seal backend to protect the root key and unseals itself on startup.

Supported options:

Option

Use case

Notes

1. AWS KMS

AWS (EKS, OpenShift on AWS)

IRSA or IAM user; no keys in cluster with IRSA.

2. Azure Key Vault

Azure (AKS, OpenShift on Azure)

Service principal or managed identity.

3. Google Cloud KMS

GCP (GKE)

Service account key or Workload Identity.

5. Transit

Existing Vault/OpenBao as root of trust; or a dedicated seal-only OpenBao that exists only to unseal production

External transit engine; token in a Secret.

Why use OpenBao when the seal is in KMS? Why not keep all secrets in KMS and skip OpenBao?

When using a cloud KMS (or Key Vault) for auto-unseal, a natural question is: why not store and retrieve all secrets from the KMS and run without OpenBao at all? OpenBao remains useful for operators and applications for several reasons:

KMS is for keys, not a full secrets platform. AWS KMS, Azure Key Vault keys, and GCP Cloud KMS are built to encrypt and decrypt small blobs (e.g. data encryption keys, or OpenBao’s seal blob). They are not a general-purpose secrets store with versioning, path-based access, dynamic credentials, and per-request audit. OpenBao provides that layer: applications request secrets by path, get short-lived tokens, and you get a single place to define who can read what.
Dynamic secrets. OpenBao can generate short-lived credentials on demand (e.g. database users, cloud IAM roles, PKI certificates). The KMS does not create or rotate such credentials; it only holds keys. If you need “give this pod a DB password that expires in 1 hour,” that is OpenBao’s domain, not the KMS.
Fine-grained, identity-aware access. OpenBao supports auth methods (Kubernetes, OIDC, AppRole, LDAP) and path-based policies. You can say “this service account may read only secret/data/myapp/*.” The KMS has IAM or Key Vault RBAC, but not the same model of “one API, many identities, many paths” that applications and operators use day to day.
Audit and compliance. OpenBao logs every secret read and write with identity and path. That gives a clear audit trail of who accessed which secret and when. KMS audit logs key usage, which is different from “which application read which secret at what time” in a unified way.
Abstraction and portability. Applications talk to OpenBao’s API. You can move the seal or the storage backend (or even the cloud) without changing how applications request secrets. If you put everything directly in a cloud KMS, you tie every app to that vendor’s API and limits.
Encryption as a service (Transit). OpenBao’s Transit secrets engine lets applications encrypt data with a key without ever seeing the key—useful for application-level encryption and key rotation. KMS can do something similar, but OpenBao integrates that with the same auth and audit as the rest of your secrets.

In short: the KMS is the root of trust for the seal (so OpenBao can unseal itself). OpenBao is the secrets management platform that applications and operators use for storing, retrieving, generating, and auditing secrets. Both layers complement each other.

General flow (same for all options):

Set up the seal backend (KMS key, Key Vault, static key, or transit server).
Add the appropriate seal "…" stanza to OpenBao configuration (e.g. in Helm server.ha.raft.config).
Deploy OpenBao; pods start and remain sealed until initialised.
Run one-time initialisation: bao operator init -recovery-shares=5 -recovery-threshold=3.
From then on, restarts are automatically unsealed.

Option: AWS KMS

Best for: EKS, OpenShift on AWS (ROSA), or any Kubernetes on AWS.

Create a KMS key

Create a symmetric KMS key used only for the OpenBao seal (do not use for application data).

AWS WebUI:
- search for: KMS → Create key → Symmetric, Encrypt/Decrypt
- Optionally set an alias (e.g. openbao-unseal).
- OPTIONAL: Define key administrative permissions
- OPTIONAL: Define key usage permissions
- Edit key policy

AWS CLI: (requires the AWS CLI to be installed and configured)

aws kms create-key \
--description "OpenBao auto-unseal key" \
--key-usage ENCRYPT_DECRYPT
aws kms create-alias \
--alias-name openbao-unseal \
--target-key-id <key-id-from-above>

Note the Key ID or Alias and the region (e.g. us-west-1) from the output of the command.

Key policy for OpenBao

The OpenBao process needs permission to use the key. Create an IAM policy. In the UI you can configure this directly when creating the key. When using the CLI, create a policy.json file with the following content:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": "arn:aws:kms:<REGION>:<ACCOUNT_ID>:key/<KEY_ID>" (1)
}
]
}

1	Replace `<REGION>`, `<ACCOUNT_ID>`, and `<KEY_ID>` with your values (or use the key ARN).

Create an IAM user for the seal

When using static credentials (e.g. on OpenShift or when IRSA is not available), create a dedicated IAM user that has permission only to use the KMS key above. Use that user’s access keys in the Kubernetes Secret.

AWS WebUI:
- search for: IAM → Users → Create user
- User name: e.g. openbao-seal
- DO NOT select "Provide user access to the AWS Management Console" (programmatic access only)
- Create user
- Open the created user → Permissions → Add permissions → Create inline policy (or Attach policies → Create policy)
- In the policy editor, choose JSON and paste the policy from Key policy for OpenBao, replacing <REGION>, <ACCOUNT_ID>, and <KEY_ID> with your KMS key ARN or alias
- Name the policy (e.g. OpenBaoSealKMS) and attach it to the user
- Security credentials → Create access key → Application running outside AWS (or "Third-party service") → Create access key
- Save the Access key ID and Secret access key; you will put them in the Kubernetes Secret openbao-aws-credentials.

AWS CLI: (requires the AWS CLI to be installed and configured)

Create a policy file (e.g. openbao-seal-policy.json) with the same JSON as in Key policy for OpenBao, then create the user, attach the policy, and create access keys:

# Replace REGION, ACCOUNT_ID, and KEY_ID in the policy (or use alias: arn:aws:kms:REGION:ACCOUNT_ID:alias/openbao-unseal)
aws iam create-policy \
--policy-name OpenBaoSealKMS \
--policy-document file://openbao-seal-policy.json
aws iam create-user --user-name openbao-seal
aws iam attach-user-policy \
--user-name openbao-seal \
--policy-arn arn:aws:iam::ACCOUNT_ID:policy/OpenBaoSealKMS
aws iam create-access-key --user-name openbao-seal

The last command returns AccessKeyId and SecretAccessKey; store them in the Secret openbao-aws-credentials with key AWS_REGION set to your region (e.g. us-west-1).

Create the Secret

Create the Secret openbao-aws-credentials in the openbao namespace. It contains the access key, secret access key, and region.

apiVersion: v1
kind: Secret
metadata:
name: openbao-aws-credentials
namespace: openbao
type: Opaque
stringData: (1)
AWS_ACCESS_KEY_ID: <your-access-key>
AWS_SECRET_ACCESS_KEY: <your-secret-key>
AWS_REGION: <your-region>

1	Replace `<your-access-key>`, `<your-secret-key>`, and `<your-region>` with the values from the previous steps.

Helm / Argo CD configuration (AWS)

Add the seal "awskms" block inside the same HCL config that contains listener, storage "raft". In Helm values this is typically under openbao.server.ha.raft.config.

Be sure to update the region and key ID.

The values file we used for Argo CD can be found at: OpenBao Argo CD values file

openbao:
server:
extraSecretEnvironmentVars: (1)
- envName: AWS_ACCESS_KEY_ID
secretName: openbao-aws-credentials
secretKey: AWS_ACCESS_KEY_ID
- envName: AWS_SECRET_ACCESS_KEY
secretName: openbao-aws-credentials
secretKey: AWS_SECRET_ACCESS_KEY
- envName: AWS_REGION
secretName: openbao-aws-credentials
secretKey: AWS_REGION
ha:
raft:
config: |
ui = true
seal "awskms" { (2)
region = "us-east-1"
kms_key_id = "alias/openbao-unseal"
}
listener "tcp" {
tls_disable = 0
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/openbao/tls/openbao-server-tls/tls.crt"
tls_key_file = "/openbao/tls/openbao-server-tls/tls.key"
tls_min_version = "tls12"
}
storage "raft" {
path = "/openbao/data"
}
service_registration "kubernetes" {}

1	If you use static credentials (e.g. for OpenShift), inject them via a Secret.
2	This must be added. Be sure to update your region and key ID.

The seal "awskms" block alone is not enough. OpenBao must have AWS credentials available at runtime. If you see an error such as NoCredentialProviders: no valid providers in chain or error fetching AWS KMS wrapping key information, the pod has no credentials. You must either inject static credentials from a Secret (see below) or use IRSA so the pod can assume an IAM role.

Approach 4: Init container with external secret store

Another option is to use an init container that retrieves secrets from an external source. Two use cases can be considered here:

Shamir unseal: The init container fetches the unseal keys from an external secret manager and then runs bao operator unseal (or writes keys to a file used by a sidecar/unseal script). This is the classic “external secret store” idea (e.g. using AWS Secrets Manager, HashiCorp Vault, another Kubernetes cluster).
Transit seal token: The init container fetches the token that production OpenBao uses to call the dedicated unseal service (see Dedicated unseal service (seal-only OpenBao)). That token is written to a file or shared volume so the main OpenBao process can use it in seal "transit". The token never lives in a Kubernetes Secret in Git or in plain YAML; it is fetched at pod start from the external store.

Below is a generic placeholder for the init container.

Option: init Container example

The following options are generic and more theoretical. Unlike the other options, I could not test them in a real environment yet. This may be covered in a future article.

Generic placeholder (implement according to your secret store):

# Additional values for the Helm chart
server:
extraInitContainers:
- name: fetch-secrets
image: registry.access.redhat.com/ubi9/ubi-minimal:latest
command:
- /bin/sh
- -c
- |
# Fetch unseal keys OR transit token from external secret manager
echo "Waiting for OpenBao to be ready..."
sleep 30
# Your fetch logic here (e.g. aws secretsmanager get-secret-value, vault kv get, curl to API) (1)
env:
- name: EXTERNAL_SECRET_ENDPOINT (2)
value: "https://your-external-secret-store.example.com"

1	Replace with your fetch logic.
2	Replace with your external secret store endpoint.

Option: Transit seal

When you already have a Vault or OpenBao cluster and want it to act as the root of trust (e.g. central Vault encrypts the seal key), you can use this option.

To prepare everything, you need to do the following:

Enable the Transit secrets engine on the external Vault/OpenBao and create a key (e.g. openbao-unseal).
Grant the token used by this OpenBao permission to encrypt/decrypt with that key.
Store the token in a Kubernetes Secret and inject via extraSecretEnvironmentVars

Do not put the token in the config file.

The following HCL configuration is used to configure the OpenBao cluster to use the transit seal.

seal "transit" {
address = "https://external-vault.example.com:8200"
token = "s.xxx"
disable_renewal = "false"
key_name = "openbao-unseal"
mount_path = "transit/"
}

See the OpenBao transit seal documentation for further details.

Dedicated unseal service (seal-only OpenBao)

A common and recommended pattern is to run a separate, standalone OpenBao instance whose only role is to provide the seal for your production OpenBao. In other words: the external OpenBao holds the unseal keys—via its Transit secrets engine—and the production OpenBao cluster uses the transit seal to auto-unseal by calling that external instance. No application secrets or other workloads run on the dedicated instance; it exists solely to unseal the production OpenBao.

Why use a dedicated unseal service?

Separation of concerns: The root of trust for unsealing lives outside the production cluster. If the production OpenBao is compromised or rebuilt, the seal keys remain in the dedicated instance.
Simpler operations: You unseal and manage only one small, locked-down OpenBao (the seal service). The production OpenBao unseals itself automatically via transit.
No cloud KMS required: On-premise or air-gapped environments can use this pattern instead of AWS KMS, Azure Key Vault, or GCP Cloud KMS.
Clear trust boundary: The dedicated instance can be hardened, network-isolated, and backed up independently.

Architecture (high level):

Dedicated unseal service: A standalone OpenBao (single node is often enough) with the Transit secrets engine enabled and a dedicated transit key (e.g. production-openbao-unseal). This instance is initialised and unsealed once; you keep its unseal keys and root token very secure. It does not store application secrets.
Production OpenBao: Your HA OpenBao cluster (e.g. in Kubernetes) is configured with seal "transit" pointing at the dedicated service URL and a token that has permission only to use that transit key. On startup, each production node asks the dedicated OpenBao to decrypt its seal blob and thus unseals automatically.

Migration from Shamir to auto-unseal

You might wonder if you can migrate from Shamir to auto-unseal. The answer is yes, you can.

If OpenBao was previously initialised with Shamir unseal keys and you want to switch to any auto-unseal backend, you can do the following:

Plan a short maintenance window. Raft HA will tolerate one node at a time.
Add the appropriate seal "…" block to the config and deploy (e.g. via Argo CD).

Do NOT re-run init.
Unseal the leader with the existing Shamir unseal keys.
Run seal migration on the leader:
```
bao operator unseal -migrate
```
Restart the leader. It should auto-unseal via the new seal.
Update and restart standby nodes. They should auto-unseal as well.

Resources

The Guide to OpenBao - GitOps Deployment with Argo CD - Part 5

Fri, 20 Feb 2026 00:00:00 +0000

Following the GitOps mantra "If it is not in Git, it does not exist", this article demonstrates how to deploy and manage OpenBao using Argo CD. This approach provides version control, audit trails, and declarative management for your secret management infrastructure.

Introduction

Deploying OpenBao via GitOps offers significant advantages:

Version Control: All configuration changes are tracked in Git
Audit Trail: Who changed what, when, and why
Declarative: Desired state is defined, not imperative commands
Reproducible: Same deployment process across environments
Self-healing: Argo CD ensures actual state matches desired state

However, there are challenges specific to secret management:

Initial unsealing requires manual intervention or automationxw
Root tokens and unseal keys must be handled carefully
Chicken-and-egg problem: How to store OpenBao secrets before OpenBao exists?

This article will focus on the deployment of the applicaiotns for the OpenBao deployment. The problem with the automatic unsealing will be addressed in the next article.

Prerequisites

Before you begin, ensure you have:

OpenShift GitOps (Argo CD) installed and configured
A Git repository for your cluster configuration
Understanding of the App-of-Apps pattern (see Configure App-of-Apps)
The official OpenBao Helm chart from Part 3

The (Wrapper) Helm Chart

The official OpenBao Helm Chart works well for deploying OpenBao itself. However, additional settings such as certificates are not covered there. Therefore, I created a (wrapper) Helm Chart that includes the official OpenBao Helm Chart, a Cert-Manager Helm Chart that I created a while ago, and allows you to add additional objects in the templates folder.

I will follow the same setup as discussed in Part 4. This means:

Create Issuer for Cert-Manager
Create CA Certificate for OpenBao
Create Certificate for OpenBao Server
Create Certificate for OpenBao Agent Injector
Install OpenBao Helm Chart

However, there is one significant issue with this approach: we need the certificate details (especially the CA certificate) before we can install the OpenBao Helm Chart, since the certificate must be referenced in the values file. This is a chicken-and-egg problem—particularly if you use self-signed CA certificates.

Since you typically create the CA certificate first and only once (or have it already), a separate Application for the CA certificate is a good approach. Once this is synchronised, the OpenBao Application can be updated and deployed.

This means we will need two Helm Charts: one for the CA certificate and one for OpenBao itself.

Helm Chart for the CA Certificate

Create a new Helm Chart for the CA certificate. This will use the Cert-Manager Helm Chart as a dependency and will create the:

Issuer for Cert-Manager
Request the CA certificate from the Issuer
Create the openbao namespace

The example I am using can be found at GitOps openbao-ca-certificate.

Please check the GitOps Repository Structure and subsequent articles for more information on how to structure your Git repository.

In the Chart.yaml file you can see two dependencies:

dependencies:
- name: tpl
version: ~1.0.0
repository: https://charts.stderr.at/
- name: cert-manager
version: ~2.0.3 (1)
repository: https://charts.stderr.at/
condition: cert-manager.enabled

1	The Cert-Manager Helm Chart version must be at least 2.0.3 to support the namespace parameter for the Issuer.

The tpl dependency is a library that contains templates for the namespace and other shared components (I keep this library in my Helm Charts repository). The cert-manager dependency is the Cert-Manager Helm Chart.

The values.yaml file below contains the configuration for the Cert-Manager:

All settings that are passed to a subchart (cert-manager) must be prefixed with the name of the subchart. In this case, cert-manager..

namespace:
create: true (1)
name: "openbao"
description: "OpenBao Namespace"
displayName: "OpenBao Namespace"
additionalLabels:
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/audit-version: latest
pod-security.kubernetes.io/warn: restricted
pod-security.kubernetes.io/warn-version: latest
cert-manager: (2)
enabled: true (3)
issuer:
# Name of issuer
- name: openbao-selfsigned (4)
# -- Syncwave to create this issuer
syncwave: 5 (5)
# -- Type can be either ClusterIssuer or Issuer
type: Issuer
# -- Enable this issuer.
# @default -- false
enabled: true
# -- Namespace for Issuer (ignored for ClusterIssuer). Defaults to the default namespace when not set.
namespace: openbao (6)
# -- Create a selfSigned issuer. The SelfSigned issuer doesn't represent a certificate authority as such, but instead denotes that certificates will "sign themselves" using a given private key.
selfSigned: true (7)
certificates: (8)
enabled: true
# List of certificates
certificate:
- name: openbao-ca
enabled: true
namespace: openbao
syncwave: "10"
secretName: openbao-ca-secret
duration: 87660h
dnsNames:
- openbao-ca
privateKey:
algorithm: ECDSA
size: 256
rotationPolicy: Always
isCA: true
# Reference to the issuer that shall be used.
issuerRef:
name: openbao-selfsigned
kind: Issuer

1	Create the openbao namespace with various settings.
2	Enable the Cert-Manager subchart. All settings below will be passed to the Cert-Manager subchart.
3	Enables Cert-Manager.
4	Name of the issuer.
5	Syncwave for this issuer; it must be lower than the syncwave of the certificate.
6	Namespace for the issuer; supported since version 2.0.3 of the Cert-Manager Helm Chart.
7	Creates a self-signed issuer.
8	The certificate section and all the settings. Verify the Part 4 for more information.

This chart creates the openbao namespace, which is required for the OpenBao deployment.

Helm Chart for the OpenBao Deployment

Create a new Helm Chart for the OpenBao deployment. This will use the official OpenBao Helm chart and the Cert-Manager Helm Chart as dependencies and will create the:

OpenBao deployment (including Route)
Issuer for the OpenBao server using the CA certificate
Required OpenBao certificates based on the CA certificate

I have added the full values.yaml file below. Please check the GitOps openbao/values.yaml to fetch the full chart.

Besides the two certificates added here in the values.yaml file, I made two further changes: fullnameOverride and nameOverride are both set to openbao. This is good practice for setting the name of the deployment when using Argo CD.

This values file contains the CA certificate in plain text. Whether that is acceptable is debatable—it is a public certificate, and here it is self-signed and used only in my demo environment.

# Full values.yaml file for the OpenBao deployment
########################################################
# Cert-Manager
########################################################
cert-manager:
enabled: true
issuer:
# Name of issuer
- name: openbao-ca-issuer (1)
# -- Syncwave to create this issuer
syncwave: '-1'
# -- Type can be either ClusterIssuer or Issuer
type: Issuer
# -- Enable this issuer.
# @default -- false
enabled: true
# -- Namespace for Issuer (ignored for ClusterIssuer). Defaults to the default namespace when not set.
namespace: openbao
ca:
secretName: openbao-ca-secret
certificates:
enabled: true
# List of certificates
certificate:
########################################################
# OpenBao Server TLS Certificate
########################################################
- name: openbao-server-tls (2)
enabled: true
namespace: openbao
syncwave: "0"
secretName: openbao-server-tls
duration: 8760h
renewBefore: 720h
dnsNames:
- openbao.openbao.svc
- openbao.apps.ocp.aws.ispworld.at # Route host (adjust to your domain)
- openbao
- openbao.openbao
- openbao.openbao.svc
- openbao.openbao.svc.cluster.local
- openbao-internal
- openbao-internal.openbao
- openbao-internal.openbao.svc
- openbao-internal.openbao.svc.cluster.local
- openbao-0.openbao-internal
- openbao-0.openbao-internal.openbao
- openbao-0.openbao-internal.openbao.svc
- openbao-0.openbao-internal.openbao.svc.cluster.local
- openbao-1.openbao-internal
- openbao-1.openbao-internal.openbao
- openbao-1.openbao-internal.openbao.svc
- openbao-2.openbao-internal
- openbao-2.openbao-internal.openbao
- openbao-2.openbao-internal.openbao.svc
ipAddresses:
- 127.0.0.1
- "::1"
# Reference to the issuer that shall be used.
issuerRef:
name: openbao-ca-issuer
kind: Issuer
########################################################
# Injector TLS Certificate
########################################################
- name: injector-certificate (3)
enabled: true
namespace: openbao
syncwave: "0"
secretName: injector-tls
duration: 24h
renewBefore: 144m
dnsNames:
- openbao-agent-injector-svc
- openbao-agent-injector-svc.openbao
- openbao-agent-injector-svc.openbao.svc
- openbao-agent-injector-svc.openbao.svc.cluster.local
# Reference to the issuer that shall be used.
issuerRef:
name: openbao-ca-issuer
kind: Issuer
########################################################
# OpenBao Deployment
########################################################
openbao: (4)
# Override the full name of the deployment via Argo CD
fullnameOverride: openbao
nameOverride: openbao
global:
# Enable OpenShift-specific settings
openshift: true
# -- The namespace to deploy to. Defaults to the `helm` installation namespace.
namespace: openbao
# Required when TLS is enabled: tells the chart to use HTTPS for readiness/liveness
# probes and for in-pod API_ADDR (127.0.0.1:8200). Otherwise you get "client sent
# an HTTP request to an HTTPS server" from the probes.
tlsDisable: false
server:
extraEnvironmentVars:
BAO_CACERT: /openbao/tls/openbao-server-tls/ca.crt
# High Availability configuration
ha:
enabled: true
replicas: 3
# Raft storage configuration
raft:
enabled: true
setNodeId: true
config: |
ui = true
listener "tcp" {
tls_disable = 0
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/openbao/tls/openbao-server-tls/tls.crt"
tls_key_file = "/openbao/tls/openbao-server-tls/tls.key"
tls_min_version = "tls12"
telemetry {
unauthenticated_metrics_access = "true"
}
}
storage "raft" {
path = "/openbao/data"
retry_join {
leader_api_addr = "https://openbao-0.openbao-internal:8200"
leader_tls_servername = "openbao-0.openbao-internal"
leader_ca_cert_file = "/openbao/tls/openbao-server-tls/ca.crt"
}
retry_join {
leader_api_addr = "https://openbao-1.openbao-internal:8200"
leader_tls_servername = "openbao-1.openbao-internal"
leader_ca_cert_file = "/openbao/tls/openbao-server-tls/ca.crt"
}
retry_join {
leader_api_addr = "https://openbao-2.openbao-internal:8200"
leader_tls_servername = "openbao-2.openbao-internal"
leader_ca_cert_file = "/openbao/tls/openbao-server-tls/ca.crt"
}
}
service_registration "kubernetes" {}
telemetry {
prometheus_retention_time = "30s"
disable_hostname = true
}
route:
enabled: true
host: openbao.apps.ocp.aws.ispworld.at
tls:
# Route terminates client TLS; backend can use reencrypt or passthrough
termination: reencrypt
insecureEdgeTerminationPolicy: Redirect
destinationCACertificate: |
-----BEGIN CERTIFICATE-----
MIIBWzCCAQCgAwIBAgIQNdbg4KIu9oi6dDClE8drmjAKBggqhkjOPQQDAjAAMB4X
DTI2MDIxODA5NDIxNFoXDTM2MDIxODIxNDIxNFowADBZMBMGByqGSM49AgEGCCqG
SM49AwEHA0IABIeYw35/kEHvyctLtOA5xMlyQNxUtXtfBbZMUfPh6AN5MFjIGuNS
cn07a3EpSpfY6/3DaPpu+4wYNFlc+/qDNYajXDBaMA4GA1UdDwEB/wQEAwICpDAP
BgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQgyRk5Vv9K0VULcCgga5mRg4O9kTAY
BgNVHREBAf8EDjAMggpvcGVuYmFvLWNhMAoGCCqGSM49BAMCA0kAMEYCIQCwQ7lZ
Q0jzUjJFzpTGkQjU2+OB159LIQMSbSQ7dz8nVQIhAIDa7f87tjQxDxbJio+/vJx2
awFaWnueGOOQpvwCcV/+
-----END CERTIFICATE-----
extraVolumes:
- type: secret
name: openbao-server-tls
path: /openbao/tls
readOnly: true
extraVolumeMounts:
- name: openbao-server-tls
mountPath: /openbao/tls
readOnly: true
# Resource requests and limits
resources:
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 1Gi
cpu: 1000m
# Persistent volume for data
dataStorage:
enabled: true
size: 10Gi
# storageClass: "gp3-csi"
# Injector configuration
injector:
enabled: true
replicas: 2 # HA for the injector too
certs:
secretName: injector-tls
# For a private CA: set caBundle to the CA cert (PEM) so the Kubernetes API server trusts the injector webhook. E.g. oc get secret openbao-ca-secret -n openbao -o jsonpath='{.data.ca\.crt}' | base64 -d
caBundle: "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJXekNDQVFDZ0F3SUJBZ0lRTmRiZzRLSXU5b2k2ZERDbEU4ZHJtakFLQmdncWhrak9QUVFEQWpBQU1CNFgKRFRJMk1ESXhPREE1TkRJeE5Gb1hEVE0yTURJeE9ESXhOREl4TkZvd0FEQlpNQk1HQnlxR1NNNDlBZ0VHQ0NxRwpTTTQ5QXdFSEEwSUFCSWVZdzM1L2tFSHZ5Y3RMdE9BNXhNbHlRTnhVdFh0ZkJiWk1VZlBoNkFONU1GaklHdU5TCmNuMDdhM0VwU3BmWTYvM0RhUHB1KzR3WU5GbGMrL3FETllhalhEQmFNQTRHQTFVZER3RUIvd1FFQXdJQ3BEQVAKQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlFneVJrNVZ2OUswVlVMY0NnZ2E1bVJnNE85a1RBWQpCZ05WSFJFQkFmOEVEakFNZ2dwdmNHVnVZbUZ2TFdOaE1Bb0dDQ3FHU000OUJBTUNBMGtBTUVZQ0lRQ3dRN2xaClEwanpVakpGenBUR2tRalUyK09CMTU5TElRTVNiU1E3ZHo4blZRSWhBSURhN2Y4N3RqUXhEeGJKaW8rL3ZKeDIKYXdGYVdudWVHT09RcHZ3Q2NWLysKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo="
certName: tls.crt
keyName: tls.key
# UI configuration
ui:
enabled: true

1	Issuer for OpenBao. The syncwave is set to '-1' to create the issuer before the certificate request and the OpenBao deployment.
2	Certificate for the OpenBao server, with all DNS names and IP addresses used to reach it. The syncwave is set to '0' so the certificate is created after the issuer.
3	Certificate for the OpenBao Agent Injector, with all DNS names used to reach it. The syncwave is set to '0' so the certificate is created after the issuer.
4	OpenBao deployment configuration; the two overrides fullnameOverride and nameOverride are both set to openbao. See Part 4 for more information.

Creating Argo CD Applications

Whilst the Helm Charts are ready, we need to create the Argo CD Applications for the CA certificate and the OpenBao deployment. If you read our blog carefully, you will notice that these Application resources are created automatically when I add something to the folder clusters/management-cluster :) This is done by leveraging ApplicationSet resources and is fully described in the GitOps Repository Structure and subsequent articles.

For the sake of simplicity, I will show the created Application resources below. The setup is the same for both; they simply target a different path in the Git repository.

# Full Application resource for the OpenBao CA Certificate
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: in-cluster-openbao-ca-certificate
namespace: openshift-gitops
spec:
destination:
name: in-cluster (1)
namespace: default (2)
info:
- name: Description
value: ApplicationSet that Deploys on Management Cluster Configuration (using Git Generator)
project: in-cluster (3)
source:
path: clusters/management-cluster/openbao-ca-certificate (4)
repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops' (5)
targetRevision: main
syncPolicy:
retry:
backoff:
duration: 5s
factor: 2
maxDuration: 3m
limit: 5
---
# Full Application resource for the OpenBao deployment
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: in-cluster-openbao
namespace: openshift-gitops
spec:
destination:
name: in-cluster (1)
namespace: openbao (2)
info:
- name: Description
value: ApplicationSet that Deploys on Management Cluster Configuration (using Git Generator)
project: in-cluster (3)
source:
path: clusters/management-cluster/openbao (6)
repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops' (5)
targetRevision: main
syncPolicy:
retry:
backoff:
duration: 5s
factor: 2
maxDuration: 3m
limit: 5

1	Target cluster; here, the local cluster
2	Namespace of the target cluster; here, the OpenBao deployment will be installed.
3	Argo CD Project (must exist)
4	Path to the Git repository for the CA certificate
5	URL of the Git repository
6	Path to the OpenBao deployment

This will create the following Applications: - in-cluster-openbao-ca-certificate - in-cluster-openbao

The ApplicationSet adds the prefix in-cluster- to each Application name so that they remain unique in Argo CD.

Figure 1. Argo CD: OpenBao Argo CD Applications

The first Application to synchronise is in-cluster-openbao-ca-certificate.

It creates the following:

Namespace
CA Issuer
CA Certificate

Then synchronise in-cluster-openbao. It creates:

OpenBao deployment
OpenBao Agent Injector
OpenBao Server TLS Certificate
OpenBao Agent Injector TLS Certificate

OpenBao is running—what next?

Deploying OpenBao via GitOps gives you version control and declarative management for your secret management infrastructure.

Whilst OpenBao is running and managed by Argo CD, the next step is to configure it: you will need to handle initialisation (for a new cluster) and unsealing (see Part 3 and Part 4). That manual approach does not scale. In the next article, I will discuss ways to automate initialisation and unsealing.

Key takeaways:

Use sync waves to control deployment order
Consider auto-unseal for production (Part 6)
Store initialisation data securely outside Git

Resources

The Guide to OpenBao - Enabling TLS on OpenShift - Part 4

Tue, 17 Feb 2026 00:00:00 +0000

In Part 3 we deployed OpenBao on OpenShift in HA mode with TLS disabled: the OpenShift Route terminates TLS at the edge, and traffic from the Route to the pods is plain HTTP. While this is ok for quick tests, for a production-ready deployment, you should consider TLS for the entire journey. This article explains why and how to enable TLS end-to-end using the cert-manager operator, what to consider, and the exact steps to achieve it.

Introduction

Part 3 uses tls_disable = 1 in the OpenBao listener and relies on the OpenShift Route for TLS. That gives encryption between the client and the Route, but not between the Route and the OpenBao pods or between pods (e.g. Raft). Enabling TLS on OpenBao itself adds encryption in transit everywhere and aligns with defense-in-depth and compliance requirements. After all, we are talking about a secrets management system here and should not compromise security.

This part assumes:

OpenBao is already deployed as in Part 3 (HA with Raft, Agent Injector enabled).
The cert-manager operator is installed and configured on your OpenShift cluster.

The article SSL Certificate Management for OpenShift describes the setup and usage of the cert-manager operator as an example.

Why Enable TLS?

Enabling TLS for OpenBao makes sense for several reasons:

Encryption in transit: Traffic between the Route and the pods, and between OpenBao peers (Raft), is encrypted. Secrets and tokens are never sent in plain text on the cluster network.
Defense in depth: Even if the Route or network is misconfigured, backend traffic remains protected.
Compliance: Many standards (e.g. PCI-DSS, SOC 2) require encryption in transit for sensitive data; TLS to the application (OpenBao) helps satisfy this.
Agent Injector: The injector webhook is called by the Kubernetes API server. Using TLS for the webhook (with a valid certificate) is required for production and when running multiple replicas.
Consistency: Using HTTPS everywhere simplifies client configuration and avoids mixing HTTP/HTTPS in the same environment.

What Must Be Considered?

Two TLS contexts: Each needs its own certificate and configuration.
1. OpenBao server (API and Raft)
2. Agent Injector (mutating webhook)
Certificate SANs: Server certificates must include all names used to reach OpenBao: Route host, internal service names (e.g. openbao.openbao.svc, openbao-0.openbao-internal.openbao.svc), 127.0.0.1 and ::1 for in-pod traffic, and any external DNS you use. The injector certificate must match the injector Service DNS name (e.g. openbao-agent-injector-svc.openbao.svc).
cert-manager: Using cert-manager gives automatic issuance and renewal. You need a ClusterIssuer (or an Issuer) in the OpenBao namespace.

In this example, we will use a self-signed CA for the OpenBao server and the Agent Injector. In a production environment, you should use a trusted CA.

OpenShift Route: With backend TLS enabled, you can keep the Route in reencrypt mode: the Route terminates TLS from the client and opens a new TLS connection to the pod. Alternatively, use passthrough if you want end-to-end TLS without re-encryption at the Route.
Raft join: After switching to TLS, retry_join and cluster addresses must use https:// and the correct hostnames. Existing unseal keys and root token are unchanged; only the transport is different.
Clients: CLI and applications must use https:// for BAO_ADDR and, if you use a private CA, BAO_CACERT (or the system trust store) so that the client trusts the server certificate.

Prerequisites

OpenBao HA deployment from Part 3 (namespace openbao, Helm release openbao).
cert-manager operator installed on OpenShift.
Sufficient rights to create Issuers, Certificates, and Secrets in the openbao namespace.

Overview of Steps

Create a Certificate Authority (CA) in the openbao namespace (or use an existing ClusterIssuer).
Issue a Certificate for the OpenBao server (API + Raft) and store it in a Secret.
Issue a Certificate for the Agent Injector and reference it in the Helm values.
Update Helm values to mount the server cert and CA, and configure the listener and Raft for TLS.
Upgrade the Helm release; initialize and unseal openbao-0 (new cluster) or re-unseal (existing).
Verify access via HTTPS and configure clients (BAO_ADDR, BAO_CACERT).

Step 1: Certificate Authority (CA) for OpenBao

If you already have a ClusterIssuer (e.g. Let’s Encrypt or an enterprise CA), you can use it for the server and injector certificates and skip this step. For a self-signed CA in the OpenBao namespace (typical for internal cluster TLS), create the following.

This self-signed CA is only for testing purposes. In a production environment, you should use a trusted CA, preferably your own.

Create a self-signed CA Issuer in the openbao namespace:

You might have your own CA or a ClusterIssuer already. You can use them, instead of creating a new one.
```
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: openbao-selfsigned
namespace: openbao (1)
spec:
selfSigned: {} (2)
```
1 The namespace where the OpenBao deployment is running.

2 The self-signed CA Issuer.

Create a self-signed CA Certificate in the openbao namespace:

This will now actually request the CA certificate from the self-signed CA Issuer. This process is fully automated by cert-manager because everything is self-signed. The requested certificate will be stored in the secret openbao-ca-secret and is available immediately.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: openbao-ca
namespace: openbao (1)
spec:
isCA: true (2)
commonName: OpenBao CA
secretName: openbao-ca-secret (3)
duration: 87660h (4)
privateKey: (5)
algorithm: ECDSA
size: 256 (6)
rotationPolicy: Always (6)
issuerRef:
name: openbao-selfsigned (7)
kind: Issuer (8)
group: cert-manager.io

1	The namespace where the OpenBao deployment is running.
2	The certificate is a CA certificate.
3	The name of the secret where the certificate and key will be stored.
4	The duration of the certificate. In this case 10 years.
5	The private key algorithm and size.
6	The rotation policy of the private key.
7	The issuer of the certificate.
8	The kind of the issuer. Can be Issuer or ClusterIssuer.

Step 2: Create an Issuer for the OpenBao Server and Agent Injector

Create an Issuer for the OpenBao Server and Agent Injector. This Issuer will reference the CA certificate and key stored in the secret openbao-ca-secret.

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: openbao-ca-issuer
namespace: openbao
spec:
ca:
secretName: openbao-ca-secret

The name of the secret where the CA certificate and key are stored.

Step 3: Certificate for the OpenBao Server

Now it is time to create the certificate for the OpenBao server. The server certificate must include every hostname used to reach OpenBao: the Route host, the headless service, and each Raft member. Adjust the dnsNames and optional uris to match your cluster and Route.

Create the following Certificate object:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: openbao-server-tls
namespace: openbao
spec:
secretName: openbao-server-tls (1)
duration: 8760h (2)
renewBefore: 720h (3)
commonName: openbao.openbao.svc (4)
dnsNames:
- openbao.apps.cluster.example.com # Route host (adjust to your domain) (5)
- openbao (6)
- openbao.openbao
- openbao.openbao.svc
- openbao.openbao.svc.cluster.local
- openbao-internal
- openbao-internal.openbao
- openbao-internal.openbao.svc
- openbao-internal.openbao.svc.cluster.local
- openbao-0.openbao-internal
- openbao-0.openbao-internal.openbao
- openbao-0.openbao-internal.openbao.svc
- openbao-0.openbao-internal.openbao.svc.cluster.local
- openbao-1.openbao-internal
- openbao-1.openbao-internal.openbao
- openbao-1.openbao-internal.openbao.svc
- openbao-2.openbao-internal
- openbao-2.openbao-internal.openbao
- openbao-2.openbao-internal.openbao.svc
ipAddresses: (7)
- 127.0.0.1
- "::1"
issuerRef:
name: openbao-ca-issuer (8)
kind: Issuer
group: cert-manager.io

1	The name of the secret where the certificate and key will be stored.
2	The duration of the certificate. In this case 1 year.
3	The duration before the certificate is renewed. In this case 30 days.
4	The common name of the certificate. This is the service name of the OpenBao server.
5	The Route host. Adjust to your domain.
6	The headless service names, all should be included.
7	Required for in-pod traffic: Readiness/liveness probes and local bao commands (e.g. raft join, operator unseal) connect to 127.0.0.1:8200. Without these IP SANs you get "tls: bad certificate" or "x509: cannot validate certificate for 127.0.0.1 because it doesn’t contain any IP SANs".
8	The issuer of the certificate, this time openbao-ca-issuer.

After a few moments the certificate will be ready and the secret will be created. The cert-manager will store the signed certificate and key in the Secret openbao-server-tls with keys tls.crt and tls.key. The Helm chart can mount this secret for the OpenBao listener.

Step 4: Certificate for the Agent Injector

The Agent Injector runs as a webhook; the Kubernetes API server calls it over TLS. The certificate must match the Service DNS name of the injector. Create a Certificate that references the same CA Issuer. In this example we use a short-lived certificate valid for 24 hours, renewed when 10% of the validity period remains.

Save as openbao-injector-cert.yaml:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: injector-certificate
namespace: openbao
spec:
secretName: injector-tls (1)
duration: 24h (2)
renewBefore: 144m
commonName: Agent Inject Cert (3)
dnsNames:
- openbao-agent-injector-svc (4)
- openbao-agent-injector-svc.openbao
- openbao-agent-injector-svc.openbao.svc
- openbao-agent-injector-svc.openbao.svc.cluster.local
issuerRef:
name: openbao-ca-issuer
kind: Issuer
group: cert-manager.io

1	The name of the secret where the certificate and key will be stored.
2	The duration of the certificate. In this case 24 hours. (renewal is done 10% before expiry)
3	The injector service name.
4	The injector service name is defined by the Helm chart. If you override the injector service name, adjust dnsNames accordingly.

Step 5: Helm Values for Server TLS

With all the certificates created, we can update the Helm values so that OpenBao uses the server certificate and listens with TLS. You need to:

Mount the Secret openbao-server-tls into the OpenBao pods.
Set the listener to use tls_cert_file and tls_key_file and disable tls_disable.
Switch Raft retry_join and cluster addresses to https://.
Add the environment variable BAO_CACERT to the OpenBao pods.

Before we start, we need to export the CA certificate from the secret openbao-ca-secret and save its value:

oc get secret openbao-ca-secret -n openbao -o jsonpath='{.data.ca\.crt}' | base64 -d

This will return the certificate like this:

-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

Save this, you will need it in the next step.

Now we need to create the Helm values file. This time we will enable TLS. Refer to Part 3 of this series to see what the initial values file looks like.

Create or update openbao-ha-values-tls.yaml (building on your Part 3 values):

global:
# Enable OpenShift-specific settings
openshift: true
# Required when TLS is enabled: tells the chart to use HTTPS for readiness/liveness
# probes and for in-pod API_ADDR (127.0.0.1:8200). Otherwise you get "client sent
# an HTTP request to an HTTPS server" from the probes.
tlsDisable: false (1)
server:
extraEnvironmentVars:
BAO_CACERT: /openbao/tls/openbao-server-tls/ca.crt (2)
# High Availability configuration
ha:
enabled: true
replicas: 3
# Raft storage configuration
raft:
enabled: true
setNodeId: true
config: |
ui = true
listener "tcp" {
tls_disable = 0 (3)
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/openbao/tls/openbao-server-tls/tls.crt" (4)
tls_key_file = "/openbao/tls/openbao-server-tls/tls.key" (5)
tls_min_version = "tls12" (6)
telemetry {
unauthenticated_metrics_access = "true"
}
}
storage "raft" {
path = "/openbao/data"
retry_join {
leader_api_addr = "https://openbao-0.openbao-internal:8200" (7)
leader_tls_servername = "openbao-0.openbao-internal"
leader_ca_cert_file = "/openbao/tls/openbao-server-tls/ca.crt" (8)
}
retry_join {
leader_api_addr = "https://openbao-1.openbao-internal:8200"
leader_tls_servername = "openbao-1.openbao-internal"
leader_ca_cert_file = "/openbao/tls/openbao-server-tls/ca.crt"
}
retry_join {
leader_api_addr = "https://openbao-2.openbao-internal:8200"
leader_tls_servername = "openbao-2.openbao-internal"
leader_ca_cert_file = "/openbao/tls/openbao-server-tls/ca.crt"
}
}
service_registration "kubernetes" {}
telemetry {
prometheus_retention_time = "30s"
disable_hostname = true
}
route:
enabled: true
host: openbao.apps.cluster.example.com
tls: (9)
# Route terminates client TLS; backend can use reencrypt or passthrough
termination: reencrypt
insecureEdgeTerminationPolicy: Redirect
destinationCACertificate: | (10)
-----BEGIN CERTIFICATE-----
# CA Certificate
-----END CERTIFICATE-----
extraVolumes: (11)
- type: secret
name: openbao-server-tls
path: /openbao/tls
readOnly: true
extraVolumeMounts: (12)
- name: openbao-server-tls
mountPath: /openbao/tls
readOnly: true
# Resource requests and limits
resources:
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 1Gi
cpu: 1000m
# Persistent volume for data
dataStorage:
enabled: true
size: 10Gi
# storageClass: "gp3-csi"
# Injector configuration
injector:
enabled: true
replicas: 2 # HA for the injector too
certs: (13)
secretName: injector-tls
# For a private CA: set caBundle to the CA cert (PEM) so the Kubernetes API server trusts the injector webhook. E.g. oc get secret openbao-ca-secret -n openbao -o jsonpath='{.data.ca\.crt}' | base64 -d
caBundle: "BASE64_ENCODED_CA_CERTIFICATE"
certName: tls.crt
keyName: tls.key
# UI configuration
ui:
enabled: true

1	global.tlsDisable: Set to false when the server listener uses TLS. This makes the chart use HTTPS for readiness/liveness probes and for the in-pod API_ADDR env var. If you leave it true (default), probes and local clients will use HTTP and you will see "client sent an HTTP request to an HTTPS server".
2	The environment variable BAO_CACERT is set to the CA certificate file path. This is helpful to execute the boa command inside the container.
3	The listener tls_disable is set to 0 to enable TLS.
4	The listener tls_cert_file is set to the certificate file path.
5	The listener tls_key_file is set to the key file path.
6	The listener tls_min_version is set to tls12.
7	The leader API address is set to the HTTPS address.
8	The leader CA certificate file is set to the CA certificate file path.
9	The Route tls termination is set to reencrypt.
10	The destination CA certificate is set to the CA certificate. Be sure not to add any extra lines or spaces.
11	The extra volumes are mounted at the /openbao/tls path.
12	The extra volume mounts are mounted at the /openbao/tls path.
13	The injector certs are set to the injector TLS secret name. The caBundle must be provided in base64 format.

The secret key names must match what cert-manager writes: tls.crt and tls.key. The OpenBao Helm chart mounts each entry in extraVolumes at path + name (e.g. with path: /openbao/tls and name: openbao-server-tls the secret is mounted at /openbao/tls/openbao-server-tls/). The listener tls_cert_file and tls_key_file must use that full path.

Be sure to change the Route host in the Helm values to the one you are using. In addition, make sure that the CA certificate is added correctly to the Route. Do not add any extra lines or spaces and do not forget the | after destinationCACertificate: and the -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- lines.

Step 6: Upgrade the Helm Release

Perform a Helm upgrade so the new volumes and configuration are applied. Pods will restart and pick up TLS; Raft will use HTTPS for join and replication.

helm upgrade openbao openbao/openbao \
--namespace openbao \
--values openbao-ha-values-tls.yaml

Watch the rollout. The first pod (openbao-0) will log "raft retry join initiated" and stay 0/1 Ready until it is initialized and unsealed (new cluster) or until it forms quorum (existing cluster).

New cluster: initialize and unseal openbao-0

openbao-0 will not become Ready until it is initialized and unsealed. Fetch the certificate, use port-forward and talk to OpenBao over HTTPS with the CA:

# 1. Save the CA cert (for BAO_CACERT)
oc get secret openbao-ca-secret -n openbao -o jsonpath='{.data.ca\.crt}' | base64 -d > openbao-ca.crt
# 2. Port-forward to openbao-0 (background)
oc port-forward openbao-0 8200:8200 -n openbao &
# 3. Use HTTPS and CA cert
export BAO_ADDR='https://127.0.0.1:8200'
export BAO_CACERT="$PWD/openbao-ca.crt"
# 4. Check status, then initialize (only once) and unseal 3 times
bao status
bao operator init -key-shares=5 -key-threshold=3 -format=json > openbao-init.json
bao operator unseal
bao operator unseal
bao operator unseal
bao status

After unsealing, openbao-0 becomes leader and goes 1/1 Ready. Then start openbao-1 and openbao-2 and join or unseal them as in Part 3 (use https://).

Perform the following steps for the other pods - join the raft cluster and unseal them (3 times):

oc exec -ti openbao-1 -- bao operator raft join https://openbao-0.openbao-internal:8200
# 3 times with 3 different unseal keys
oc exec -ti openbao-1 -- bao operator unseal
Unseal Key (will be hidden):

Repeat the same steps for openbao-2.

Existing cluster: re-unseal after restart

The existing cluster is already initialized. Re-unseal the pods (and re-join the raft cluster if necessary).

oc get pods -n openbao -w
oc exec -ti openbao-1 -- bao operator raft join https://openbao-0.openbao-internal:8200
oc exec -ti openbao-1 -- bao operator unseal # three times; same for openbao-2

Repeat the same steps for openbao-2.

Step 7: Verify and Use HTTPS from Clients

Verify server health over HTTPS (from inside the cluster):

curl -k https://openbao.apps.cluster.example.com/v1/sys/health | jq

Be sure to use the Route host in the Helm values.

From your workstation, use the Route URL with HTTPS. If the Route host is signed by your internal CA, set BAO_CACERT to the CA file:

export BAO_ADDR='https://openbao.apps.cluster.example.com'
export BAO_CACERT='/path/to/openbao-ca.crt'
bao status

Be sure to use the Route host in the Helm values.

Agent Injector: New pods that use the injector should start without webhook certificate errors. Check injector logs if you see TLS or certificate errors:

oc logs -n openbao -l app.kubernetes.io/name=openbao-agent-injector -f

Troubleshooting

MutatingWebhookConfiguration conflict (vault-k8s)

conflict occurred while applying object ... MutatingWebhookConfiguration: Apply failed with 1 conflict: conflict with "vault-k8s" using ... .webhooks[name="vault.hashicorp.com"].clientConfig.caBundle
this is a server-side apply (SSA) field ownership conflict. The OpenBao chart uses the same webhook name (vault.hashicorp.com) as the HashiCorp Vault agent injector for annotation compatibility. The clientConfig.caBundle field is still owned by a previous manager (e.g. a prior Vault Helm release or the Vault injector), so Helm cannot update it when you change the injector certificate.

Fix option 1 – Delete the webhook and re-upgrade (recommended)

Remove the MutatingWebhookConfiguration so Helm can recreate it and own all fields. There will be a short window where the injector webhook is missing (new pods requesting injection may fail until the upgrade completes).

# Replace RELEASE_NAME with your Helm release name (e.g. openbao)
RELEASE_NAME=openbao
oc delete mutatingwebhookconfiguration ${RELEASE_NAME}-agent-injector-cfg
# Re-run the upgrade
helm upgrade ${RELEASE_NAME} openbao/openbao \
--namespace openbao \
--values openbao-ha-values-tls.yaml

Fix option 2 – Force takeover of the conflicting field

If you cannot delete the webhook (e.g. in production), take over the caBundle field with server-side apply, then run the Helm upgrade again:

Export the MutatingWebhookConfiguration, set webhooks[0].clientConfig.caBundle to your injector CA (base64 PEM from openbao-ca-secret), then re-apply with --server-side --force-conflicts:

# Get the current object and the new CA bundle
oc get mutatingwebhookconfiguration openbao-agent-injector-cfg -o yaml > mwc.yaml
CA_BUNDLE=$(oc get secret openbao-ca-secret -n openbao -o jsonpath='{.data.ca\.crt}')
# Edit mwc.yaml: set .webhooks[0].clientConfig.caBundle to the value of CA_BUNDLE (no quotes in YAML).
# Then apply with force-conflicts so Helm can later manage it:
oc apply -f mwc.yaml --server-side --force-conflicts
# Re-run the Helm upgrade
helm upgrade openbao openbao/openbao --namespace openbao --values openbao-ha-values-tls.yaml

If you still have HashiCorp Vault’s agent injector installed on the same cluster, ensure only one injector is active for a given namespace (e.g. use namespace selectors) or uninstall the Vault injector to avoid two webhooks with the same name in different resources.

Route / UI – tls: bad record MAC

Pod logs show tls: bad record MAC from the router IP. The Route is likely using edge termination (HTTP to pod). Fix: Use reencrypt and set destinationCACertificate (Step 5).

oc get route openbao -n openbao -o jsonpath='{.spec.tls.termination}'

If the OpenBao UI does not load and the pod logs show:

http: TLS handshake error from 10.x.x.x:xxxxx: local error: tls: bad record MAC

the traffic is coming from the OpenShift router (the IP is typically a cluster pod IP).

Cause: The Route is likely using edge termination: the router terminates TLS at the edge and sends plain HTTP to the pod. The pod expects HTTPS, so the TLS layer receives non-TLS data and reports "bad record MAC".

Fix: Use reencrypt (or passthrough) and set destinationCACertificate so the router talks HTTPS to the pod. See Step 5 above. Quick check:

oc get route openbao -n openbao -o jsonpath='{.spec.tls.termination}'
# Must be "reencrypt" or "passthrough", not "edge"

If the output is edge, patch the Route to reencrypt and set spec.tls.destinationCACertificate to the CA PEM (Step 5).

TLS handshake / 127.0.0.1 certificate errors

If you see in the pod logs:

remote error: tls: bad certificate (from 127.0.0.1)

or when running:

oc exec -ti openbao-0 — bao operator raft join https://…;: x509: cannot validate certificate for 127.0.0.1 because it doesn’t contain any IP SANs

The server certificate does not include 127.0.0.1 (and optionally ::1) as Subject Alternative Names. Readiness/liveness probes and in-pod bao commands connect to the listener on 127.0.0.1, so the certificate must include these IP SANs.

Fix: Add ipAddresses to the OpenBao server Certificate and let cert-manager re-issue:

# Edit the Certificate (or re-apply the YAML from Step 3 with ipAddresses added)
oc edit certificate openbao-server-tls -n openbao
# Add under spec:
# ipAddresses:
# - 127.0.0.1
# - "::1"
# cert-manager will issue a new cert; wait until the secret is updated
oc get certificate openbao-server-tls -n openbao
# Restart OpenBao pods so they load the new cert
oc rollout restart statefulset/openbao -n openbao

Resources

The Guide to OpenBao - OpenShift Deployment with Helm - Part 3

Fri, 13 Feb 2026 00:00:00 +0000

After understanding standalone installation in Part 2, it is time to deploy OpenBao on OpenShift/Kubernetes using the official Helm chart. This approach provides high availability, Kubernetes-native management, and seamless integration with the OpenShift ecosystem.

Introduction

Deploying OpenBao on OpenShift/Kubernetes offers several advantages:

High Availability: Multiple replicas with automatic failover
Kubernetes-native: Managed by standard Kubernetes primitives
Persistent Storage: Data survives pod restarts via PVCs
Integration: Works with Kubernetes service accounts for authentication
Scalability: Easy to scale and manage

The official OpenBao Helm chart supports multiple deployment modes:

Dev: Single server, in-memory storage (testing only)
Standalone: Single server, persistent storage
HA: Multiple servers with Raft consensus (recommended)
External: Connect to external OpenBao cluster

Integrations

Currently, OpenBao supports the following integrations that help seamlessly load secrets into applications without the need to modify the application code:

Agent Injector: A mutating webhook that automatically injects a sidecar container that retrieves and renews secrets.
CSI Provider: A (vendor neutral) CSI driver that mounts secrets as volumes.

Prerequisites

Before deploying, ensure you have:

OpenShift 4.12+ or Kubernetes 1.30+
Helm 3.6+
oc or kubectl CLI configured
The OpenBao CLI (bao) for initialization and unsealing (see OpenBao installation)
Cluster-admin privileges (for initial setup)
A storage class that supports ReadWriteOnce PVCs

# Verify prerequisites
oc version
helm version
# Check available storage classes
oc get storageclass

Adding the Helm Repository

First, add the OpenBao Helm repository:

# Add the OpenBao Helm repository
helm repo add openbao https://openbao.github.io/openbao-helm
# Update repository cache
helm repo update
# Search for available charts
helm search repo openbao
# Expected output:
# NAME CHART VERSION APP VERSION DESCRIPTION
# openbao/openbao 0.x.x 2.x.x Official OpenBao Helm chart

According to the official documentation, the Helm chart is new and under significant development. It should always be run with --dry-run before any install or upgrade to verify changes.

Creating the Namespace

Create a dedicated namespace for OpenBao:

While I am using the oc CLI, you can also use the kubectl CLI as in-place replacement.

oc new-project openbao

Deployment Mode: High Availability (Recommended)

For production environments, deploy in HA mode with Raft. This is the recommended deployment mode for production and is straightforward to achieve on Kubernetes and OpenShift. The following values definitions will use OpenShift specific settings. I will mark them in the callouts.

We will first create a values file, deploy openbao and then discuss what must be done to activate all OpenBao pods.

Create HA Values File

Create openbao-ha-values.yaml:

The values file is based on the official values file from that chart, but only modified values or important changes are listed here.

global:
# Enable OpenShift-specific settings
openshift: true (1)
server:
# High Availability configuration
ha:
enabled: true (2)
replicas: 3 (3)
# Raft storage configuration
raft:
enabled: true (4)
setNodeId: true
config: | (5)
ui = true
listener "tcp" {
tls_disable = 1
address = "[::]:8200"
cluster_address = "[::]:8201"
telemetry {
unauthenticated_metrics_access = "true"
}
}
storage "raft" {
path = "/openbao/data"
retry_join {
leader_api_addr = "http://openbao-0.openbao-internal:8200"
}
retry_join {
leader_api_addr = "http://openbao-1.openbao-internal:8200"
}
retry_join {
leader_api_addr = "http://openbao-2.openbao-internal:8200"
}
}
service_registration "kubernetes" {}
telemetry {
prometheus_retention_time = "30s"
disable_hostname = true
}
route:
enabled: true (6)
host: openbao.apps.cluster.example.com (7)
tls:
termination: edge
# Resource requests and limits
resources:
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 1Gi
cpu: 1000m
# Persistent volume for data
dataStorage: (8)
enabled: true
size: 10Gi
# storageClass: "gp3-csi"
# Injector configuration
injector:
enabled: true
replicas: 2 # HA for the injector too (9)
# UI configuration
ui: (10)
enabled: true

1	OpenShift specific: Activate OpenShift Mode: Critical setting, if you install on OpenShift. It adjusts the Helm chart to use Routes instead of Ingress and modifies RoleBindings to work with OpenShift’s stricter authentication.
2	High Availability (HA): Deploys OpenBao as a StatefulSet rather than a Deployment.
3	Raft Consensus Quorum: Sets the cluster size to 3. Raft requires an odd number of nodes to handle leader elections and avoid split-brain scenarios. This cluster can survive the loss of 1 node.
4	Integrated Raft Storage: Enables the internal Raft storage backend, removing the need for external dependencies like Consul or Etcd.
5	Server Configuration (HCL): The actual OpenBao server configuration file. Note that tls_disable = 1 is used because the OpenShift Route handles TLS termination at the edge, passing unencrypted traffic to the pod.
6	OpenShift specific: Route: Tells Helm to create an OpenShift Route object automatically.
7	OpenShift specific: Host: The external DNS address where users and applications will access the OpenBao API and UI.
8	Persistent Storage: Allocates a 10Gi Persistent Volume Claim (PVC) for each of the 3 pods to store encrypted data and Raft logs.
9	Injector Redundancy: Runs 2 replicas of the sidecar injector. If the injector service is down, new application pods attempting to start with secrets will fail.
10	Web UI: Enables the graphical dashboard service.

Deploy HA Cluster

# Deploy with HA values
helm install openbao openbao/openbao \
--namespace openbao \
--values openbao-ha-values.yaml

Verify the Deployment

oc get pods -n openbao

This will result in the following output:

NAME READY STATUS RESTARTS AGE
openbao-0 0/1 Running 0 60s (1)
openbao-agent-injector-xxx 1/1 Running 0 60s
openbao-agent-injector-yyy 1/1 Running 0 60s

1	Only 1 openbao pod (instead of 3) is running, and the pod is not in "ready" state.

As you can see, only one OpenBao pod exists so far, and it is not in "ready" state. This is because OpenBao is not yet initialized and unsealed. Once that is done, the other pods will appear and join the cluster.

The OpenBao pods show 0/1 ready because they are sealed and need initialization.

This is also indicated in the logs of the openbao pod:

2026-02-13T14:44:52.587Z [ERROR] core: failed to get raft challenge: leader_addr=http://openbao-0.openbao-internal.openbao.svc:8200
error=
| error during raft bootstrap init call: Error making API request.
|
| URL: PUT http://openbao-0.openbao-internal.openbao.svc:8200/v1/sys/storage/raft/bootstrap/challenge
| Code: 503. Errors:
|
| * Vault is sealed (1)
2026-02-13T14:44:52.588Z [ERROR] core: failed to get raft challenge: leader_addr=http://openbao-2.openbao-internal.openbao.svc:8200 error="error during raft bootstrap init call: Put \"http://openbao-2.openbao-internal.openbao.svc:8200/v1/sys/storage/raft/bootstrap/challenge\": dial tcp: lookup openbao-2.openbao-internal.openbao.svc on 172.30.0.10:53: no such host" (2)
2026-02-13T14:44:52.589Z [ERROR] core: failed to get raft challenge: leader_addr=http://openbao-1.openbao-internal.openbao.svc:8200 error="error during raft bootstrap init call: Put \"http://openbao-1.openbao-internal.openbao.svc:8200/v1/sys/storage/raft/bootstrap/challenge\": dial tcp: lookup openbao-1.openbao-internal.openbao.svc on 172.30.0.10:53: no such host"
2026-02-13T14:44:52.589Z [ERROR] core: failed to retry join raft cluster: retry=2s err="failed to get raft challenge"

1	Vault is sealed: This means that the OpenBao service is not yet initialized and unsealed.
2	No such host: This means that the OpenBao service is not yet available.

Initializing and Unsealing OpenBao

After deployment, OpenBao needs to be initialized and unsealed. This is done on the first pod. Once this is done, the other pods will appear and can join the cluster. We will create a local portforwarding to the first pod to initialize it.

Initialize the Cluster

Create a local port forwarding to the first pod

oc port-forward openbao-0 8200:8200 -n openbao &

Set environment variable
```
export BAO_ADDR='http://127.0.0.1:8200'
```

Check status

bao status

This will result in the following output:

Key Value
--- -----
Seal Type shamir
Initialized false (1)
Sealed true (2)
Total Shares 0
Threshold 0
Unseal Progress 0/0
Unseal Nonce n/a
Version 2.5.0
Build Date 2026-02-04T16:19:33Z
Storage Type raft
HA Enabled true (3)

1	Not yet initialized
2	Vault is sealed
3	High Availability is enabled

Initialize the cluster with 5 key shares and 3 threshold
```
bao operator init -key-shares=5 -key-threshold=3 -format=json > openbao-init.json
```
This will create the file openbao-init.json with the unseal keys and root token.
```
cat openbao-init.json
```

Take care of the openbao-init.json file. It contains the unseal keys and root token!

Unseal Pod openbao-0

After initialization, we need to unseal the first pod. This is done by providing 3 different unseal keys. (threshold is 3)

# Unseal openbao-0 (3 times with different keys)
bao operator unseal # Enter first key
bao operator unseal # Enter second key
bao operator unseal # Enter third key

This unseals openbao-0, which can be verified with the command bao status.

Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false (1)
Total Shares 5
Threshold 3
Version 2.5.0
Build Date 2026-02-04T16:19:33Z
Storage Type raft
Cluster Name vault-cluster-80c01167
Cluster ID b81ecb85-9751-655a-95b7-69463dd13241
HA Enabled true
HA Cluster https://openbao-0.openbao-internal:8201
HA Mode active
Active Since 2026-02-13T14:46:13.292643992Z
Raft Committed Index 29
Raft Applied Index 29

1	Vault is unsealed

This makes openbao-0 ready and openbao-1 is trying to start:

oc get pods
NAME READY STATUS RESTARTS AGE
openbao-0 1/1 Running 0 2m10s (1)
openbao-1 0/1 Running 0 14s (2)
openbao-agent-injector-98769cf97-r4stk 1/1 Running 0 2m12s
openbao-agent-injector-98769cf97-xldgm 1/1 Running 0 2m12s

1	openbao-0 is ready
2	openbao-1 is trying to start and wants to join the Raft cluster

Activate Pod openbao-1

Since OpenBao is already initialized, we can skip the initialization step. However, we must join the Raft cluster.

oc exec -ti openbao-1 -- bao operator raft join http://openbao-0.openbao-internal:8200

Once done, we can unseal openbao-1. We need to provide 3 different unseal keys again. Execute the following command 3 times:

oc exec -ti openbao-1 -- bao operator unseal
Unseal Key (will be hidden):

Now openbao-1 is ready and openbao-2 is trying to start:

oc get pods
NAME READY STATUS RESTARTS AGE
openbao-0 1/1 Running 0 3m48s
openbao-1 1/1 Running 0 112s
openbao-2 0/1 Running 0 18s
openbao-agent-injector-98769cf97-r4stk 1/1 Running 0 3m50s
openbao-agent-injector-98769cf97-xldgm 1/1 Running 0 3m50s

Activate Pod openbao-2

Now we need to repeat the previous steps for openbao-2.

oc exec -ti openbao-2 -- bao operator raft join http://openbao-0.openbao-internal:8200
# Run 3 times (enter a different unseal key each time):
oc exec -ti openbao-2 -- bao operator unseal

Verify Raft Cluster

You can verify the Raft cluster by logging in with the root token and then checking the Raft peer list.

oc exec -ti openbao-0 -- bao login

Check the peer list:

# Check Raft peer list
oc exec -ti openbao-0 -- bao operator raft list-peers
Node Address State Voter
---- ------- ----- -----
openbao-0 openbao-0.openbao-internal:8201 leader true
openbao-1 openbao-1.openbao-internal:8201 follower true
openbao-2 openbao-2.openbao-internal:8201 follower true

Accessing the UI

Once unsealed, access the OpenBao UI:

Via Route (Production)

oc get route openbao -n openbao
# Open browser: https://openbao.apps.cluster.example.com

Figure 1. OpenBao Login Form

Upgrading OpenBao

Keep your secrets management up to date. To upgrade an existing deployment:

# Update Helm repository
helm repo update
# Check available versions
helm search repo openbao --versions
# Upgrade with your values file
helm upgrade openbao openbao/openbao \
--namespace openbao \
--values openbao-ha-values.yaml
# Watch the rolling update
oc get pods -n openbao -w

After upgrade, pods may need to be unsealed again if they restart.

Troubleshooting

Pods Not Starting

# Check pod status
oc describe pod openbao-0 -n openbao
# Check pod logs
oc logs openbao-0 -n openbao
# Check events
oc get events -n openbao --sort-by='.lastTimestamp'

PVC Issues

# Check PVC status
oc get pvc -n openbao
# If pending, check storage class
oc describe pvc data-openbao-0 -n openbao

Raft Join Failures

If pods cannot join the Raft cluster:

# Check internal DNS resolution
oc exec -it openbao-0 -n openbao -- nslookup openbao-internal
# Check connectivity between pods
oc exec -it openbao-0 -n openbao -- wget -O- http://openbao-1.openbao-internal:8200/v1/sys/health

What Should Be Considered Next?

Securely store the unseal keys and root token
Configure pod anti-affinity for true HA
Consider auto-unseal for operational ease (upcoming article)
Put everything into a GitOps pipeline

Conclusion

You now have OpenBao running on OpenShift in high-availability mode. This deployment:

Survives pod failures and restarts
Uses Raft for distributed consensus
Integrates with OpenShift security model
Is ready for production use (after unsealing automation)

Key points to remember:

Use HA mode for production
Store unseal keys securely
Configure pod anti-affinity for true HA
Consider auto-unseal for operational ease (upcoming article)

Resources

Hosted Control Planes behind a Proxy

Mon, 15 Dec 2025 00:00:00 +0000

Recently, I encountered a problem deploying a Hosted Control Plane (HCP) at a customer site. The installation started successfully—etcd came up fine—but then it just stopped. The virtual machines were created, but they never joined the cluster. No OVN or Multus pods ever started. The only meaningful message in the cluster-version-operator pod logs was:

I1204 08:22:35.473783 1 status.go:185] Synchronizing status errs=field.ErrorList(nil) status=&cvo.SyncWorkerStatus{Generation:1, Failure:(*payload.UpdateError)(0xc0006313b0), Done:575, Total:623,

This message did appear over and over again.

Problem Summary

etcd started successfully, but the installation stalled
API server was running
VMs started but never joined the cluster
No OVN or Multus pods ever started
The installation was stuck in a loop

Troubleshooting

Troubleshooting was not straightforward. The cluster-version-operator logs provided the only clue. However, the VMs were already running, so we could log in to them—and there it was, the reason for the stalled installation.

Connect to a VM

To connect to a VM, use the virtctl command. This connects you to the machine as the core user:

You can download virtctl from the OpenShift Downloads page in the OpenShift web console.

You need the SSH key for the core user.

virtctl ssh -n clusters-my-hosted-cluster core@vmi/my-node-pool-dsj7z-sss8w (1)

1	Replace my-hosted-cluster with the name of your hosted cluster and my-node-pool-dsj7z-sss8w with the name of your VM.

Verify the Problem

Once logged in, check the journalctl -xf output. In case of a proxy issue, you’ll see an error like this:

> Dec 10 06:23:09 my-node-pool2-gwvjh-rwtx8 sh[2143]: time="2025-12-10T06:23:09Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:500704de3ef374e61417cc14eda99585450c317d72f454dda0dadd5dda1ba57a: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 3.209.93.201:443: i/o timeout"

So we have a timeout when trying to pull the image from the container registry. This is a classic proxy issue. When you try curl or manual pulls you will get the same message.

Configuring the Proxy for the Hosted Control Plane

The big question is: How do we configure the proxy for the Hosted Control Plane?

This is not well documented yet—in fact, it’s barely documented at all.

The Hosted Control Plane is managed through a custom resource called HostedCluster, and that’s exactly where we configure the proxy. The upstream documentation at HyperShift Documentation explains that you can add a configuration section to the resource. Let’s do that:

Update the HostedCluster resource with the following configuration:

spec:
configuration:
proxy:
httpProxy: http://proxy.example.com:8080 (1)
httpsProxy: https://proxy.example.com:8080 (2)
noProxy: .cluster.local,.svc,10.128.0.0/14,127.0.0.1,localhost (3)
trustedCA:
name: user-ca-bundle (4)

1	HTTP proxy URL
2	HTTPS proxy URL
3	Comma-separated list of domains, IPs, or CIDRs to exclude from proxy routing
4	OPTIONAL: Name of a ConfigMap containing a custom CA certificate bundle

If you’re using a custom CA, create the ConfigMap beforehand or alongside the HostedCluster resource. The ConfigMap must contain a key named ca-bundle.crt with your CA certificate(s).

But wait there is more…

If you are using such a proxy, which is injecting it’s own certificate, then you probably saw the following: The Release Image is not available when you try to deploy a new Hosted Control Plane. The drop down in the UI is empty and you can’t select a release image.

This happens because the Pod that is trying to fetch the image is not able to connect to Github.

You can test this with the following commands:

oc rsh cluster-image-set-controller-XXXXX -n multicluster-engine (1)
curl -v https://github.com/stolostron/acm-hive-openshift-releases.git/info/refs?service=git-upload-pack (2)

1	Replace XXXXX with the name of the Pod cluster-image-set-controller in the multicluster-engine namespace
2	Try to execute the curl command to see if you can connect to Github

If this command fails with a certificate error, then you need to add the certificate to the Pod/Deployment. A Configmap called trusted-ca-bundle should already exist in the multicluster-engine namespace. If not, it must be created with the certificate chain of your Proxy (key: ca-bundle.crt)

The following command will add the ConfigMap to the deployment.

oc -n multicluster-engine set volume deployment/cluster-image-set-controller --add --type configmap --configmap-name trusted-ca-bundle --name trusted-ca-bundle --mount-path /etc/pki/tls/certs/ --overwrite

Wait a couple of minutes after the Pods has been restarted. It will try to download the release images from Github again.

You can check the logs of the Pod to see if the download was successful.

oc logs cluster-image-set-controller-XXXXX -n multicluster-engine (1)

1	Replace XXXXX with the name of the Pod cluster-image-set-controller in the multicluster-engine namespace

That should do it, you should now be able to select a release image and deploy a new Hosted Control Plane.

Figure 1. Drow Down with Release Images

Summary

That’s actually everything you need for proxy configuration with Hosted Control Planes. Hopefully, the official OpenShift documentation will be updated soon to include this information. red Hat created two tickets to track the progress of this:

Helm Charts Repository Updates

Fri, 12 Dec 2025 00:00:00 +0000

This page shows the latest updates to the stderr.at Helm Charts Repository. The charts are designed for OpenShift and Kubernetes deployments, with a focus on GitOps workflows using Argo CD.

The content below is dynamically loaded from the Helm repository and always shows the most recent changes.

Quick Links

Resource

Link

Helm Repository

https://charts.stderr.at/

GitHub Source

https://github.com/tjungbauer/helm-charts

ArtifactHub

https://artifacthub.io/packages/search?repo=tjungbauer

Example GitOps Repo

https://github.com/tjungbauer/openshift-clusterconfig-gitops

The Hitchhiker's Guide to Observability - Limit Read Access to Traces - Part 8

Sat, 06 Dec 2025 00:00:00 +0000

In the previous articles, we deployed a distributed tracing infrastructure with TempoStack and OpenTelemetry Collector. We also deployed a Grafana instance to visualize the traces. The configuration was done in a way that allows everybody to read the traces. Every system:authenticated user is able to read ALL traces. This is usually not what you want. You want to limit trace access to only the appropriate namespace.

In this article, we’ll limit the read access to traces. The users of the team-a namespace will only be able to see their own traces.

Prerequisites

Before we begin, make sure you have:

TempoStack deployed and configured (from Part 2)
Team-a namespace with traces flowing (from Part 4)
A separate user for the team-a namespace.

Verify Trace Access

Let’s verify what a user can see when they are authenticated. We have user1 who is a member of the team-a namespace. Let’s log in as this user and verify what they can see.

Navigate to Observability > Traces and select the tempostack/simplest datasource:

You should see the traces for the team-a namespace. This is fine—that’s what we want.

But now let’s change the tenant to tenantB and verify what the user can see.

As you can see, the user can see traces from the tenantB namespace, although they are a member of the team-a namespace. This is not what we want. We want to limit trace access to only the appropriate namespace.

The Original RBAC Configuration

In Part 2, we created a ClusterRoleBinding to grant read access to traces for everybody who is authenticated.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: tempostack-traces-reader
subjects:
- kind: Group
apiGroup: rbac.authorization.k8s.io
name: 'system:authenticated'
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: tempostack-traces-reader

This ClusterRoleBinding allows system:authenticated users to read all traces from all tenants and is bound to the ClusterRole tempostack-traces-reader.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tempostack-traces-reader
rules:
- verbs:
- get
apiGroups:
- tempo.grafana.com
resources:
- tenantA
- tenantB
resourceNames:
- traces

This is the configuration we want to change. We want to limit read access to traces to only the appropriate namespace.

Change the RBAC Configuration

We will change the RBAC configuration to limit read access to traces. To do so, we will first create a new ClusterRole for the team-a namespace and bind it to users of that namespace.

Create the new ClusterRole:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tempostack-traces-reader-team-a (1)
rules:
- verbs:
- get
apiGroups:
- tempo.grafana.com
resources:
- tenantA (2)
resourceNames:
- traces

1	Name of the ClusterRole, now with the prefix team-a.
2	The Tenant that is allowed to read the traces. This time it is only the tenantA tenant.

Create the new ClusterRoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: tempostack-traces-reader-team-a (1)
subjects:
- kind: User (2)
apiGroup: rbac.authorization.k8s.io
name: user1
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: tempostack-traces-reader-team-a (3)

1	Name of the ClusterRoleBinding, now with the prefix team-a.
2	The User that is allowed to read the traces. In this example: user1.
3	The ClusterRole that is allowed to read the traces.

In this example, we are using a single user. In a real-world scenario, you would most likely have a group of users. In that case, you would use a Group instead of a User.

Modify the Original ClusterRole:

As a final step, we need to modify the original ClusterRole to remove tenantB from the list of tenants that are allowed to read the traces.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tempostack-traces-reader
rules:
- verbs:
- get
apiGroups:
- tempo.grafana.com
resources:
- tenantB (1)
resourceNames:
- traces

1	Only tenantA remains. Remove tenantB from the list of resources.

This removes the permission to read traces from the tenantB namespace for the system:authenticated group.

Eventually, the original ClusterRoleBinding might be deleted once every user has been assigned to a separate ClusterRole.

Verify the Changes

Let’s see what these changes do. As user1, you should still be able to see the traces from the tenantA namespace.

But if you change the tenant to tenantB, you should see an error message like this:

The user will still see the list of tenants. Hopefully, this will be fixed in a future version of TempoStack.

The Guide to OpenBao - Standalone Installation - Part 2

Thu, 12 Feb 2026 00:00:00 +0000

In the previous article, we introduced OpenBao and its core concepts. Now it is time to get our hands dirty with a standalone installation. This approach is useful for testing, development environments, edge deployments, or scenarios where Kubernetes is not available.

Introduction

While OpenBao shines in Kubernetes environments, understanding the standalone installation helps you grasp the fundamentals. This knowledge is valuable whether you are:

Learning OpenBao before deploying to production
Running OpenBao outside of Kubernetes (edge, legacy systems)
Debugging issues in containerized deployments
Setting up a development environment

Installation Methods Overview

OpenBao can be installed through multiple methods:

Method

Best For

Complexity

Package managers (apt, dnf, brew)

Production Linux/macOS systems

Low

Container images (Podman/Docker)

Quick testing, isolated environments

Low

Binary download

Air-gapped environments

Low

Source compilation

Custom builds, development

Medium

For this article, we will focus on macOS for local testing with binary and container image (using Podman) and Red Hat Enterprise Linux to set up an example production-ready server. The deployment on the Kubernetes environment will be discussed in the next article.

Method 1: Package Manager Installation

RHEL/Fedora/CentOS

On RHEL or CentOS, before you can install OpenBao you need to install the EPEL repository. To do so, you first need to enable the Code Ready Repository. The following commands will do the trick. Be sure that your system is registered, in case of RHEL.

If you get the error "Repositories disabled by configuration." you need to tell the subscription manager that you want to manage the repositories. This can be done permanently or temporarily. You can use the command: sudo subscription-manager config --rhsm.manage_repos=1 to do so.

RHEL

Enable Code Ready Repository

sudo subscription-manager repos --enable codeready-builder-for-rhel-9-$(arch)-rpms

Install EPEL Repository

sudo dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm

Install OpenBao
```
sudo dnf install -y openbao
```

CentOS

Enable Code Ready Repository on CentOS

sudo dnf config-manager --set-enabled crb

Install EPEL Repository

sudo dnf install epel-release epel-next-release

Install OpenBao
```
sudo dnf install -y openbao
```

macOS (Homebrew)

Install OpenBao
```
brew install openbao
```

Verify Installation

After installation, verify OpenBao is available:

bao version
# Output:
OpenBao v2.5.0 (bcbb6036ec2b747bceb98c7706ce9b974faa1b23), built 2026-02-04T15:57:17Z (cgo)

The OpenBao CLI command is bao, not vault. This distinguishes it from HashiCorp Vault.

Start Development Server

To start a development environment, you can use the following command.

This is NOT suitable for production, it is just a test to evaluate the basic concepts of OpenBao.

bao server -dev -dev-root-token-id="dev-only-token"

This will start the server. The UI is accessible at http://localhost:8200 where you can login using the root token dev-only-token.

Method 2: Container Image

For quick testing or isolated environments, container images are ideal. Luckily, OpenBao offers several types of containers suitable for any environment. We will use the image hosted on quay.io, which is based on RHEL UBI and can be found at: quay.io/openbao/openbao-ubi

Using Podman

The following command will fetch the image and start the container.

This is NOT suitable for production, it is just a test to evaluate the basic concepts of OpenBao.

# Run in dev mode (for testing only!)
podman run --rm -d \
--name openbao-dev \
-p 8200:8200 \
-e 'BAO_DEV_ROOT_TOKEN_ID=dev-only-token' \
-e 'BAO_DEV_LISTEN_ADDRESS=0.0.0.0:8200' \
quay.io/openbao/openbao-ubi:latest server -dev

# Verify it is running
podman logs -f openbao-dev

This will start the server. The UI is accessible at http://localhost:8200 where you can login using the root token dev-only-token.

If you prefer to use Docker, simply replace podman with docker in the commands.

Dev Mode: Quick Start for Testing

Dev mode is the fastest way to start using OpenBao for learning and testing.

The characteristics in this mode are:

In-memory storage (data lost on restart)
Automatically initialized and unsealed
Root token printed to stdout
TLS disabled
Single server (no HA)

Let’s create an example secret and try to retrieve it again.

Authenticate against OpenBao

export VAULT_TOKEN="dev-only-token"
export BAO_ADDR='http://127.0.0.1:8200'

Create Secret

curl \
--header "X-Vault-Token: $VAULT_TOKEN" \
--header "Content-Type: application/json" \
--request POST \
--data '{"data": {"password": "OpenBao123"}}' \
$BAO_ADDR/v1/secret/data/my-secret-password &&
echo "Secret written successfully."

Retrieve Secret

curl --header "X-Vault-Token: $VAULT_TOKEN" \
$BAO_ADDR/v1/secret/data/my-secret-password | jq '.data.data'

You should see the password that was created before:

{
"password": "OpenBao123"
}

Check Status of OpenBao Server

bao status

This will give you the status of your running OpenBao instance.

Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 2.5.0
Build Date 2026-02-04T15:57:17Z
Storage Type inmem
Cluster Name vault-cluster-421b2431
Cluster ID 6d42dcd2-e399-211e-999a-49b1874cc8ce
HA Enabled false

Hardening your System

OpenBao does a good job to secure your secrets, however, memory paging (or swap) can undermine the protection. Your OS should either have swap disabled completely or encrypt the swap space.

As I am testing on macOS, the swap space is encrypted out of the box.

However, OpenBao has documented what must be done for various operating systems at Post-installation hardening

Production Standalone Setup

For a proper standalone installation, follow these steps:

Step 1: Create Configuration File

Create /etc/openbao.d/openbao.hcl:

# Full configuration for standalone OpenBao server
# Cluster name for identification
cluster_name = "openbao-standalone"
# Storage backend using integrated Raft
storage "raft" {
path = "/var/lib/openbao/data"
node_id = "node1"
}
# HTTP listener (for internal communication)
listener "tcp" {
address = "0.0.0.0:8200"
cluster_address = "0.0.0.0:8201"
tls_disable = false
tls_cert_file = "/etc/openbao.d/tls/tls.crt"
tls_key_file = "/etc/openbao.d/tls/tls.key"
}
# API address for clients
api_addr = "https://openbao.example.com:8200"
# Cluster address for raft communication
cluster_addr = "https://openbao.example.com:8201"
# UI enabled
ui = true
# Logging
log_level = "info"
log_file = "/var/log/openbao/openbao.log"
# Disable memory locking (enable in production if possible)
disable_mlock = true
# Telemetry (optional)
telemetry {
prometheus_retention_time = "30s"
disable_hostname = true
}

Step 2: Generate TLS Certificates

For production, you should use proper certificates. For testing, create self-signed ones:

This is only for testing purposes. In production, you should use proper certificates.

First, prepare the TLS directory if it does not exist
```
sudo mkdir -p /etc/openbao.d/tls
```

Then create a configuration file for the TLS certificates:

# Create TLS config
cat <<EOF > openbao.cnf
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no
[req_distinguished_name]
CN = openbao.example.com
[v3_req]
subjectAltName = @alt_names
[alt_names]
DNS.1 = openbao.example.com
IP.1 = 127.0.0.1 (1)
EOF

1	The IP address of the server. In this case we are using localhost, but you can add your IPs here.

Then generate the certificate:

# Generate private key
sudo openssl genrsa -out /etc/openbao.d/tls/tls.key 4096
# Generate certificate signing request
sudo openssl req -new -key /etc/openbao.d/tls/tls.key -out /etc/openbao.d/tls/tls.csr -subj "/CN=openbao.example.com"
# Generate self-signed certificate
sudo openssl x509 -req -days 365 -in /etc/openbao.d/tls/tls.csr -signkey /etc/openbao.d/tls/tls.key -out /etc/openbao.d/tls/tls.crt -extfile openbao.cnf -extensions v3_req
# Set permissions
sudo chown -R openbao:openbao /etc/openbao.d/tls
sudo chmod 600 /etc/openbao.d/tls/tls.key
sudo chmod 755 /etc/openbao.d/tls

Step 3: Create Systemd Service

Create /etc/systemd/system/openbao.service:

[Unit]
Description=OpenBao Secret Management
Documentation=https://openbao.org/docs
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/openbao.d/openbao.hcl
[Service]
User=openbao
Group=openbao
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/bin/bao server -config=/etc/openbao.d/openbao.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
LimitNOFILE=65536
LimitMEMLOCK=infinity
[Install]
WantedBy=multi-user.target

Step 4: Start the Service

# Reload systemd
sudo systemctl daemon-reload
# Enable and start OpenBao
sudo systemctl enable openbao
sudo systemctl start openbao
# Check status
sudo systemctl status openbao
# View logs
sudo journalctl -u openbao -f

Initialize and Unseal OpenBao

After starting OpenBao for the first time, it needs to be initialized and unsealed.

Set Environment Variables

We will do the following commands as root user.

Be sure that the hostname is resolvable. In this case I am using the test domain: openbao.example.com.

# Set the OpenBao address
export BAO_ADDR='https://openbao.example.com:8200'
# If using self-signed certificates
export BAO_CACERT='/etc/openbao.d/tls/tls.crt'

Check Status

You will see that OpenBao is running but not yet initialized. This will be the next step.

bao status
Key Value
--- -----
Seal Type shamir
Initialized false (1)
Sealed true
Total Shares 0
Threshold 0
Unseal Progress 0/0
Unseal Nonce n/a
Version 2.4.4-1.el9
Build Date 2025-11-24
Storage Type file
HA Enabled false

1	Not yet initialized

Initialize OpenBao

Before we use OpenBao, we need to initialize it. This will create the unseal keys and the root token. The unseal keys are used to … well, unseal the OpenBao service. The root token is used as a master key to authenticate against the OpenBao service. This token has access to ALL secrets. Treat this root token with care.

We will initialize OpenBao with default options of 5 key shares and a threshold of 3. This means that we need 3 different unseal keys to unseal the OpenBao service.

bao operator init -key-shares=5 -key-threshold=3 -format=json > /root/openbao-init.json

Store the unseal keys and root token securely! Anyone with these can access all secrets.

The output looks like:

{
"unseal_keys_b64": [
"key1...",
"key2...",
"key3...",
"key4...",
"key5..."
],
"unseal_keys_hex": [...],
"unseal_shares": 5,
"unseal_threshold": 3,
"recovery_keys_b64": [],
"recovery_keys_hex": [],
"root_token": "sbr.xxxxxxxxxxxx"
}

Unseal OpenBao

Now everything is set up and we can use the unseal keys to unseal the OpenBao service.

You need to provide 3 (threshold) different unseal keys:

# First key
bao operator unseal
# Enter first unseal key
# Second key
bao operator unseal
# Enter second unseal key
# Third key
bao operator unseal
# Enter third unseal key

After providing enough keys:

bao status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false (1)
Total Shares 5
Threshold 3
Version 2.4.4-1.el9
Build Date 2025-11-24
Storage Type file
Cluster Name openbao-standalone
Cluster ID d07874ef-df52-e45f-1723-ade333c6d609
HA Enabled false

1	Now unsealed and ready

Now we can login with the root token to the OpenBao service.

Login with Root Token

Since we have not created any users yet, we will use the root token to authenticate.

# Enter the root token from initialization
bao login

Basic Verification

Let us verify the installation works:

List enabled secrets engines

bao secrets list

This lists the enabled secrets engines.

Path Type Accessor Description
---- ---- -------- -----------
cubbyhole/ cubbyhole cubbyhole_84c3d2a5 per-token private secret storage
identity/ identity identity_61809171 identity store
sys/ system system_034ae772 system endpoints used for control, policy and debugging

List enabled auth methods

bao auth list

This lists the enabled authentication methods.

Path Type Accessor Description Version
---- ---- -------- ----------- -------
token/ token auth_token_c7feca17 token based credentials n/a

Create a test secret
```
bao secrets enable -path=secret kv-v2
```
This enables the KV (key/value) secrets engine at the path secret.
```
bao kv put secret/test message="Hello from OpenBao"
```
This creates a test secret at the path secret/test.
```
bao kv get secret/test
```
This retrieves the test secret from the path secret/test.

That’s it for the basic verification. You can now start to use OpenBao in your production environment. Let’s see what we can do in the UI.

Visiting the UI

If you watched the previous commands closely, you will have noticed that the UI has been enabled in the configuration file:

# UI enabled
ui = true

The UI is accessible at http://openbao.example.com:8200. You can login with the root token from the initialization.

Figure 1. OpenBao Login Form

Here you will see several options to explore. For now we are interested in the Secrets Engine section.

Figure 2. OpenBao Secrets Engine

We have created the path "secret" and inside it a secret called "test". We can retrieve it now:

Figure 3. OpenBao Secret Retrieval

Security Hardening Checklist

Before using OpenBao in production you should always consider the following checklist:

Item

Recommendation

TLS

Always enable TLS with valid certificates

Root Token

Revoke root token after initial setup and create admin users instead

Unseal Keys

Distribute to different people/locations, consider auto-unseal

Network

Restrict access with firewall rules

Audit

Enable audit logging

Backups

Regular Raft snapshots

Updates

Keep OpenBao updated

What is Coming Next?

In Part 3, we will deploy OpenBao on OpenShift/Kubernetes using the official Helm chart. This provides:

High availability out of the box
Kubernetes-native management
Integration with OpenShift security features
Persistent storage via PVCs

Conclusion

You now have a working standalone OpenBao installation. This forms the foundation for understanding how OpenBao operates. While standalone mode is useful for testing and edge cases, most production deployments will use Kubernetes, which we will cover next.

Key takeaways:

OpenBao can run standalone or in containers
Dev mode is for testing only
Production requires proper TLS, initialization, and security hardening
Unseal keys must be stored securely

Resources

[Ep.15] OpenShift GitOps - Argo CD Agent

Wed, 14 Jan 2026 00:00:00 +0000

OpenShift GitOps based on Argo CD is a powerful tool to manage the infrastructure and applications on an OpenShift cluster. Initially, there were two ways of deployment: centralized and decentralized (or distributed). Both methods had their own advantages and disadvantages. The choice was mainly between scalability and centralization. With OpenShift GitOps v1.19, the Argo CD Agent was finally generally available. This agent tries to solve this problem by bringing the best of both worlds together. In this quite long article, I will show you how to install and configure the Argo CD Agent with OpenShift GitOps using hub and spoke architecture.

Classic Deployment Models:

Prior to the Argo CD Agent, there were two classic and often used deployment models: centralized and decentralized.

Centralized Model

In a centralized deployment, all changes are applied to a single, central Argo CD instance, often installed on a management cluster. This is the traditional way of deploying Argo CD, at least in my opinion. With this model, you have a single pane of glass to manage all your clusters. You have a single UI and will see all your clusters in one place, which makes this model very convenient when you have multiple clusters. However, the scalability of this model is limited. Organizations with a huge number of clusters or Argo CD applications would hit some boundaries at some point. A sharding configuration would help, but only to a certain extent. The performance would degrade significantly. In addition, this model creates a Single Point of Failure. If this instance is down, the company loses the ability to manage their clusters through Argo CD.

I often saw or used this model for the cluster configuration.

Decentralized Model

In a decentralized deployment, multiple instances of Argo CD, often one for each cluster, are installed. With this approach, the issue with scalability is solved. Moreover, the Single Point of Failure is eliminated as well, since a broken instance will not affect the other instances. However, the disadvantages of this model are that the complexity for the operational teams will increase, since they need to manage multiple instances now. Also, the single pane of glass for management is lost, as there are multiple UIs to manage.

I often saw or used this model for the application deployment.

The - not so secret - Argo CD Agent

The Argo CD Agent, released as generally available in OpenShift GitOps v1.19, is a new way to use Argo CD. It tries to solve the challenges of the classic deployment models by combining the best of both worlds. The Agent allows you to have a single UI in a central control plane, while the application controller is distributed across the fleet of clusters. Agents on the different clusters will communicate with the central Argo CD instance.

The Agent model introduces a hub and spoke architecture:

Control plane cluster (hub) - The control plane cluster is the central cluster that manages the configuration for multiple spokes.
Workload cluster (spoke) - The workload cluster is the cluster that runs the application workloads deployed by Argo CD.

Each Argo CD Agent on a cluster manages the local Argo CD instance and ensures that applications, AppProjects, and secrets remain synchronized with their source of truth.

The official documentation describes a comparison between the classic deployment models and the Argo CD Agent: GitOps Architecture - Argo CD Agent Comparison

Argo CD Agent Modes

The Argo CD Agent supports two modes of operation: Managed and Autonomous. The mode determines where the authoritative source of truth for the Application .spec field resides.

Managed mode — the control plane/hub defines Argo CD applications and their specifications.
Autonomous mode — each workload cluster/spoke defines its own Argo CD applications and their specifications.

A mixed mode is also possible.

Managed Mode

Using the managed mode means that the control plane is the source of truth and is responsible for the Argo CD application resources and their distribution across the different workload clusters. Any change on the hub cluster will be propagated to the spoke clusters. Any changes made on the spoke/workload cluster will be reverted to match the control plane configuration.

Autonomous Mode

Using this mode, the Argo CD applications are defined on the workload clusters, which serve as their own source of truth. The applications are synchronized back to the control plane for observability. Changes made on the workload cluster are not reverted, but will appear on the control plane. On the other hand, you cannot modify applications directly from the control plane.

Security

The Argo CD Agent uses mTLS certificates to communicate between the hub and the spoke clusters.

The certificate must be created and managed by the user.

Argo CD Agent Installation

The agent consists of two components that are responsible to synchronize the Argo CD applications between the hub and the spoke clusters.

Principal - Deployed on the control plane cluster together with Argo CD. Here the central UI (and API) can be found.
Agent - Deployed on the workload clusters to synchronize the Argo CD applications.

The installation of both is done differently. But before we dive into the installation, let’s have a look at the terminology to understand the different components and their roles.

This is a quote from the official documentation (Argo CD Agent Terminologies)

Principal namespace - Specifies the namespace where you install the Principal component. This namespace is not created by default, you must create it before adding the resources in this namespace. In Argo CD Agent CLI commands, this value is provided using the --principal-namespace flag.
Agent namespace - Specifies the namespace hosting the Agent component. This namespace is not created by default, you must create it before adding the resources in this namespace. In Argo CD Agent CLI commands, this value is provided using the --agent-namespace flag.
Context - A context refers to a named configuration in the oc CLI that allows you to switch between different clusters. You must be logged in to all clusters and assign distinct context names for the hub and spoke clusters. Examples for cluster names include principal-cluster, hub-cluster, managed-agent-cluster, or autonomous-agent-cluster.
Principal context - The context name you provide for the hub (control plane) cluster. For example, if you log in to the hub cluster and rename its context to principal-cluster, you specify it in Argo CD Agent CLI commands as --principal-context principal-cluster.
Agent context - The context name you provide for the spoke (workload) cluster. For example, if you log in to a spoke cluster and rename its context to autonomous-agent-cluster, you specify it in Argo CD Agent CLI commands as --agent-context autonomous-agent-cluster.

A Word about the Setup

To create some kind of real customer scenario, I have created two clusters:

The cluster where we installed the Principal component. This will be the Hub/Management/Principal cluster (We have too many words for this…)
A separate cluster that will be the Agent cluster. This will be the Spoke/Workload cluster.

The first one I have installed on AWS, and the second one is a Bare Metal Single node cluster. Both can reach each other via the Internet.

We are using different contexts for the different clusters for the command line. The argocd-agentctl tool knows the flags --principal-context and --agent-context to switch between the different clusters. Be sure to create the resources on the correct cluster.

Prerequisites

Before we start with the installation of the Principal or Agent component, we need to ensure that the following prerequisites are met:

We have two OpenShift test clusters.
Both clusters can reach each other
OpenShift GitOps Operator is already installed (possible configuration modifications are described in the following sections)

Because of my main focus on OpenShift GitOps, we will try to deploy cluster configurations and not just workload. Therefore, the example Argo CD applications will configure cluster settings. (A banner on the top and bottom of the UI)

Configure OpenShift GitOps Subscription on the Hub Cluster

The OpenShift GitOps Operator installs by default an Argo CD instance. In this test we will disable this, as we do not need that instance. Moreover and even more important, we need to tell the Operator for which namespaces it should feel responsible for. In this case, we will tell the Operator to be responsible for all namespaces on the cluster.

We need to modify the Subscription openshift-gitops-operator and add the following environment variables:

spec:
config:
env:
- name: DISABLE_DEFAULT_ARGOCD_INSTANCE (1)
value: 'true'
- name: ARGOCD_CLUSTER_CONFIG_NAMESPACES (2)
value: '*'

1	Optional: Disable the default Argo CD instance.
2	Tell the Operator for which namespaces it should feel responsible for. In this case, all Namespaces. This is important for a namespace-scoped Argo CD instance, which we will install in the next step.

Activate the Principal Component

To activate the Principal components we first need a cluster (the Hub) where OpenShift GitOps is installed. On this cluster, there might be a running instance of Argo CD already. However, at this time an existing Argo CD cannot be used. Instead, a new Argo CD instance must be created. (This is because the controller must not be activated)

In my case, I will install the Principal component in the namespace argocd-principal.

To create an Argo CD instance we need to create the following configuration:

At this very stage, the principal component must be installed in a separate Argo CD instance, since the controller must not be activated. Therefore we create a new Argo CD instance in a new namespace.

apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
name: hub-argocd
namespace: argocd-principal
spec:
controller:
enabled: false (1)
argoCDAgent:
principal:
enabled: true (2)
auth: "mtls:CN=([^,]+)" (3)
logLevel: "info"
namespace:
allowedNamespaces: (4)
- "*"
tls:
insecureGenerate: false (5)
jwt:
insecureGenerate: false
sourceNamespaces: (6)
- "argocd-agent-bm01"
server:
route:
enabled: true (7)

1	Disable the controller component.
2	Enable the Principal component.
3	Authentication method for the Principal component.
4	Allowed namespaces for the Principal component.
5	Insecure generation of the TLS certificate.
6	Specifies the sourceNamespaces configuration. (Such a list might already exist)
7	Enable the Route for the Principal component.

This will start a Pod called hub-gitops-agent-principal in the namespace argocd-principal. However, this pod will fail at this moment and that is fine.

Pod hub-gitops-agent-principal is failing with:
{"level":"info","msg":"Setting loglevel to info","time":"2026-01-19T13:45:57Z"}
time="2026-01-19T13:45:57Z" level=info msg="Loading gRPC TLS certificate from secret argocd-principal/argocd-agent-principal-tls"
time="2026-01-19T13:45:57Z" level=info msg="Loading root CA certificate from secret argocd-principal/argocd-agent-ca"
time="2026-01-19T13:45:57Z" level=info msg="Loading resource proxy TLS certificate from secrets argocd-principal/argocd-agent-resource-proxy-tls and argocd-principal/argocd-agent-ca"
[FATAL]: Error reading TLS config for resource proxy: error getting proxy certificate: could not read TLS secret argocd-principal/argocd-agent-resource-proxy-tls: secrets "argocd-agent-resource-proxy-tls" not found (1)

1	The secret is not yet available.

The Pod is failing at the moment because the different secrets for authentication, are not yet available. The Secrets are created in a later step, because some settings, such as the principal hostname and resource proxy service names, are available only after the Red Hat OpenShift GitOps Operator enables the principal component.

At this point the Operator created the Route object already:

Route: hub-gitops-agent-principal
Hostname: https://hub-gitops-agent-principal-argocd-principal.apps.ocp.aws.ispworld.at
Service: hub-gitops-agent-principal

Configure the AppProject

If you configured the AppProject with sourceNamespaces, you need to add the following to the AppProject (for example to the default AppProject). This must match exactly the namespaces you have created for the Agent.

spec:
sourceNamespaces:
- "argocd-agent-bm01"

You can also use this patch command:

oc patch appproject default -n argocd-principal --type='merge' \
-p '{"spec": {"sourceNamespaces": ["argocd-agent-bm01"]}}' --context aws

Restart the Argo CD Pods to apply the changes.

Download argocd-agentctl

To create the required secrets, we need to download the argocd-agentctl tool.

This can be found at: https://developers.redhat.com/content-gateway/rest/browse/pub/cgw/openshift-gitops/

Download and install it for your platform.

Create Required Secrets

The following steps will create the required secrets for the Principal component. In this example, we will create our own CA and certificates. This is suitable for development and testing purposes. For production environments, you should use certificates issued by your organization’s PKI or a trusted certificate authority.

Use your company’s CA and certificates for production environments.

Initialize the Certificate Authority (CA)

To create a certificate authority (CA) that signs other certificates, we need to run the following command:

argocd-agentctl pki init \
--principal-namespace argocd-principal \ (1)
--principal-context aws

1	The namespace where the Principal component is running.

This will initialize the CA and store it in the secret argocd-principal/argocd-agent-ca. The certificate looks like:

"Certificate Information:
Common Name: argocd-agent-ca
Subject Alternative Names:
Organization: DO NOT USE IN PRODUCTION
Organization Unit:
Locality:
State:
Country:
Valid From: January 15, 2026
Valid To: January 15, 2036
Issuer: argocd-agent-ca, DO NOT USE IN PRODUCTION
Serial Number: 1 (0x1)"

Generate Service Certificate for the Principal

To generate the server certificate for the Principal’s gRPC service, run the following command:

argocd-agentctl pki issue principal \
--principal-namespace argocd-principal \
--principal-context aws \
--dns "<YOUR PRINCIPAL HOSTNAME>" (1)

1	The hostname of the Principal service. This must match with the hostname of the Principal’s route (spec.host) or, in case a LoadBalancer Service is used, with .status.loadBalancer.ingress.hostname.

Generate the Resource Proxy Certificate

The resource proxy service requires a certificate as well. Since the proxy will run on the same cluster as the Principal, we can use the service name directly. This is generated by the following command:

argocd-agentctl pki issue resource-proxy \
--principal-namespace argocd-principal \
--principal-context aws \
--dns hub-argocd-agent-principal-resource-proxy (1)

1	The service name for the resource-proxy. This must match with the service name of the Resource Proxy service.

Generate the JWT Signing Key

Generate the RSA private key for the JWT signing key by running the following command:

argocd-agentctl jwt create-key \
--principal-namespace argocd-principal \
--principal-context aws

This will generate the RSA private key and store it in the secret argocd-principal/argocd-agent-jwt.

Verify the Principal Component

Now the principal pod should be running successfully.

If the Pod still shows an error, wait a few moments or restart the Pod.

In the logs you should see the following:

{"level":"info","msg":"Setting loglevel to info","time":"2026-01-19T14:09:31Z"}
time="2026-01-19T14:09:31Z" level=info msg="Loading gRPC TLS certificate from secret argocd-principal/argocd-agent-principal-tls"
time="2026-01-19T14:09:31Z" level=info msg="Loading root CA certificate from secret argocd-principal/argocd-agent-ca"
time="2026-01-19T14:09:31Z" level=info msg="Loading resource proxy TLS certificate from secrets argocd-principal/argocd-agent-resource-proxy-tls and argocd-principal/argocd-agent-ca"
time="2026-01-19T14:09:31Z" level=info msg="Loading JWT signing key from secret argocd-principal/argocd-agent-jwt"
time="2026-01-19T14:09:31Z" level=info msg="Starting argocd-agent (server) v99.9.9-unreleased (ns=argocd-principal, allowed_namespaces=[*])" module=server

This concludes the configuration of the Principal component. There are a lot of steps to create the required secrets, but this is only done once. A GitOps-friendly way to achieve this might be done using a Kubernetes Job (if you consider this as GitOps-friendly… which I do).

Activate the Agent Component

After the Principal component is configured, you can activate one or more Agents (spoke or workload clusters) and connect them with the Hub.

The prerequisites are:

The Principal component is configured and running.
You have access to both the Principal and Agent clusters.
The argocd-agentctl CLI tool is installed and accessible from your environment.
The helm CLI is installed and configured. Ensure that the helm CLI version is later than v3.8.0.
OpenShift GitOps Operator is installed and configured on the Agent cluster.

Yes, a separate Helm Chart will be used to install the Agent component on the target cluster. This time it is not a Chart that I created, but one provided by Red Hat. :)

Create Agent Secret on Principal Cluster

We first need to create an agent on the Principal cluster.

argocd-agentctl agent create "argocd-agent-bm01" \ (1)
--principal-context "aws" \ (2)
--principal-namespace "argocd-principal" \ (3)
--resource-proxy-server "hub-argocd-agent-principal-resource-proxy:9090" (4)

1	A (unique) name for the Agent.
2	The context name for the Principal cluster. In my case it is "aws".
3	The namespace where the Principal component is running.
4	The resource proxy server URL. This is the URL of the Principal’s resource proxy service including the port (9090).

This will create the secret cluster-argocd-agent-bm01 with the label argocd.argoproj.io/secret-type: cluster in the Argo CD namespace.

Create the Agent Namespace on the Agent Cluster

Be sure that the target namespace on the agent or workload cluster exists. If not, create it first.

oc create namespace argocd-agent-bm01 --context bm (1)

1	The name of the namespace.

Propagate the Principal CA to the Agent Cluster

To copy the CA certificate from the principal to the agent cluster the following command is used:

argocd-agentctl pki propagate \
--agent-context bm \ (1)
--principal-context aws \ (2)
--principal-namespace argocd-principal \ (3)
--agent-namespace argocd-agent-bm01 (4)

1	The context name for the Agent cluster.
2	The context name for the Principal cluster.
3	The namespace where the Principal component is running.
4	The namespace where the Agent component is running.

This will copy the CA certificate from the Principal cluster to the Agent cluster into the namespace and secret argocd-agent-bm01/argocd-agent-ca.

Only the certificate is copied. The private key is not copied.

Generate a client certificate for the Agent

Now we need to create, based on the imported CA, a client certificate for the Agent. This is done by the following command:

argocd-agentctl pki issue agent "argocd-agent-bm01" \
--principal-context "aws" \
--agent-context "bm" \
--agent-namespace "argocd-agent-bm01" \
--principal-namespace "argocd-principal"

This will create the secret argocd-agent-client-tls on the workload cluster, containing a certificate and a key, signed by the CA certificate imported from the Principal cluster.

Configure OpenShift GitOps Subscription on the Spoke Cluster

We need to modify the Subscription openshift-gitops-operator and add the following environment variables:

spec:
config:
env:
- name: DISABLE_DEFAULT_ARGOCD_INSTANCE (1)
value: 'true'
- name: ARGOCD_CLUSTER_CONFIG_NAMESPACES (2)
value: '*'

1	Disable the default Argo CD instance.
2	Tell the Operator for which namespaces it should feel responsible for. In this case, all Namespaces.

Create Argo CD Instance on the Agent Cluster

To create a minimalistic Argo CD instance on the Agent cluster, we can use the following Argo CD configuration:

apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
name: agent-argocd
namespace: argocd-agent-bm01 (1)
spec:
server:
enabled: false

1	The namespace where the Argo CD instance is running. This is also the name of the Agent we have created earlier on the principal cluster.

This will create a lightweight Argo CD instance in the namespace argocd-agent-bm01.

oc get pod -n argocd-agent-bm01 --context bm
NAME READY STATUS RESTARTS AGE
agent-argocd-application-controller-0 1/1 Running 0 2m49s
agent-argocd-redis-5f6759f6fb-2fdnt 1/1 Running 0 2m49s
agent-argocd-repo-server-7949d97dfd-dsk6b 1/1 Running 0 2m49s

Installing the Agent

To install the agent we will use a Helm Chart provided by Red Hat. This will install the Agent component on the target cluster.

As a reminder, we have two modes of operation for the Agent:

Managed mode — the control plane/hub defines Argo CD applications and their specifications.
Autonomous mode — each workload cluster/spoke defines its own Argo CD applications and their specifications.

Create Required Network Policy

Before we start with the actual installation of the Agent, we need to ensure that the Redis instance on the spoke cluster is accessible for the Agent. We need to create a NetworkPolicy accordingly:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: argocd-agent-bm01-redis-network-policy
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: agent-argocd-redis (1)
ingress:
- ports:
- protocol: TCP
port: 6379
from:
- podSelector:
matchLabels:
app.kubernetes.io/name: argocd-agent-agent
policyTypes:
- Ingress

1	The name of the Redis instance. The label is based on <instance name>-redis.

Apply the NetworkPolicy to the spoke cluster:

oc apply -f network-policy.yaml -n argocd-agent-bm01 --context bm

Add the Helm Chart Repository

Add the Helm repository:

helm repo add openshift-helm-charts https://charts.openshift.io/
helm repo update

Install a managed Agent with the Helm Chart

Install the agent in the managed mode using the Helm Chart. The following parameters are used:

namespaceOverride - The namespace where the Agent is running.
agentMode - The mode of the Agent.
server - The server URL of the Principal component. This is the spec.host setting of the Principal’s route.
argoCdRedisSecretName - The name of the Redis secret.
argoCdRedisPasswordKey - The key of the Redis password.
redisAddress - The address of the Redis instance.

helm install redhat-argocd-agent openshift-helm-charts/redhat-argocd-agent \
--set namespaceOverride=argocd-agent-bm01 \
--set agentMode="managed" \
--set server="serverURL of principal route" \
--set argoCdRedisSecretName="agent-argocd-redis-initial-password" \
--set argoCdRedisPasswordKey="admin.password" \
--set redisAddress="agent-argocd-redis:6379" \
--kube-context "bm"

With this chart a pod on the spoke cluster will be created and will start to synchronize the Argo CD applications between the hub and the spoke cluster.

Verify the Managed Agent

To verify the Agent in managed mode we need to create an Argo CD Application on the hub cluster. We can try the following Application. The Application is taken from the openshift-clusterconfig-gitops repository and simply adds a banner to the top of the OpenShift UI. I typically use this as a quick test if GitOps is working.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: branding
namespace: argocd-agent-bm01 (1)
spec:
destination:
namespace: default
server: 'https://hub-argocd-agent-principal-resource-proxy:9090?agentName=argocd-agent-bm01' (2)
project: default
source:
path: clusters/management-cluster/branding
repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops'
targetRevision: main
syncPolicy:
automated:
prune: true
selfHeal: true

1	The namespace of the agent
2	The server URL of the Principal component. This is the URL plus the port and the agentName. As an alternative you can also use name: argocd-agent-bm01 which is the name of the cluster and might be easier to read.

Apply the Application to the hub cluster:

oc apply -f application.yaml -n argocd-agent-bm01 --context aws

Since the application will try to automatically synchronize the configuration, the status will change to Synced after a few seconds:

status:
resources:
- group: console.openshift.io
kind: ConsoleNotification
name: topbanner
status: Synced
version: v1

and the (top) banner will be visible on the UI:

Figure 1. Banner on the top of the UI

On the hub cluster, the Argo CD Application will be Synced:

oc get applications --context aws -A
NAMESPACE NAME SYNC STATUS HEALTH STATUS
argocd-agent-bm01 branding-banner Synced Healthy

Install an autonomous Agent with the Helm Chart

Let’s cleanup the first installation of the Chart (managed agent) in order to not have any conflicts.

helm uninstall redhat-argocd-agent --kube-context "bm"

Install the agent in the autonomous mode using the Helm Chart. The following parameters are used:

namespaceOverride - The namespace where the Agent is running.
agentMode - The mode of the Agent.
server - The server URL of the Principal component. This is the spec.host setting of the Principal’s route.
argoCdRedisSecretName - The name of the Redis secret.
argoCdRedisPasswordKey - The key of the Redis password.
redisAddress - The address of the Redis instance.

The only difference to the managed mode is the agentMode parameter.

helm install redhat-argocd-agent-autonomous openshift-helm-charts/redhat-argocd-agent \
--set namespaceOverride=argocd-agent-bm01 \
--set agentMode="autonomous" \
--set server="serverURL of principal route" \
--set argoCdRedisSecretName="agent-argocd-redis-initial-password" \
--set argoCdRedisPasswordKey="admin.password" \
--set redisAddress="agent-argocd-redis:6379" \
--kube-context "bm"

With this chart a pod on the spoke cluster will be created and will start to synchronize the Argo CD applications between the hub and the spoke cluster.

Verify the Autonomous Agent

To verify the Agent in autonomous mode we need to create an Argo CD Application on the spoke cluster.

We can try the following Application. It is basically the same as we used for the test for the managed mode, except that this time we will add a banner on the bottom of the UI.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: branding-bottom-banner
namespace: argocd-agent-bm01 (1)
spec:
destination:
namespace: default
server: 'https://kubernetes.default.svc' (2)
project: default
source:
path: clusters/management-cluster/branding-bottom
repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops'
targetRevision: main
syncPolicy:
automated:
prune: true
selfHeal: true

1	The namespace of the agent
2	The server URL of the local cluster, since in autonomous mode the application is managed locally.

Apply the Application to the spoke cluster:

oc apply -f application.yaml -n argocd-agent-bm01 --context bm

Like the managed mode, the status will change to Synced after a few seconds and the (this time) bottom banner will be visible on the UI:

Figure 2. Banner on the bottom of the UI

Moreover, the Argo CD Application will appear on the hub cluster:

oc get applications --context aws -A
NAMESPACE NAME SYNC STATUS HEALTH STATUS
argocd-agent-bm01 branding-bottom-banner Synced Healthy
argocd-agent-bm01 branding-banner Synced Healthy

Troubleshooting

During the installation and configuration of the Argo CD Agent, you might encounter some issues. Here are some issues I encountered during the tests:

Principal Pod Fails to Start

If the Principal pod fails to start with errors about missing secrets, verify that all required secrets have been created:

oc get secrets -n argocd-principal | grep argocd-agent

You should see the following secrets:

argocd-agent-ca - The CA certificate
argocd-agent-principal-tls - The Principal’s TLS certificate
argocd-agent-resource-proxy-tls - The Resource Proxy’s TLS certificate
argocd-agent-jwt - The JWT signing key

If any of these are missing, re-run the corresponding argocd-agentctl command to create them.

Redis Errors in the Principal Pod

When you see errors like the following in the logs of the Principal pod, ensure that the Argo CD instance does not have the controller enabled in that namespace (set spec.controller.enabled: false). Hopefully, this will change in the future.

time="2026-01-20T04:57:10Z" level=error msg="unexpected lack of '_' namespace/name separate: 'app|managed-resources|branding|1.8.3'" connUUID=3c13b30b-9f84-4af6-93d8-e1c03c4c7898 function=redisFxn module=redisProxy

Limitations

While the Argo CD Agent brings significant improvements, there are some limitations to be aware of:

Separate Argo CD Instance Required

Currently, the Principal component cannot be installed alongside an existing Argo CD instance where the application controller is enabled. You must create a separate Argo CD instance with the controller disabled (spec.controller.enabled: false). To me, this is one of the biggest limitations. However, this will be addressed in the future and is tracked in the issue: https://github.com/argoproj-labs/argocd-agent/issues/708

Manual Certificate Management

The mTLS certificates must be created and managed manually by the user. There is no automatic certificate rotation or renewal. For production environments, you should integrate with your organization’s PKI infrastructure and implement a certificate rotation strategy.

Summary

The Argo CD Agent provides a powerful solution for managing multiple clusters with Argo CD. By combining the benefits of centralized management with distributed application controllers, it addresses the scalability and single point of failure challenges of traditional deployment models.

While the initial setup requires several steps, especially around certificate management, the resulting architecture offers a robust foundation for GitOps at scale.

The Hitchhiker's Guide to Observability - Here Comes Grafana - Part 7

Thu, 04 Dec 2025 00:00:00 +0000

While we have been using the integrated tracing UI in OpenShift, it is time to summon Grafana. Grafana is a visualization powerhouse that allows teams to build custom dashboards, correlate traces with logs and metrics, and gain deep insights into their applications. In this article, we’ll deploy a dedicated Grafana instance for team-a in their namespace, configure a Tempo datasource, and create a dashboard to explore distributed traces.

Prerequisites

Before we begin, make sure you have:

The Grafana Operator installed cluster-wide (we’ll cover this first)
TempoStack deployed and configured (from Part 2)
Team-a namespace with traces flowing (from Part 4)

The Grafana Operator

The Grafana Operator provides Custom Resource Definitions (CRDs) for managing Grafana instances, datasources, and dashboards declaratively.

Be sure that the Operator is installed. Typically, it is installed in the openshift-operators namespace. If you keep that namespace, all you need to do is create a Subscription. Otherwise you will also need to create an OperatorGroup.

If you select a different namespace, be sure to verify possible RBAC bindings, that you might need to set up.

Install Subscription

The Grafana Operator is available from the OperatorHub. Either use the UI to install it, or simply create a Subscription to install it:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: grafana-operator
namespace: openshift-operators (1)
spec:
channel: v5 (2)
installPlanApproval: Automatic
name: grafana-operator
source: community-operators (3)
sourceNamespace: openshift-marketplace

1	Installing in `openshift-operators` makes the operator available cluster-wide
2	Use the v5 channel for the operator. This is the only available channel currently.
3	The source is the OperatorHub catalog. Grafana is a community operator.

Verify the operator is running:

oc get pods -n openshift-operators -l app.kubernetes.io/name=grafana-operator

You should see output similar to:

NAME READY STATUS RESTARTS AGE
grafana-operator-7d8f9c6b5-xyz12 1/1 Running 0 2m

Create Grafana Instance for Team-A

Now let’s deploy a Grafana instance in the team-a namespace. This gives the team full control over their dashboards while isolating them from other teams.

We are logging in as the project admin user of the team-a namespace. So everything we do in this namespace will be done as this user.

Create the Grafana Admin Secret

First, we need to create a secret containing the Grafana admin credentials, since we do not want to store passwords in plain text in our manifests!

Never store passwords in plain text in your manifests and use a secrets management solution like Sealed Secrets or External Secrets Operator.

Create the following example. Replace the password with your own strong password.

apiVersion: v1
kind: Secret
metadata:
name: grafana-admin-credentials
namespace: team-a
type: Opaque
stringData:
GF_SECURITY_ADMIN_USER: admin (1)
GF_SECURITY_ADMIN_PASSWORD: <your-secure-password> (2)

1	The admin username for Grafana
2	Replace with a strong password

Deploy the Grafana Instance

The Grafana Operator provides a Custom Resource Definition (CRD) for deploying Grafana instances. We can use this to deploy a Grafana instance in the team-a namespace. This instance is labelled with "dashboards: "grafana-team-a"" to make it easier to target it with GrafanaDashboard resources.

apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
name: grafana
namespace: team-a
labels:
dashboards: "grafana-team-a" (1)
spec:
config:
log:
mode: "console"
auth:
disable_login_form: "false"
security:
admin_user: ${GF_SECURITY_ADMIN_USER} (2)
admin_password: ${GF_SECURITY_ADMIN_PASSWORD}
deployment:
spec:
replicas: 1
template:
spec:
containers:
- name: grafana
envFrom:
- secretRef:
name: grafana-admin-credentials (3)
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
route:
spec:
tls:
termination: edge (4)
disableDefaultAdminSecret: true (5)

1	Label used by GrafanaDashboard resources to target this instance. Required so that the datasource can find the instance.
2	References environment variables from the secret
3	Mounts the secret as environment variables
4	Creates an OpenShift Route with TLS edge termination
5	Disables the default admin secret created by the Grafana Operator. We will use our own secret and this setting prevents that the operator overwrites it.

After a few moments, the Grafana pod should be ready.

oc get pods -n team-a -l app=grafana -w

Get the Route URL:

oc get route grafana-route -n team-a -o jsonpath='{.spec.host}'

Use this route and the credentials from the secret to log into Grafana.

Configure Tempo Datasource

The datasource connects Grafana to TempoStack (or any other datasource like Loki, Prometheus, etc.), allowing you to query and visualize traces. We need to authenticate with a client certificate to TempoStack and be sure the send the tenant ID of tenantA in the header of the queries.

When you read the previous part of this series, you saw that we created a ClusterRole to grant read access to the traces. The Binding for this role allows read access to everybody who is authenticated against the system. Therefore, we do not need to take care of permissions… at this point.

TempoStack Client Certificate Authentication

To query the TempoStack instance, we need to authenticate with it. This is done by using a client certificate. Therefore, we need to create a secret with the client certificate and key. In addition, this secret will also contain the tenant ID, which must be sent to the TempoStack instance as a header.

First, get the client certificate and key from the TempoStack instance:

oc get secret -n tempostack | grep gateway-mtls

You should find something like this:

tempo-simplest-gateway-mtls kubernetes.io/tls 2 19d

Fetch the client certificate and key and store them in a file locally:

oc get secret -n tempostack tempo-simplest-gateway-mtls -o jsonpath='{.data.tls\.crt}' | base64 -d > tls.crt
oc get secret -n tempostack tempo-simplest-gateway-mtls -o jsonpath='{.data.tls\.key}' | base64 -d > tls.key

Now, let’s get the tenant ID from the TempoStack instance:

oc get tempostack/simplest -n tempostack -o jsonpath='{.spec.tenants.authentication}'

You should find something like this (tenantA is the one we are interested in):

[
{
"tenantId": "1610b0c3-c509-4592-a256-a1871353dbfc", (1)
"tenantName": "tenantA"
},
{
"tenantId": "1610b0c3-c509-4592-a256-a1871353dbfd",
"tenantName": "tenantB"
}
]

1	The tenant ID of tenantA

Now, let’s create the secret with the client certificate and key and the tenant ID:

oc create secret generic tempo-auth -n team-a --from-file=tls.crt=tls.crt --from-file=tls.key=tls.key --from-literal=tenantA="1610b0c3-c509-4592-a256-a1871353dbfc"

That’s everything we need to authenticate with TempoStack for tenantA.

As you prefer, you can create a separate secret for each tenantID. Just be sure to update the "valuesFrom" section of the GrafanaDatasource resource.

Create the Tempo Datasource

Now we can create the GrafanaDatasource that connects to TempoStack:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
name: tempo-tenanta-datasource
namespace: team-a
spec:
allowCrossNamespaceImport: false
datasource:
access: proxy
editable: true
isDefault: true
jsonData:
httpHeaderName1: X-Scope-OrgID (1)
timeInterval: 5s
tlsAuth: true
tlsAuthWithCACert: false
tlsSkipVerify: true
name: tempo-tenanta (2)
secureJsonData: (3)
httpHeaderValue1: '${tenantA}'
tlsClientCert: '${tls.crt}'
tlsClientKey: '${tls.key}'
type: tempo (4)
url: 'https://tempo-simplest-query-frontend.tempostack.svc.cluster.local:3200' (5)
instanceSelector:
matchLabels:
dashboards: grafana-team-a (6)
resyncPeriod: 10m0s
valuesFrom: (7)
- targetPath: secureJsonData.tlsClientCert
valueFrom:
secretKeyRef:
key: tls.crt
name: tempo-auth
- targetPath: secureJsonData.tlsClientKey
valueFrom:
secretKeyRef:
key: tls.key
name: tempo-auth
- targetPath: secureJsonData.httpHeaderValue1
valueFrom:
secretKeyRef:
key: tenantA
name: tempo-auth

1	The header name for the tenant ID.
2	The name of the Tempo datasource.
3	The reference to the client certificate, key and tenantID. They are coming from the secret we created earlier.
4	The type of the datasource. This type must be available in Grafana. For Tempo, the type is `tempo`.
5	The URL of the Tempo query frontend. This is the URL of the Tempo query frontend service in the TempoStack namespace.
6	The label of the Grafana instance to target. This is the label we added to the Grafana instance when we created it.
7	The reference to the secrets and the keys inside the secret.

Verify the Datasource

Log into Grafana and navigate to Connection > Data sources. You should see the Tempo datasource listed.

Click on the Tempo datasource to see the details:

As depicted in the image above, the datasource is configured with TLS Client Authentication and a HTTP Header.

Any changes in the Grafana UI will not be synced back to the GrafanaDatasource resource. You need to update the resource manually.

To verify the datasource, we can create a test query. Navigate to Explore.

Enter the query {}, which will simply get all traces from the TempoStack instance from (by default) the last minute.

If you have multiple data sources configured already, be sure to select the correct one in the dropdown menu.

You should see the traces in the Grafana Explore UI.

You can select any trace and see the details in the Grafana Explore UI.

Create a Traces Dashboard

Now let’s create a dashboard to visualize and explore traces. The Grafana Operator allows us to define dashboards as Kubernetes resources.

Create the Dashboard ConfigMap

To have the first showable data, we need to create a dashboard. The Grafana Operator provides a Custom Resource Definition called GrafanaDashboard for deploying such dashboards.

Now, dashboards are their own beast. There are so many things to configure and set that probably a whole separate blog series is needed to cover all of it. I am by far not an expert in this field and I am happy that I got this one up and running, so I will just create a simple dashboard with a few panels.

Honestly, I am not sure if it is a good way to create dashboards using the CRD. It is probably better to use the Grafana UI and then export the JSON data if you want to keep it declarative.

This dashboard provides:

A trace search panel
Latency histogram -→ I have set this to "higher than 3ms" to at least see something.
Recent traces table

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
name: grafana-team-a-traces-dashboard
namespace: team-a
spec:
instanceSelector:
matchLabels:
dashboards: "grafana-team-a" (1)
json: | (2)
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "grafana",
"uid": "-- Grafana --"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"id": 1,
"links": [],
"panels": [
{
"fieldConfig": {
"defaults": {},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 24,
"x": 0,
"y": 0
},
"id": 1,
"options": {
"code": {
"language": "plaintext",
"showLineNumbers": false,
"showMiniMap": false
},
"content": "# Team-A Distributed Traces\n\nThis dashboard allows you to explore distributed traces from your applications.\n\n",
"mode": "markdown"
},
"pluginVersion": "12.1.0",
"title": "Welcome",
"type": "text"
},
{
"datasource": {
"type": "tempo",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"custom": {
"align": "auto",
"cellOptions": {
"type": "auto"
},
"inspect": false
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": 0
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 4
},
"id": 2,
"options": {
"cellHeight": "sm",
"footer": {
"countRows": false,
"fields": "",
"reducer": [
"sum"
],
"show": false
},
"showHeader": true
},
"pluginVersion": "12.1.0",
"targets": [
{
"datasource": {
"type": "tempo",
"uid": "${datasource}"
},
"filters": [
{
"id": "81ea3f00",
"operator": "=",
"scope": "span"
}
],
"limit": 20,
"metricsQueryType": "range",
"queryType": "traceqlSearch",
"refId": "A",
"tableType": "traces"
}
],
"title": "Trace Search",
"type": "table"
},
{
"datasource": {
"type": "tempo",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "bars",
"fillOpacity": 100,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": 0
},
{
"color": "red",
"value": 80
}
]
},
"unit": "ms"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 4
},
"id": 3,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "12.1.0",
"targets": [
{
"datasource": {
"type": "tempo",
"uid": "${datasource}"
},
"filters": [
{
"id": "6ff20d0d",
"operator": "=",
"scope": "span"
},
{
"id": "min-duration",
"operator": ">",
"tag": "duration",
"value": "0.3ms",
"valueType": "duration"
}
],
"limit": 20,
"metricsQueryType": "range",
"queryType": "traceqlSearch",
"refId": "A",
"tableType": "spans"
}
],
"title": "Span Duration Distribution",
"type": "timeseries"
},
{
"datasource": {
"type": "tempo",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"custom": {
"align": "auto",
"cellOptions": {
"type": "auto"
},
"inspect": false
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": 0
}
]
}
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "traceID"
},
"properties": [
{
"id": "links",
"value": [
{
"title": "View Trace",
"url": "/explore?orgId=1&left=%7B%22datasource%22:%22${datasource}%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22datasource%22:%7B%22type%22:%22tempo%22,%22uid%22:%22${datasource}%22%7D,%22queryType%22:%22traceql%22,%22limit%22:20,%22query%22:%22${__value.raw}%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D"
}
]
}
]
},
{
"matcher": {
"id": "byName",
"options": "duration"
},
"properties": [
{
"id": "unit",
"value": "ns"
}
]
}
]
},
"gridPos": {
"h": 10,
"w": 24,
"x": 0,
"y": 12
},
"id": 4,
"options": {
"cellHeight": "sm",
"footer": {
"countRows": false,
"fields": "",
"reducer": [
"sum"
],
"show": false
},
"showHeader": true,
"sortBy": [
{
"desc": true,
"displayName": "startTime"
}
]
},
"pluginVersion": "12.1.0",
"targets": [
{
"datasource": {
"type": "tempo",
"uid": "${datasource}"
},
"limit": 50,
"queryType": "traceqlSearch",
"refId": "A",
"tableType": "traces"
}
],
"title": "Recent Traces",
"type": "table"
}
],
"preload": false,
"refresh": "30s",
"schemaVersion": 41,
"tags": [
"tracing",
"tempo",
"team-a"
],
"templating": {
"list": [
{
"current": {
"text": "tempo-tenanta",
"value": "90b71ee3-693a-4c41-8cdf-624a3bb78e7a"
},
"includeAll": false,
"label": "Datasource",
"name": "datasource",
"options": [],
"query": "tempo",
"refresh": 1,
"regex": "",
"type": "datasource"
}
]
},
"time": {
"from": "now-1h",
"to": "now"
},
"timepicker": {},
"timezone": "",
"title": "Team-A Distributed Traces",
"uid": "team-a-traces",
"version": 3
}

1	The label of the Grafana instance to target. This is the label we added to the Grafana instance when we created it.
2	The JSON data of the dashboard (yes, it’s a monster!).

Now you will see a new dashboard in the Grafana UI. Navigate to Dashboards and find Team-A Distributed Traces.

Next Steps

Congratulations! The Cat has summoned Grafana .

You now have a functional Grafana instance with Tempo integration for team-a. Here are some ideas for extending this setup:

Add Loki datasource: Correlate traces with logs using derived fields
Add Prometheus datasource: Link metrics to traces for full observability
Create alerting rules: Set up alerts for high latency or error rates
Build more dashboards: Create service-specific dashboards for different applications

As mentioned above, Grafana and its dashboards are powerful tools for visualizing and exploring traces. There are tons of things to configure and set up. Maybe I will cover some of these things in a future article.

Happy tracing! 🚀

The Guide to OpenBao - Introduction - Part 1

Wed, 11 Feb 2026 00:00:00 +0000

I finally had some time to dig into Secret Management. For my demo environments, SealedSecrets is usually enough to quickly test something. But if you want to deploy a real application with Secret Management, you need to think of a more permanent solution.

This article is the first of a series of articles about OpenBao, a HashiCorp Vault fork. Today, we will explore what OpenBao is, why it was created, and when you should consider using it for your secret management needs. If you are familiar with HashiCorp Vault, you will find many similarities, but also some important differences that we will discuss.

Introduction

In general, sensitive information in any system should be stored in a secure way. When it comes to OpenShift or Kubernetes, this is especially true, since the secrets are stored in the etcd database. Even if etcd is encrypted at rest, anybody can decode a given base64 string which is stored in the Secret.

Base64 is not an encryption format. It is an encoding format.

For example, the string Thomas encoded as base64 is VGhvbWFzCg==. This is simply masked plain text and it is not secure to share these values, especially not on Git. To make your CI/CD pipelines or GitOps process secure, you need to think of a secure way to manage your Secrets.

This is where OpenBao comes in.

What is OpenBao?

OpenBao is an identity-based secrets and encryption management system. It is an open-source, community driven fork of HashiCorp Vault and provides secure storage, fine-grained access control, and lifecycle management for secrets such as API or SSH keys, passwords, certificates, and encryption keys.

The core OpenBao workflow consists of four stages:

Authentication: Verifying the identity of the client to determine if they are who they say they are. Once authenticated, a token is created and associated with a so-called policy.
Validation: Checking if the client has the required permissions
Authorization: Granting access based on policies that provide or deny access to certain paths and operations in OpenBao.
Access: Limiting what the client can do with the secret

At its core, OpenBao:

Stores and encrypts sensitive information at rest and in transit
Provides fine-grained access controls (ACL)
Supports dynamic secret generation
Offers a comprehensive audit trail (must be enabled)
Enables encryption as a service

The History: Why OpenBao Exists

OpenBao is a community-driven, open-source fork of HashiCorp Vault. But why was it created?

In August 2023, HashiCorp announced a significant change to the licensing of their products, including Vault. They moved from the Mozilla Public License 2.0 (MPL 2.0) to the Business Source License 1.1 (BSL 1.1).

The BSL 1.1 license includes restrictions on competitive use:

Competitors could no longer use Vault’s code to offer competing services
Cloud providers and managed service providers faced restrictions
The open-source community lost the freedom to fork and commercialize

The Fork

In response to this license change, the Linux Foundation announced the OpenBao project in December 2023. OpenBao is:

A fork of HashiCorp Vault (from version 1.14.x)
Licensed under the OSI-approved Mozilla Public License 2.0
Governed by a community-driven model
Maintained independently of HashiCorp
Continues to enable innovation without license restrictions

OpenBao vs. HashiCorp Vault: Key Differences

While OpenBao shares its heritage with Vault, there are several important differences:

Licensing

Aspect

OpenBao

HashiCorp Vault

License

MPL 2.0 (OSI-approved open source)

BSL 1.1 (with staged conversion)

Governance

Community-driven (Linux Foundation)

HashiCorp/IBM controlled

Commercial restrictions

None

Restrictions on competitive use

Enterprise Features

Community-driven additions

Paid Enterprise tier

Support

Community support

Paid support available

Branding

bao CLI, OpenBao naming

vault CLI, Vault naming

Technical Differences

Token Format - OpenBao uses shorter tokens in the format sbr.[random], while Vault uses longer tokens (hvs., hvb., hvr. prefixes followed by long random strings). Old Vault tokens are still accepted according to their TTLs, but newly issued tokens follow the OpenBao format.
Plugin Ecosystem - OpenBao comes with fewer built-in plugins by default, focusing on OSI-licensed integrations. Proprietary cloud vendor plugins have been moved to external repositories.
Storage Backend - OpenBao has simplified its storage options, primarily supporting Raft as the recommended backend.
API Compatibility - OpenBao’s API is designed to be compatible with Vault, meaning existing clients and integrations should work without modification. However, some edge cases may require updates.

Core Concepts

Before diving into installation and configuration in the next articles, it is essential to understand the fundamental concepts of OpenBao:

Secrets Engines

Secrets engines are components that store, generate, or encrypt data. OpenBao supports multiple types:

KV (Key-Value): Simple static secret storage with optional versioning
PKI: Certificate Authority for generating TLS certificates
Database: Dynamic credential generation for databases
Transit: Encryption as a Service (EaaS)
SSH: SSH key signing and OTP generation

Authentication Methods

Authentication methods verify client identity before granting access:

Kubernetes: Uses Kubernetes service account tokens. Essential for Kubernetes/OpenShift deployments.
OIDC: OpenID Connect for user authentication, like Keycloak, Okta, or Azure AD.
LDAP: Directory service authentication, like Active Directory, OpenLDAP, or Microsoft AD.
AppRole: Machine-oriented authentication for applications. Ideal for CI/CD pipelines.
Token: Direct token-based authentication. The root token is created during initialization.

Storage Backend

OpenBao encrypts all data before writing to storage. Supported backends include:

Integrated Raft (recommended): Built-in distributed storage
Consul: HashiCorp’s service discovery and KV store (less common with OpenBao)
File: Single-node deployments only
PostgreSQL/MySQL: Database-backed storage

Policies

Policies define what a client can do after authentication. They use path-based access control (deny-by-default mode):

path "secret/data/myapp/*" {
capabilities = ["read", "list"]
}
path "pki/issue/my-role" {
capabilities = ["create", "update"]
}
path "database/creds/myapp-role" {
capabilities = ["read"]
}

Tokens

Tokens are the primary authentication credential in OpenBao. After successful authentication, clients receive a token that:

Has a TTL (Time To Live)
Is associated with one or more policies
Can be renewed (if renewable)
Can create child tokens

Seal/Unseal

OpenBao starts in a sealed state where it cannot access encrypted data. The unseal process requires multiple key shares (using Shamir’s Secret Sharing) or auto-unseal mechanisms to decrypt the master key.

Leases

Most secrets in OpenBao have an associated lease - a duration after which the secret expires. This enables:

Automatic secret rotation
Revocation of compromised credentials
Audit trail of secret usage

Use Cases

Dynamic Database Credentials

Instead of storing static database passwords:

Application authenticates to OpenBao
Requests database credentials
OpenBao creates a temporary database user
Returns credentials with a short TTL
Credentials automatically expire

Benefits: No long-lived credentials, automatic rotation, per-application isolation.

Certificate Management with PKI

Instead of manually managing TLS certificates:

Configure OpenBao as an intermediate CA
Applications request certificates on demand
Short-lived certificates (hours/days instead of years)
Automatic renewal before expiration

Benefits: No certificate sprawl, automated rotation, reduced attack surface.

Encryption as a Service

Instead of implementing encryption in each application:

Applications send plaintext to OpenBao’s Transit engine
OpenBao encrypts with managed keys
Applications store ciphertext
Decryption requests go through OpenBao

Benefits: Centralized key management, separation of duties, key rotation without re-encryption.

Kubernetes Secret Injection

Instead of storing secrets in Kubernetes Secrets:

Deploy OpenBao with Kubernetes auth
Configure injector or External Secrets Operator
Pods automatically receive secrets at startup
Secrets never stored in etcd

Benefits: Secrets not in cluster, dynamic injection, centralized management.

When to Use OpenBao

OpenBao is an excellent choice when you need:

Centralized Secret Management - If you have secrets scattered across configuration files, environment variables, and various secret stores, OpenBao provides a single source of truth.
Dynamic Secrets - For use cases where you need short-lived, automatically rotated credentials (e.g., database passwords), OpenBao can generate them on-demand.
Encryption as a Service - If applications need encryption capabilities without managing encryption keys, OpenBao’s Transit engine provides this functionality.
Certificate Management - OpenBao’s PKI engine can act as a Certificate Authority, issuing and managing TLS certificates.
Compliance Requirements - For environments with strict audit requirements, OpenBao provides comprehensive audit logging of all secret access.

When NOT to Use OpenBao

OpenBao might be overkill for:

Simple, Static Secrets - If you only have a few static secrets that rarely change, simpler solutions like Sealed Secrets or External Secrets Operator with a basic backend might suffice.
Small Teams with Limited Resources - OpenBao requires operational expertise to maintain. If you do not have the resources to operate it properly, consider managed alternatives.
Single-Application Deployments - If you have a single application with minimal secret requirements, the complexity of OpenBao may not be justified.

Architecture Overview

A typical OpenBao deployment consists of:

Figure 1. Architecture Overview

Multiple Nodes: For high availability (HA), OpenBao runs as a cluster
Raft Consensus: Leader election and data replication
Persistent Storage: Encrypted data stored on persistent volumes
Load Balancer: Distributes client requests

Conclusion

OpenBao provides a powerful, open-source solution for secret management that addresses the challenges of modern cloud-native environments. Its fork from HashiCorp Vault means it benefits from years of development while remaining truly open source.

In the next article, we will start with a standalone installation to understand the fundamentals before moving to Kubernetes deployments.

Resources

The Hitchhiker's Guide to Observability Introduction - Part 1

Sun, 23 Nov 2025 00:00:00 +0000

With this article I would like to summarize and, especially, remember my setup. This is Part 1 of a series of articles that I split up so it is easier to read and understand and not too long. Initially, there will be 6 parts, but I will add more as needed.

Introduction

In modern microservices architectures, understanding how requests flow through your distributed system is crucial for debugging, performance optimization, and maintaining system health. Distributed tracing provides visibility into these complex interactions by tracking requests as they traverse multiple services.

This whole guide demonstrates how to set up a distributed tracing infrastructure using

OpenShift (4.16+) as base platform
Red Hat Build of OpenTelemetry - The observability framework based on OpenTelemetry
TempoStack - Grafana’s distributed tracing backend for Kubernetes
Multi-tenant architecture - Isolating traces by team or environment
Cluster Observability Operator - For now this Operator is only used to extend the OpenShift UI with the tracing UI.

Thanks to

This article would not have been possible without the help of Michaela Lang. Check out her articles on LinkedIn mainly discussing Tracing and Service Mesh.

What is OpenTelemetry?

OpenTelemetry is an observability framework and toolkit which aims to provide unified, standardized, and vendor-neutral telemetry data collection for traces, metrics and logs for cloud-native software.

In this article we will focus on traces only.

When it comes to Red Hat and OpenShift, the supported installation is based on the Operator Red Hat Build of OpenTelemetry, which is based on the open source OpenTelemetry project and adds supportability.

Core Features:

Source: OpenShift OTEL Product Overview

The OpenTelemetry Collector can receive, process, and forward telemetry data in multiple formats, making it the ideal component for telemetry processing and interoperability between telemetry systems. The Collector provides a unified solution for collecting and processing metrics, traces, and logs.

The core features of the OpenTelemetry Collector include:

Data Collection and Processing Hub It acts as a central component that gathers telemetry data like metrics and traces from various sources. This data can be created from instrumented applications and infrastructure.
Customizable telemetry data pipeline The OpenTelemetry Collector is customizable and supports various processors, exporters, and receivers.
Auto-instrumentation features Automatic instrumentation simplifies the process of adding observability to applications. If used, developers do not need to manually instrument their code for basic telemetry data. (This depends a bit on the used coding language and framework, maybe this is worth a separate article)

Here are some of the use cases for the OpenTelemetry Collector:

Centralized data collection In a microservices architecture, the Collector can be deployed to aggregate data from multiple services.
Data enrichment and processing Before forwarding data to analysis tools, the Collector can enrich, filter, and process this data.
Multi-backend receiving and exporting The Collector can receive and send data to multiple monitoring and analysis platforms simultaneously. You can use Red Hat build of OpenTelemetry in combination with Red Hat OpenShift Distributed Tracing Platform.

What is Grafana Tempo?

Grafana Tempo is an open-source, easy-to-use, and high-scale distributed tracing backend. Tempo lets you search for traces, generate metrics from spans, and link your tracing data with logs and metrics. It is deeply integrated with Grafana, Prometheus and Loki and can ingest traces from various sources, such as OpenTelemetry, Jaeger, Zipkin and more.

Core Features:

Source: Grafana Tempo

Built for massive scale The only dependency is object storage which provides affordable long-term storage of traces.
Cost-effective Not indexing the traces makes it possible to store orders of magnitude more trace data for the same cost.
Strong integration with open source tools Compatible with open source tracing protocols.

In addition, it is deeply integrated with Grafana, allowing you to visualize the traces in a Grafana dashboard and link logs, metrics and traces together.

Use Case for this Article

The use case that was tested in this article was the following:

Several applications (team-a, team-b, …) are hosted on separate OpenShift namespaces. On each application namespace, a local OpenTelemetry Collector (OTC) is configured to collect the traces from the application. These local OpenTelemetry Collectors will export the traces to a central OpenTelemetry Collector (hosted in the namespace tempostack). The central Collector will then export the data to a TempoStack instance (also hosted in the namespace tempostack), which will store the traces in object storage. The storage itself is provided by S3-compatible storage, in this example OpenShift Data Foundation.

For a more detailed view see the next section.

Architecture Overview

As described the implementation follows a two-tier collector architecture with multi-tenancy support:

---
title: "Architecture Overview"
config:
theme: 'dark'
---
graph TB
subgraph app["Application"]
mockbin1["Mockbin #1<br/>(team-a namespace)<br/>(tenantA)"]
mockbin2["Mockbin #2<br/>(team-b namespace)<br/>(tenantB)"]
end
subgraph local["Local OTC"]
otc_a["OTC-team-a<br/>• Add namespace<br/>• Batch processing<br/>• Forward to central"]
otc_b["OTC-team-b<br/>• Add namespace<br/>• Batch processing<br/>• Forward to central"]
end
subgraph central["Central OTC (tempostack namespace)"]
otc_central["OTC-central<br/>• Receive from local collectors<br/>• Add K8s metadata (k8sattributes)<br/>• Route by namespace (routing connector)<br/>• Authenticate with bearer token<br/>• Forward to TempoStack with tenant ID"]
end
subgraph tempo["TempoStack (tempostack namespace)"]
tempostack["Multi-tenant Trace Storage<br/>• tenantA, tenantB, ...<br/>• S3 backend storage<br/>• 48-hour retention"]
end
mockbin1 -->|"OTLP"| otc_a
mockbin2 -->|"OTLP"| otc_b
otc_a -->|"OTLP(with namespace)"| otc_central
otc_b -->|"OTLP(with namespace)"| otc_central
otc_central -->|"OTLP<br/>(X-Scope-OrgID header)"| tempostack
classDef appStyle fill:#2f652a,stroke:#2f652a,stroke-width:2px
classDef localStyle fill:#425cc6,stroke:#425cc6,stroke-width:2px;
classDef centralStyle fill:#425cc6,stroke:#425cc6,stroke-width:2px;
classDef tempoStyle fill:#906403,stroke:#906403,stroke-width:2px
class mockbin1,mockbin2 appStyle
class otc_a,otc_b localStyle
class otc_central centralStyle
class tempostack tempoStyle

As quick summary Traces from the application Mockbin #1 are collected by the "OTC-team-a" and forwarded to the "Central OTC". From there the traces are further forwarded to Tempo.

Why 2-Tier Architecture?

You may ask yourself why there are two OpenTelemetry Collectors and if the application could not send directly to the Central OTC or if the Local OTC could not write directly into the Tempo storage. Both options are fine and will work, however, I tried to make it more secure. Only one OTC is allowed to perform write actions and applications can only send to the Local OTC, which forwards to the Central OTC where the traces are routed based on the source namespace. This way, nobody can interfere with other namespaces.

Therefore:

Separation of Concerns: Application namespaces handle local processing; central namespace handles routing and storage. The Central decides where and how to store. Application owners cannot overwrite this.
Resource Efficiency: Lightweight collectors in app namespaces, heavy processing centralized
Security: Applications don’t need direct access to TempoStack
Scalability: Each tier can scale independently
Multi-tenancy: Central collector routes traces to appropriate tenants

What now?

The next articles will cover the actual implementation. We will first deploy Tempo and the Central Collector. Then we will deploy example applications and the Local Collector. If everything works as planned, we will be able to see traces on the OpenShift UI.

The Hitchhiker's Guide to Observability - Grafana Tempo - Part 2

Mon, 24 Nov 2025 00:00:00 +0000

After covering the fundamentals and architecture in Part 1, it’s time to get our hands dirty! This article walks through the complete implementation of a distributed tracing infrastructure on OpenShift.

We’ll deploy and configure the Tempo Operator and a multi-tenant TempoStack instance. For S3 storage we will use the integrated OpenShift Data Foundation. However, you can use whatever S3-compatible storage you have available.

Grafana Tempo - Step-by-Step Implementation

Prerequisites

Before starting, ensure you have the following Systems or Operators installed (used Operator versions in this article):

OpenShift or Kubernetes cluster (OpenShift v4.20)
Red Hat build of OpenTelemetry installed (v0.135.0-1)
Tempo Operator installed (v0.18.0-2)
S3-compatible storage (for TempoStack, based on OpenShift Data Foundation)
Cluster Observability Operator (v1.3.0) - for now this Operator is only used to extend the OpenShift UI with the tracing UI.

For all configurations I also created a proper GitOps implementation (of course :)). However, first I would like to show the actual configuration. The GitOps implementation can be found at the section GitOps Deployment.

Step 1: Verify/Deploy Tempo Operator

Let’s first verify if the Tempo Operator is installed and ready to use. If everything is fine, then the Operator will be deployed in the namespace openshift-tempo-operator:

Step 2: Deploy TempoStack Resource

TempoStack is the central trace storage backend. We are using the Tempo Operator here, which provides the TempoStack resource and multi-tenancy capability. During my tests I deployed the TempoStack resource and also created a Helm Chart that is able to render this resource. However, the Operator itself also provides a TempoMonolith resource. This will put everything into one Pod, while the TempoStack rolls out the stack in separate containers (like ingester, gateway, etc.).

My Helm Chart currently does not support the TempoMonolith resource yet. If you require this, please ping me and I will try to add it.

Be sure that the S3 bucket is available and a Secret (here called tempo-s3) with the following keys exists (valid for OpenShift Data Foundation): access_key_id, access_key_secret, bucket, endpoint. The layout of the Secret will look slightly different depending on the S3 storage backend you are using.

Create the TempoStack instance:

The following TempoStack resource has been used

apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
name: simplest (1)
namespace: tempostack (2)
spec:
managementState: Managed
replicationFactor: 1 (3)
# Resource limits
resources: (4)
total:
limits:
cpu: "2"
memory: 2Gi
# Trace retention
retention: (5)
global:
traces: 48h0m0s
# S3 storage configuration
storage:
secret:
credentialMode: static (6)
name: tempo-s3 (7)
type: s3
tls:
enabled: false
storageSize: 500Gi
# Multi-tenancy configuration
tenants:
mode: openshift
authentication: (8)
- tenantId: 1610b0c3-c509-4592-a256-a1871353dbfa
tenantName: tenantA
- tenantId: 1610b0c3-c509-4592-a256-a1871353dbfb
tenantName: tenantB
- tenantId: 1610b0c3-c509-4592-a256-a1871353dbfc
tenantName: tenantC
# Gateway and UI
template: (9)
gateway:
enabled: true
component:
replicas: 1
ingress:
type: route
route:
termination: reencrypt
queryFrontend:
component:
replicas: 1
jaegerQuery:
enabled: true
servicesQueryDuration: 72h0m0s

1	Name of the TempoStack instance
2	Namespace of the TempoStack instance
3	Integer value for the number of ingesters that must acknowledge the data from the distributors before accepting a span.
4	Defines resources for the TempoStack instance. Default is (limit only) 2 CPU and 2Gi memory.
5	Configuration options for retention of traces. The default value is 48h.
6	Credential mode for the S3 storage. Depends how the storage will be integrated. Default is static.
7	Secret name for the S3 storage.
8	Configuration options for the tenants. In this example: tenantA, tenantB, tenantC. Consists of tenantName and tenantId, both can be defined by the user.
9	Configuration options for the different Tempo components. In this example: gateway, query-frontend. Other components could be: distributor, ingester, compactor or querier.

Key Configuration Points:

Multi-tenancy: Supports 3 tenants currently (tenantA, tenantB, tenantC). This part must be modified when a new tenant enters the realm.
Retention: Traces stored for 48 hours.
Storage: Uses S3-compatible backend (requires separate secret).
Gateway: Exposes OTLP endpoint with TLS.

Step 3: Configure RBAC for TempoStack Trace Access read/write

Set up ClusterRoles to control who can read and write traces.

The ClusterRoles must be updated whenever a new tenant is configured in TempoStack. The name of the tenant must be added in the resources array.

Traces Reader Role:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tempostack-traces-reader (1)
rules:
- verbs: (2)
- get
apiGroups:
- tempo.grafana.com
resources:
- tenantA (3)
- tenantB
- tenantC
resourceNames:
- traces

1	Name of the ClusterRole
2	Verbs for the ClusterRole. apiGroups: tempo.grafana.com resources: List of Tenants resourceNames: traces verbs: get
3	List of tenants that are allowed to read the traces.

Bind Reader Role to Authenticated Users:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: tempostack-traces-reader
subjects:
- kind: Group
apiGroup: rbac.authorization.k8s.io
name: 'system:authenticated' (1)
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: tempostack-traces-reader

1	The Group that is allowed to read the traces. In this example: system:authenticated. This means ALL authenticated users will be able to read all the traces (see warning below).

With this ClusterRoleBinding, anybody who is authenticated (system:authenticated) will be able to see the traces for the defined tenants (A, B, C). This is for an easy showcase in this article. For production environments, you should implement more granular RBAC controls per tenant.

Traces Writer Role:

This ClusterRole is used to write traces into TempoStack. Typically you will use this ClusterRole for the OpenTelemetry Collector, so it can write into TempoStack.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tempostack-traces-write (1)
rules:
- verbs:
- create (2)
apiGroups:
- tempo.grafana.com
resources: (3)
- tenantB
- tenantA
- tenantC
resourceNames:
- traces

1	Name of the ClusterRoleBinding
2	This time the verb is create. This means the user will be able to write new traces into TempoStack.
3	List of tenants.

Bind Writer Role to Central Collector:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: tempostack-traces
subjects:
- kind: ServiceAccount
name: otel-collector (1)
namespace: tempostack (2)
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: tempostack-traces-write

1	The ServiceAccount that is allowed to write the traces. In this example: otel-collector. We will create this Service Account in the next article.
2	The namespace of the ServiceAccount. In this example: tempostack.

GitOps Deployment

While the above is good for quick tests, it always makes sense to have a proper GitOps deployment. I have created a Chart and GitOps configuration that will:

Deploy the Tempo Operator
Configure the TempoStack instance
Configure the RBAC configurations for the TempoStack instance

The following sources will be used:

Helm Repository - to fetch the Helm Chart for the Tempo Operator including required Sub-Charts.
Setup Tempo Operator - To deploy and configure the Tempo Operator, configure object storage, magically create a Secret with the required keys, etc.

Feel free to clone or use whatever you need.

The following Sub-Charts are used:

helper-objectstore (version ~1.0.0) - Creates S3 Bucket
helper-odf-bucket-secret (version ~1.0.0) - Creates the Secret usable by the TempoStack instance
helper-operator (version ~1.0.18) - Installs the Tempo Operator
helper-status-checker (version ~4.0.0) - Verifies the status of the Tempo Operator
tempo-tracing (version ~1.0.0) - Installs the TempoStack instance
tpl (version ~1.0.0) - Template Library

The this Argo CD Application we can deploy the Tempo Operator:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: setup-tempo-operator
namespace: openshift-gitops
spec:
destination:
name: in-cluster (1)
namespace: openshift-tempo-operator (2)
info:
- name: Description
value: ApplicationSet that Deploys on Management Cluster Configuration (using Git Generator)
project: in-cluster (3)
source:
path: clusters/management-cluster/setup-tempo-operator (4)
repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops' (5)
targetRevision: main
syncPolicy:
retry:
backoff:
duration: 5s
factor: 2
maxDuration: 3m
limit: 5

1	Target cluster, here the local cluster
2	Namespace of the target cluster, here the Operator will be installed.
3	Project of the target cluster
4	Path to the Git repository
5	URL to the Git repository

Since many things are moving in the background, it will take a while until the Argo CD Application is synced (i.e. Creation of S3 Bucket)

This will create the Argo CD Application that can be synchronized with the cluster:

As seen above, many resources are created using a single Helm Chart (ok, with some Sub-Charts). The actual configuration is done in the values.yaml file. The full values file is quite long, so I will break it down to the different Sub-Charts and add the complete file at the end that latest file I am using for testing can be found at values.yaml.

Values File Snippets for GitOps

TempStack Settings

The following will configure the TempoStack resource. Verify the upstream Helm Chart tempo-tracing for detailed and additional information about the settings.

tempo-tracing:
tempostack: (1)
enabled: true
name: simplest
managementState: Managed
namespace: (2)
create: true
name: tempostack
descr: "Namespace for the TempoStack"
display: "TempoStack"
additionalAnnotations: {}
additionalLabels: {}
storage: (3)
secret:
name: tempo-s3
type: s3
credentialMode: static
storageSize: 500Gi
replicationFactor: 1
serviceAccount: tempo-simplest (4)
tenants: (5)
mode: openshift
enabled: true
authentication: (6)
- tenantName: 'tenantA'
tenantId: '1610b0c3-c509-4592-a256-a1871353dbfc'
permissions: (7)
- write
- read
- tenantName: 'tenantB'
tenantId: '1610b0c3-c509-4592-a256-a1871353dbfd'
permissions:
- write
- read
observability:
enabled: true
tracing:
jaeger_agent_endpoint: 'localhost:6831'
otlp_http_endpoint: 'http://localhost:4320'
template:
gateway:
enabled: true
rbac: false
ingress:
type: 'route'
termination: 'reencrypt'
component:
replicas: 1
queryFrontend:
jaegerQuery:
enabled: true
component:
replicas: 1

1	Basic settings, like name of the instance
2	Namespace for the TempoStack instance.
3	Storage configuration for the TempoStack instance. Here we use type s3 and the Secret called "tempo-s3" which will be generated.
4	ServiceAccount for the TempoStack instance.
5	Tenant configuration for the TempoStack instance.
6	List of tenants. This list must be extended when new tenants shall be configured.
7	RBAC permissions for this tenant. This will add the tenant as resource to the READ and/or WRITE ClusterRole

Tempo Operator Deployment

The following settings will deploy the Operator and verify the status of the Operator installation. Only when the Operator has been installed successfully will Argo CD continue with the synchronization.

helper-operator:
operators:
tempo-operator: (1)
enabled: true
namespace: (2)
name: openshift-tempo-operator
create: true
subscription: (3)
channel: stable
approval: Automatic
operatorName: tempo-product
source: redhat-operators
sourceNamespace: openshift-marketplace
operatorgroup:
create: true
notownnamespace: true
########################################
# SUBCHART: helper-status-checker
# Verify the status of a given operator.
########################################
helper-status-checker:
enabled: true
approver: false (4)
checks:
- operatorName: tempo-operator (5)
namespace:
name: openshift-tempo-operator
syncwave: 1
serviceAccount:
name: "status-checker-tempo"

1	Install the Operator
2	Namespace settings of the Operator
3	Settings of the Operator itself, like name, channel and approval strategy.
4	Approver settings for the status checker. Here disabled, because we are using automatic approval.
5	Operator name to be verified.

S3 Bucket Deployment

The following Sub-Chart can be used to automatically create a Bucket in OpenShift Data Foundation.

########################################
# SUBCHART: helper-objectstore
# A helper chart that simply creates another backingstore for logging.
# This is a chart in a very early state, and not everything can be customized for now.
# It will create the objects:
# - BackingStore
# - BackingClass
# - StorageClass
# NOTE: Currently only PV type is supported
########################################
helper-objectstore:
# -- Enable objectstore configuration
# @default -- false
enabled: true
# -- Syncwave for Argo CD
# @default - 1
syncwave: 1
# -- Name of the BackingStore
backingstore_name: tempo-backingstore
# -- Size of the BackingStore that each volume shall have.
backingstore_size: 400Gi (1)
# -- CPU Limit for the Noobaa Pod
# @default -- 500m
limits_cpu: 500m
# -- Memory Limit for the Noobaa Pod.
# @default -- 2Gi
limits_memory: 2Gi
pvPool:
# -- Number of volumes that shall be used
# @default -- 1
numOfVolumes: 1
# Type of BackingStore. Currently pv-pool is the only one supported by this Helm Chart.
# @default -- pv-pool
type: pv-pool
# -- The StorageClass the BackingStore is based on
baseStorageClass: gp3-csi (2)
# -- Name of the StorageClass that shall be created for the bucket.
storageclass_name: tempo-bucket-storage-class (3)
# Bucket that shall be created
bucket:
# -- Shall a new bucket be enabled?
# @default -- false
enabled: true
# -- Name of the bucket that shall be created
name: tempo-bucket (4)
# -- Target Namespace for that bucket.
namespace: tempostack (5)
# -- Syncwave for bucketclaim creation. This should be done very early, but it depends on ODF.
# @default -- 2
syncwave: 2
# -- Name of the storageclass for our bucket
# @default -- openshift-storage.noobaa.io
storageclass: tempo-bucket-storage-class (6)

1	Size of the bucket.
2	StorageClass the bucket is based on.
3	Name of the StorageClass that shall be created for the bucket.
4	Name of the bucket that shall be created.
5	Namespace for the bucket.
6	StorageClass for the bucket.

Automatic Secret Creation for TempoStack

TempoStack is expecting a Secret with the following keys:

access_key_id
access_key_secret
bucket
endpoint
region (only for specific settings)

OpenShift Data Foundation creates a secret with access_key_id and access_key_secret and a ConfigMap with endpoint, region and the name of the bucket. Unfortunately, this does not work for TempoStack. Therefore, we are using a helper-odf-bucket-secret chart that will create a new Secret with the required keys. This chart creates a Job that reads the required information and creates a new Secret with the required keys.

##############################################
# SUBCHART: helper-odf-bucket-secret
# Creates a Secret that Tempo requires
#
# A Kubernetes Job is created, that reads the
# data from the Secret and ConfigMap and
# creates a new secret for Tempo.
##############################################
helper-odf-bucket-secret:
# -- Enable Job to create a Secret for TempoStack.
# @default -- false
enabled: true
# -- Syncwave for Argo CD.
# @default -- 3
syncwave: 3
# -- Namespace where TempoStack is deployed and where the Secret shall be created.
namespace: tempostack
# -- Name of Secret that shall be created.
secretname: tempo-s3 (1)
# Bucket Configuration
bucket:
# -- Name of the Bucket shall has been created.
name: tempo-bucket (2)
# -- Keys that shall be used to create the Secret.
keys: (3)
# -- Overwrite access_key_id key.
# @default -- access_key_id
access_key_id: access_key_id
# -- Overwrite access_key_secret key.
# @default -- access_key_secret
access_key_secret: access_key_secret
# -- Overwrite bucket key.
# @default -- bucket
bucket: bucket
# -- Overwrite endpoint key.
# @default -- endpoint
endpoint: endpoint
# -- Overwrite region key. Region is only set if set_region is true.
# @default -- region
region: region
# -- Set region key.
# @default -- false
set_region: false (4)

1	Name of the Secret that shall be created.
2	Name of the bucket that shall be used.
3	Keys that shall be used to create the Secret.
4	Set region key. Here disabled, because we are not using a specific region.

Using OpenShift Data Foundation with TempoStack requires you to NOT set a region. Therefore, it is disabled above.

Complete values file

To see the whole file expand the code:

Expand me...

tempo: &channel-tempo stable
tempo-namespace: &tempo-namespace openshift-tempo-operator
bucketname: &bucketname tempo-bucket
tempo-secret: &tempo-secret-name tempo-s3
storageclassname: &storageclassname tempo-bucket-storage-class
tempostack-namespace: &tempostack-namespace tempostack
tempo-tracing:
tempostack:
enabled: true
name: simplest
managementState: Managed
namespace:
create: true
name: *tempostack-namespace
descr: "Namespace for the TempoStack"
display: "TempoStack"
additionalAnnotations: {}
additionalLabels: {}
storage:
secret:
name: tempo-s3
type: s3
credentialMode: static
storageSize: 500Gi
replicationFactor: 1
serviceAccount: tempo-simplest
tenants:
mode: openshift
enabled: true
authentication:
- tenantName: 'tenantA'
tenantId: '1610b0c3-c509-4592-a256-a1871353dbfc'
permissions:
- write
- read
- tenantName: 'tenantB'
tenantId: '1610b0c3-c509-4592-a256-a1871353dbfd'
permissions:
- write
- read
observability:
enabled: true
tracing:
jaeger_agent_endpoint: 'localhost:6831'
otlp_http_endpoint: 'http://localhost:4320'
template:
gateway:
enabled: true
rbac: false
ingress:
type: 'route'
termination: 'reencrypt'
component:
replicas: 1
queryFrontend:
jaegerQuery:
enabled: true
component:
replicas: 1
######################################
# SUBCHART: helper-operator
# Operators that shall be installed.
######################################
helper-operator:
operators:
tempo-operator:
enabled: true
namespace:
name: *tempo-namespace
create: true
subscription:
channel: *channel-tempo
approval: Automatic
operatorName: tempo-product
source: redhat-operators
sourceNamespace: openshift-marketplace
operatorgroup:
create: true
notownnamespace: true
########################################
# SUBCHART: helper-status-checker
# Verify the status of a given operator.
########################################
helper-status-checker:
enabled: true
approver: false
checks:
- operatorName: tempo-operator
namespace:
name: *tempo-namespace
syncwave: 1
serviceAccount:
name: "status-checker-tempo"
########################################
# SUBCHART: helper-objectstore
# A helper chart that simply creates another backingstore for logging.
# This is a chart in a very early state, and not everything can be customized for now.
# It will create the objects:
# - BackingStore
# - BackingClass
# - StorageClass
# NOTE: Currently only PV type is supported
########################################
helper-objectstore:
# -- Enable objectstore configuration
# @default -- false
enabled: true
# -- Syncwave for Argo CD
# @default - 1
syncwave: 1
# -- Name of the BackingStore
backingstore_name: tempo-backingstore
# -- Size of the BackingStore that each volume shall have.
backingstore_size: 400Gi
# -- CPU Limit for the Noobaa Pod
# @default -- 500m
limits_cpu: 500m
# -- Memory Limit for the Noobaa Pod.
# @default -- 2Gi
limits_memory: 2Gi
pvPool:
# -- Number of volumes that shall be used
# @default -- 1
numOfVolumes: 1
# Type of BackingStore. Currently pv-pool is the only one supported by this Helm Chart.
# @default -- pv-pool
type: pv-pool
# -- The StorageClass the BackingStore is based on
baseStorageClass: gp3-csi
# -- Name of the StorageClass that shall be created for the bucket.
storageclass_name: *storageclassname
# Bucket that shall be created
bucket:
# -- Shall a new bucket be enabled?
# @default -- false
enabled: true
# -- Name of the bucket that shall be created
name: *bucketname
# -- Target Namespace for that bucket.
namespace: *tempo-namespace
# -- Syncwave for bucketclaim creation. This should be done very early, but it depends on ODF.
# @default -- 2
syncwave: 2
# -- Name of the storageclass for our bucket
# @default -- openshift-storage.noobaa.io
storageclass: *storageclassname
##############################################
# SUBCHART: helper-odf-bucket-secret
# Creates a Secret that Tempo requires
#
# A Kubernetes Job is created, that reads the
# data from the Secret and ConfigMap and
# creates a new secret for Tempo.
##############################################
helper-odf-bucket-secret:
# -- Enable Job to create a Secret for TempoStack.
# @default -- false
enabled: true
# -- Syncwave for Argo CD.
# @default -- 3
syncwave: 3
# -- Namespace where TempoStack is deployed and where the Secret shall be created.
namespace: *tempostack-namespace
# -- Name of Secret that shall be created.
secretname: *tempo-secret-name
# Bucket Configuration
bucket:
# -- Name of the Bucket shall has been created.
name: *bucketname
# -- Keys that shall be used to create the Secret.
keys:
# -- Overwrite access_key_id key.
# @default -- access_key_id
access_key_id: access_key_id
# -- Overwrite access_key_secret key.
# @default -- access_key_secret
access_key_secret: access_key_secret
# -- Overwrite bucket key.
# @default -- bucket
bucket: bucket
# -- Overwrite endpoint key.
# @default -- endpoint
endpoint: endpoint
# -- Overwrite region key. Region is only set if set_region is true.
# @default -- region
region: region
# -- Set region key.
# @default -- false
set_region: false

The TempoStack

Let’s imagine all of the above works and we have a TempoStack instance running in the namespace tempostack with the name simplest. Several Pods are running, like the distributor, ingester, gateway, query-frontend, compactor and querier. I admit it is not highly available, but this can be easily changed in the values file above (replica count).

oc get pods -n tempostack | grep simplest
tempo-simplest-compactor-584689c78f-t7pxb 1/1 Running 0 3d2h
tempo-simplest-distributor-6fb5d7dc9d-wrzt4 1/1 Running 0 3d2h
tempo-simplest-gateway-bbcb774b9-p44lq 2/2 Running 0 11h
tempo-simplest-ingester-0 1/1 Running 0 3d2h
tempo-simplest-querier-6cf9d7b6d8-mvc9d 1/1 Running 0 3d2h
tempo-simplest-query-frontend-7d859f9f9f-xzj97 3/3 Running 0 3d2h

Now we can continue with the OpenTelemetry Collector deployment. But before we do this, let’s first discuss how to:

Add Tracing UI to OpenShift

Extend the OpenShift UI

While we have Tempo installed, it will not be visible in the OpenShift UI by default. Here a separate Operator has been created by Red Hat, that will take care of this extension: Cluster Observability Operator.

This Operator be installed:

The Operator can be deployed with the Chart helper-operator as well. However, I did not merge this together with the TempoStack deployment, because this Operator also serves different purposes.

We only require a very small bit of this Operator. Amongst other resources, it also provides a resource called UIPlugin.

The resource must be configured as follows:

apiVersion: observability.openshift.io/v1alpha1
kind: UIPlugincd
metadata:
name: distributed-tracing (1)
spec:
type: DistributedTracing (2)

1	The name MUST be distributed-tracing.
2	The type MUST be DistributedTracing.

This will extend the OpenShift UI with a new navigation link "Observe" > "Traces".

On the next Episode

The next article will cover the deployment of the Central OpenTelemetry Collector. We will configure the RBAC permissions required for the Central Collector to enrich traces with Kubernetes metadata and deploy the Central OpenTelemetry Collector with its complete configuration.

The Hitchhiker's Guide to Observability - Central Collector - Part 3

Tue, 25 Nov 2025 00:00:00 +0000

With the architecture defined in Part 1 and TempoStack deployed in Part 2, it’s time to tackle the heart of our distributed tracing system: the Central OpenTelemetry Collector. This is the critical component that sits between your application namespaces and TempoStack, orchestrating trace flow, metadata enrichment, and tenant routing.

In this article, we’ll configure the RBAC permissions required for the Central Collector to enrich traces with Kubernetes metadata and deploy the Central OpenTelemetry Collector with its complete configuration. You’ll learn how to set up receivers for accepting traces from local collectors, configure processors to enrich traces with Kubernetes and OpenShift metadata, and implement routing connectors to direct traces to the appropriate TempoStack tenants based on namespace.

To be honest, this was the most challenging part of the entire setup to get right, as you can easily miss a setting, or misconfigure a part of the configuration. But once you understand the configuration, it becomes straightforward to extend and modify.

Central OpenTelemetry Collector - Step-by-Step Implementation

Prerequisites

Before starting, ensure you have completed the previous parts and have:

Part 1 completed: Hitchhiker’s Guide to Observability with OpenTelemetry and Tempo
Part 2 completed: Deploy Grafana Tempo and TempoStack
OpenShift cluster (v4.20) with Red Hat build of OpenTelemetry Operator installed (v0.135.0-1)
Access Cluster-admin access

Step 1: Verify/Deploy Red Hat Build of OpenTelemetry Operator

Verify if the Operator Red Hat Build of OpenTelemetry is installed and ready. The Operator itself is deployed in the namespace openshift-opentelemetry-operator

Step 2: Configure RBAC for Central Collector

For the OpenTelemetry Collector to function properly with processors like k8sattributes and resourcedetection, it requires cluster-wide read access to Kubernetes resources. Depending on the configuration of the central collector, you might need to configure different RBAC settings to allow the collector to perform specific things. The central collector in our example uses the k8sattributes processor and resourcedetection processor to enrich traces with Kubernetes and OpenShift metadata. These processors require read access to cluster resources.

Why These Permissions Are Needed

The k8sattributes processor enriches telemetry data by querying the Kubernetes API to add metadata such as:

Pod information: Pod name, UID, start time
Namespace details: Namespace name and labels
Deployment context: ReplicaSet and Deployment names
Node information: Node name where the pod is running

The resourcedetection processor detects the OpenShift environment by querying:

Infrastructure resources: Cluster name, platform type, region (from config.openshift.io/infrastructures)

Without these permissions, traces would lack critical context needed for debugging and analysis.

Always check the latest documentation of the appropriate component to get the most up to date information. I prefer to use separate ClusterRoles for each processor to keep the permissions as granular as possible. However, that causes some overhead, so you might want to combine the permissions into a single ClusterRole.

Create ClusterRole for Kubernetes Attributes

The following ClusterRole has been created:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: k8sattributes-otel-collector
rules:
# Permissions for k8sattributes processor
# Allows the collector to read pod, namespace, and replicaset information
# to enrich traces with Kubernetes metadata
- verbs:
- get
- watch
- list
apiGroups:
- '*'
resources:
- pods
- namespaces
- replicasets
# Permissions for resourcedetection processor (OpenShift)
# Allows detection of OpenShift cluster information
# such as cluster name, platform type, and region
- verbs:
- get
- watch
- list
apiGroups:
- config.openshift.io
resources:
- infrastructures
- infrastructures/status

Key Points:

Read-Only Access: The collector only needs get, watch, and list verbs (no write permissions)
Cluster-Wide Scope: ClusterRole grants permissions across all namespaces, necessary for monitoring multi-tenant environments
Essential Resources:
- pods: Source of trace context (which pod generated the span)
- namespaces: Namespace metadata and labels for routing
- replicasets: Determine the owning Deployment for better trace attribution
OpenShift Infrastructure: Access to config.openshift.io/infrastructures allows detection of cluster-level properties

Create ClusterRoleBinding

Our OpenTelemetry Collector will use the ServiceAccount otel-collector (in the namespace tempostack) to read the Kubernetes resources. Thus, we need to create a ClusterRoleBinding to grant the necessary permissions to the ServiceAccount.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: k8sattributes-collector-tempo
subjects:
- kind: ServiceAccount
name: otel-collector (1)
namespace: tempostack (2)
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: k8sattributes-otel-collector (3)

1	The ServiceAccount that is allowed to write the traces. In this example: otel-collector.
2	The namespace of the ServiceAccount. In this example: tempostack.
3	The reference to the ClusterRole

Security Note

The above permissions in the ClusterRole follow the principle of least privilege:

Only read operations are granted (no create, update, or delete)
Access is limited to specific resource types needed for metadata enrichment
The ServiceAccount otel-collector is dedicated to the collector and not shared with other applications

Step 3: Create ServiceAccount for Central OpenTelemetry Collector

The ServiceAccount used by the OpenTelemetry Collector. This is the service account that will be used to authenticate to the TempoStack instance. Thus, the Bindings created earlier to write into TempoStack and to read from Kubernetes will be required.

In this article we are installing the central Collector into the same namespace as the TempoStack instance. However, you might want to install it in a different namespace to keep the namespaces separated. Keep an eye on possible Network Policies that might be required to allow the communication between the namespaces.

apiVersion: v1
kind: ServiceAccount
metadata:
name: otel-collector
namespace: tempostack

Step 4: Deploy Central Collector

The central collector receives traces from local collectors, enriches them with Kubernetes metadata, and routes them to appropriate TempoStack tenants. For the sake of simplicity, I have taken snippets from the whole Configuration manifest. At the end of this section you will find the whole manifest.

Basic Configuration

The very basic settings of the OpenTelemetry Collector, are the amount of replicas, the ServiceAccount to use and the deployment mode.

The Collector can be deployed in one of the following modes:

Deployment (default) - Creates a Deployment with the given numbers of replicas.
StatefulSet - Creates a StatefulSet with the given numbers of replicas. Useful for stateful workloads, for example when using the Collector’s File Storage Extension or Tail Sampling Processor.
DaemonSet - Creates a DaemonSet with the given numbers of replicas. Useful for scraping telemetry data from every node, for example by using the Collector’s Filelog Receiver to read container logs.
Sidecar - Injects the Collector as a sidecar into the pod. Useful for accessing log files inside a container, for example by using the Collector’s Filelog Receiver and a shared volume such as emptyDir.

In the examples in this series of articles we will use the modes: deployment and sidecar.

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
name: otel
namespace: tempostack
spec:
mode: deployment (1)
replicas: 1 (2)
serviceAccount: otel-collector (3)
[...]

1	The deployment mode to use.
2	The number of replicas to use.
3	The ServiceAccount to use, created in the previous step.

Receivers

The Receivers are the components that receive the traces from the local collectors. Receivers accept data in a specified format and translate it into the internal format. In our example we want to receive the traces from the local collectors. For our tests we are using the oltp Receiver, which is collecting traces, metrics and logs by using the OpenTelemetry Protocol (OTLP).

The easiest configuration is:

 receivers:
otlp: (1)
protocols:
grpc:
endpoint: 0.0.0.0:4317 (2)
http:
endpoint: 0.0.0.0:4318 (3)

1	The name of the receiver.
2	The gRPC endpoint to receive.
3	The http endpoint to receive.

Besides the otlp receiver, there are other receivers available. For example:

Jaeger
Kubernetes Object Receiver
Kubelet Stats Receiver
Prometheus Receiver
Filelog Receiver
Journald Receiver
Kubernetes Events Receiver
Kubernetes Cluster Receiver
OpenCensus Receiver
Zipkin Receiver
Kafka Receiver.

Please check the OpenShift OTEL Receivers documentation for more details.

Processors

The Processors are the components that process the data after it is received and before it is exported. Processors are completely optional, but they are useful to transform, enrich, or filter traces.

The order of processors matters.

The example configuration is using the following processors:

k8sattributes: Can add Kubernetes metadata to the traces.
resourcedetection: Can detect OpenShift/K8s environment info.
memory_limiter: Periodically checks the Collector’s memory usage and pauses data processing when the soft memory limit is reached.
batch: Batches the traces for efficiency. This is a very important processor to improve the performance of the Collector.

Additional processors are available. Please check the OpenShift OTEL Processors documentation for more details.

Some processors will requires additional ClusterRole configuration.

 # Processors - enrich and batch traces
processors:
# Add Kubernetes metadata
k8sattributes: {} (1)
# Detect OpenShift/K8s environment info
resourcedetection: (2)
detectors:
- openshift
timeout: 2s
# Memory protection
memory_limiter: (3)
check_interval: 1s
limit_percentage: 75
spike_limit_percentage: 15
# Batch for efficiency
batch: (4)
send_batch_size: 10000
timeout: 10s

1	The k8sattributes processor to add Kubernetes metadata to the traces.
2	The resourcedetection processor to detect OpenShift/K8s environment info.
3	The memory_limiter processor to protect the Collector’s memory usage.
4	The batch processor to batch the traces for efficiency.

Connectors

A Connector joins two pipelines together. It consumes data as an exporter at the end of one pipeline and emits data as a receiver at the start of another pipeline. It can consume and emit data of the same or different data type. It can generate and emit data to summarize the consumed data, or it can merely replicate or route data.

Several Connectors are available, for example:

Count Connector: Counts traces spans, trace span events, metrics, metric data points, and log records.
Routing Connector: Routes the traces to different pipelines.
Forward Connector: Merges two pipelines of the same type.
Spanmetrics Connector: Aggregates Request, Error, and Duration (R.E.D) OpenTelemetry metrics from span data.

The OpenShift OTEL Connectors documentation lists all available Connectors and their configuration options.

In our example we are using the routing Connector, which is able to route the traces to different pipelines based on the namespace. This helps to route the traces to the correct tenant, without the need to configure this in the local OpenTelemetry Collector. (In other words, the project cannot change this setting, because it is configured in the central OpenTelemetry Collector.)

In this example traces from the namespace "team-a" will be routed to the pipeline "tenantA", traces from the namespace "team-b" will be routed to the pipeline "tenantB", and so on.

 # Connectors - route traces to different pipelines
connectors:
routing/traces:
default_pipelines: (1)
- traces/Default
error_mode: ignore (2)
table:
# Route team-a namespace to tenantA
- statement: route() where attributes["k8s.namespace.name"] == "team-a" (3)
pipelines:
- traces/tenantA
# Route team-b namespace to tenantB
- statement: route() where attributes["k8s.namespace.name"] == "team-b"
pipelines:
- traces/tenantB
# Route team-c namespace to tenantC
- statement: route() where attributes["k8s.namespace.name"] == "team-c"
pipelines:
- traces/tenantC

1	Destination pipelines for routing the telemetry data for which no routing condition is satisfied.
2	Error-handling mode: Defines how the connector handles routing errors: `propagate`: Logs an error and drops the payload (stops processing) `ignore`: Logs the error but continues attempting to match subsequent routing rules `silent`: Same as `ignore` but without logging Default: `propagate`
3	Route traces from namespace team-a to the pipeline tenantA.

Exporters

The Exporters are the components that export the traces to a destination. Exporters accept data in a specified format and translate it into the destination format. In our example we want to export the traces to the TempoStack instance. The X-Scope-OrgID header is used to identify the tenant and is sent to the TempoStack instance. The authentication is done by using the ServiceAccount token.

Many different Exporters are available. The OpenShift OTEL Exporters documentation lists all available Exporters and their configuration options.

For our tests we are using the otlp Exporter, which will export using the OpenTelemetry Protocol (OTLP):

 exporters:
# Tenant A exporter
otlp/tenantA: (1)
endpoint: tempo-simplest-gateway:8090 (2)
auth:
authenticator: bearertokenauth
headers:
X-Scope-OrgID: tenantA (3)
tls:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
insecure_skip_verify: true (4)
server_name_override: tempo-simplest-gateway.tempostack.svc.cluster.local (5)

1	Protocol/name of the exporter.
2	The endpoint to export to. Here, the endpoint is the address of the TempoStack instance.
3	The scope org ID to use.
4	Whether to skip certificate verification. I did not bother with certificates in this example.
5	The server name override to use. This was just for testing purposes and can be omitted.

Extensions

The Extensions extend the Collector capabilities. In our example we are using the bearertokenauth Extension, which is used to authenticate to the TempoStack instance.

 # Extensions - authentication
extensions:
bearertokenauth:
filename: /var/run/secrets/kubernetes.io/serviceaccount/token

Service:

Components are enabled by adding them into a Pipeline. If a component is not configured in a pipeline, it is not enabled.

In this example we are using the following pipelines (snippets):

traces/in: The incoming traces pipeline.
traces/tenantA: The tenant A pipeline.

The traces/in pipeline is the incoming traces pipeline. It is used to receive the traces from the local collectors. It leverages the otlp Receiver to receive the traces. It uses the resourcedetection, k8sattributes, memory_limiter, and batch processors to process the traces. And finally, it uses the routing/traces Connector to route the traces.

The routing/traces Connector is used to route the traces to the correct tenant based on the namespace.

The traces/tenantA Pipeline will receive the traces from routing/traces and export the traces to otlp/tenantA which then sends everything to TempoStack to store the traces using the header X-Scope-OrgID: tenantA.

 # Service pipelines
service:
extensions:
- bearertokenauth
pipelines:
# Incoming traces pipeline
traces/in: (1)
receivers: (2)
- otlp
processors: (3)
- resourcedetection
- k8sattributes
- memory_limiter
- batch
exporters: (4)
- routing/traces
# Tenant A pipeline
traces/tenantA: (5)
receivers: (6)
- routing/traces
exporters: (7)
- otlp/tenantA

1	The incoming traces pipeline. It takes the traces from the otlp receiver.
2	The receivers to use.
3	The processors to use.
4	The exporters to use. It is exporting the data to the routing connector.
5	The tenant A pipeline.
6	The receivers to use.
7	The exporters to use. It is exporting to the exporter, that will send the data to TempoStack.

The service automatically creates Kubernetes Services for the receivers. The Central Collector will be accessible at otel-collector.tempostack.svc.cluster.local:4317 (gRPC) and :4318 (HTTP) for local collectors to send traces.

Data Flow Visualization

To better understand how traces flow through the Central Collector, the following Mermaid diagram visualizes the complete journey from application to storage using team-a as an example:

---
title: "Central Collector Data Flow (Wrapped Layout)"
config:
theme: 'dark'
---
flowchart LR
%% --------------------------
%% Column 1: Local Collector
%% --------------------------
subgraph local["(team-a namespace)"]
app["Application<br/>(team-a)"]
end
%% --------------------------
%% Column 2: Central Collector
%% --------------------------
subgraph central["Central Collector (tempostack namespace)"]
%% We force this specific column to stack vertically (Top-Bottom)
direction TB
%% --- ROW 1: Receiver + Processors ---
subgraph row1[" "]
direction LR
receiver["OTLP Receiver<br/>Port: 4317/4318"]
subgraph pipeline_in["traces/in Processors"]
proc1["resourcedetection<br/>(Detect OpenShift)"]
proc2["k8sattributes<br/>(Add K8s metadata)"]
proc3["memory_limiter<br/>(Protect memory)"]
proc4["batch<br/>(Batch traces)"]
end
end
%% --- ROW 2: Connector + Tenant Pipeline ---
subgraph row2[" "]
direction LR
exporter_routing["Exporter: to connector"]
connector["Export to routing/traces Connector<br/>(Route by namespace)"]
subgraph pipeline_tenant["Pipeline: traces/tenantA"]
receiver_tenant["Receiver:<br/>routing/traces"]
exporter_tenant["Exporter:<br/>otlp/tenantA"]
end
end
end
%% --------------------------
%% Column 3: TempoStack
%% --------------------------
subgraph tempo["TempoStack"]
gateway["Tempo Gateway<br/>(X-Scope-OrgID: tenantA)"]
storage["S3 Storage<br/>(tenantA traces)"]
end
%% --------------------------
%% Connections
%% --------------------------
app -->|"OTLP traces"| receiver
receiver --> proc1 --> proc2 --> proc3 --> proc4
%% The Wrap: Connect end of Row 1 to start of Row 2
proc4 --> exporter_routing
exporter_routing --> connector
connector -->|"Route by namespace=team-a<br/>→ tenantA"| receiver_tenant
receiver_tenant --> exporter_tenant
exporter_tenant -->|"OTLP + bearer token<br/>Header: X-Scope-OrgID"| gateway
gateway --> storage
%% --------------------------
%% Styles
%% --------------------------
classDef appStyle fill:#2f652a,stroke:#2f652a,stroke-width:2px
classDef receiverStyle fill:#425cc6,stroke:#425cc6,stroke-width:2px
classDef processorStyle fill:#4a90e2,stroke:#4a90e2,stroke-width:2px
classDef connectorStyle fill:#906403,stroke:#906403,stroke-width:2px
classDef exporterStyle fill:#7b4397,stroke:#7b4397,stroke-width:2px
classDef tempoStyle fill:#d35400,stroke:#d35400,stroke-width:2px
class app appStyle
class receiver,receiver_tenant receiverStyle
class proc1,proc2,proc3,proc4 processorStyle
class connector connectorStyle
class exporter_tenant exporterStyle
class exporter_routing exporterStyle
class gateway,storage tempoStyle
%% Hide the structural boxes for the rows so they look seamless
style row1 fill:none,stroke:none
style row2 fill:none,stroke:none

Key Points in the Flow:

Application in team-a namespace sends traces to local collector
Local Collector forwards to Central Collector’s OTLP receiver
Pipeline traces/in processes the traces sequentially:
- Detects OpenShift environment info
- Adds Kubernetes metadata (namespace, pod, labels)
- Applies memory limits
- Batches traces for efficiency
Routing Connector examines the namespace attribute and routes to the correct tenant pipeline
Pipeline traces/tenantA receives from connector and exports to TempoStack
Exporter adds authentication (bearer token) and tenant ID header (X-Scope-OrgID: tenantA)
TempoStack receives and stores traces in the appropriate tenant storage

This architecture ensures complete isolation between tenants while maintaining a single, centralized collection point.

Complete OpenTelemetry Collector Manifest

Let’s put everything together in the complete OpenTelemetry Collector Manifest. The following defines the Central OpenTelemetry Collector:

mode: The deployment mode to use.
replicas: The number of replicas to use.
serviceAccount: The ServiceAccount to use, created in the previous step.
config: The configuration of the OpenTelemetry Collector.
- receivers: The receivers to use.
- processors: The processors to use.
- connectors: The connectors to use.
- exporters: The exporters to use.
- extensions: The extensions to use.
- service: The service to use.

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
name: otel
namespace: tempostack
spec:
mode: deployment
replicas: 1
serviceAccount: otel-collector
config:
# Receivers - accept traces from local collectors
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
# Processors - enrich and batch traces
processors:
# Add Kubernetes metadata
k8sattributes: {}
# Detect OpenShift/K8s environment info
resourcedetection:
detectors:
- openshift
timeout: 2s
# Memory protection
memory_limiter:
check_interval: 1s
limit_percentage: 75
spike_limit_percentage: 15
# Batch for efficiency
batch:
send_batch_size: 10000
timeout: 10s
# Connectors - route traces to different pipelines
connectors:
routing/traces:
default_pipelines:
- traces/Default
error_mode: ignore
table:
# Route team-a namespace to tenantA
- statement: route() where attributes["k8s.namespace.name"] == "team-a"
pipelines:
- traces/tenantA
# Route team-b namespace to tenantB
- statement: route() where attributes["k8s.namespace.name"] == "team-b"
pipelines:
- traces/tenantB
# Exporters - send to TempoStack
exporters:
# Default tenant exporter
otlp/Default:
endpoint: tempo-simplest-gateway:8090
auth:
authenticator: bearertokenauth
headers:
X-Scope-OrgID: dev
tls:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
insecure_skip_verify: true
server_name_override: tempo-simplest-gateway.tempostack.svc.cluster.local
# Tenant A exporter
otlp/tenantA:
endpoint: tempo-simplest-gateway:8090
auth:
authenticator: bearertokenauth
headers:
X-Scope-OrgID: tenantA
tls:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
insecure_skip_verify: true
server_name_override: tempo-simplest-gateway.tempostack.svc.cluster.local
# Tenant B exporter
otlp/tenantB:
endpoint: tempo-simplest-gateway:8090
auth:
authenticator: bearertokenauth
headers:
X-Scope-OrgID: tenantB
tls:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
insecure_skip_verify: true
server_name_override: tempo-simplest-gateway.tempostack.svc.cluster.local
# Extensions - authentication
extensions:
bearertokenauth:
filename: /var/run/secrets/kubernetes.io/serviceaccount/token
# Service pipelines
service:
extensions:
- bearertokenauth
pipelines:
# Incoming traces pipeline
traces/in:
receivers:
- otlp
processors:
- resourcedetection
- k8sattributes
- memory_limiter
- batch
exporters:
- routing/traces
# Default tenant pipeline
traces/Default:
receivers:
- routing/traces
exporters:
- otlp/Default
# Tenant A pipeline
traces/tenantA:
receivers:
- routing/traces
exporters:
- otlp/tenantA
# Tenant B pipeline
traces/tenantB:
receivers:
- routing/traces
exporters:
- otlp/tenantB

GitOps Deployment

While the above is good for quick tests, it always makes sense to have a proper GitOps deployment. I have created a Chart and GitOps configuration that will:

Deploy the OTEL Operator
Configure the the Central Collector instance

The following sources will be used:

Helm Repository - to fetch the Helm Chart for the OTEL Operator including required Sub-Charts.
Setup OTEL Operator - To deploy and configure the OpenTelemetry Operator.

Feel free to clone or use whatever you need.

The following Sub-Charts are used:

helper-operator (version ~1.0.18) - Installs the OpenTelemetry Operator
helper-status-checker (version ~4.0.0) - Verifies the status of the OpenTelemetry Operator
opentelemetry-collector (version ~1.0.0) - Creates the Central OpenTelemetry Collector instance
tpl (version ~1.0.0) - Template Library

The following Argo CD Application will deploy the OTEL Operator:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: setup-otel-operator
namespace: openshift-gitops
spec:
destination:
name: in-cluster (1)
namespace: openshift-opentelemetry-operator (2)
info:
- name: Description
value: ApplicationSet that Deploys on Management Cluster Configuration (using Git Generator)
project: in-cluster (3)
source:
path: clusters/management-cluster/setup-otel-operator (4)
repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops' (5)
targetRevision: main
syncPolicy:
retry:
backoff:
duration: 5s
factor: 2
maxDuration: 3m
limit: 5

1	Target cluster, here the local cluster
2	Namespace of the target cluster, here the Operator will be installed.
3	Project of the target cluster
4	Path to the Git repository
5	URL to the Git repository

This will create the Argo CD Application that can be synchronized with the cluster:

Complete values file

To see the whole file expand the code:

---
otel: &channel-otel stable
otel-namespace: &otel-namespace openshift-opentelemetry-operator
######################################
# SUBCHART: helper-operator
# Operators that shall be installed.
######################################
helper-operator:
operators:
opentelemetry-product:
enabled: true
namespace:
name: *otel-namespace
create: true
subscription:
channel: *channel-otel
approval: Automatic
operatorName: opentelemetry-product
source: redhat-operators
sourceNamespace: openshift-marketplace
operatorgroup:
create: true
notownnamespace: true
########################################
# SUBCHART: helper-status-checker
# Verify the status of a given operator.
########################################
helper-status-checker:
enabled: true
approver: false
checks:
- operatorName: opentelemetry-product
namespace:
name: *otel-namespace
syncwave: 1
serviceAccount:
name: "status-checker-otel"
rh-build-of-opentelemetry:
#########################################################################################
# namespace ... disabled here, since we deployed it via Tempo already
#########################################################################################
namespace:
name: tempostack
create: false
#########################################################################################
# OPENTELEMETRY COLLECTOR - Production Configuration
#########################################################################################
collector:
enabled: true
name: otel
mode: deployment
replicas: 1
serviceAccount: otel-collector
managementState: managed
resources: {}
tolerations: []
config:
connectors:
routing/traces:
default_pipelines:
- traces/Default
error_mode: ignore
table:
- pipelines:
- traces/tenantA
statement: 'route() where attributes["k8s.namespace.name"] == "team-a"'
- pipelines:
- traces/tenantB
statement: 'route() where attributes["k8s.namespace.name"] == "team-b"'
- pipelines:
- traces/tenantC
statement: 'route() where attributes["k8s.namespace.name"] == "team-c"'
- pipelines:
- traces/tenantD
statement: 'route() where attributes["k8s.namespace.name"] == "team-d"'
- pipelines:
- traces/tenantX
statement: 'route() where attributes["k8s.namespace.name"] == "mockbin-1"'
receivers:
# OTLP receivers for traces, metrics, and logs
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
# Jaeger receiver (if migrating from Jaeger)
jaeger:
protocols:
grpc:
endpoint: 0.0.0.0:14250
thrift_http:
endpoint: 0.0.0.0:14268
processors:
batch:
send_batch_size: 10000
timeout: 10s
k8sattributes: {}
memory_limiter:
check_interval: 1s
limit_percentage: 75
spike_limit_percentage: 15
resourcedetection:
detectors:
- openshift
timeout: 2s
exporters:
otlp/tenantX:
auth:
authenticator: bearertokenauth
endpoint: 'tempo-simplest-gateway:8090'
headers:
X-Scope-OrgID: tenantX
tls:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
insecure: true
insecure_skip_verify: true
server_name_override: tempo-simplest-gateway.tempostack.svc.cluster.local
otlp:
auth:
authenticator: bearertokenauth
endpoint: 'tempo-simplest-gateway:8090'
headers:
X-Scope-OrgID: prod
tls:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
insecure: true
insecure_skip_verify: true
server_name_override: tempo-simplest-gateway.tempostack.svc.cluster.local
otlp/tenantA:
auth:
authenticator: bearertokenauth
endpoint: 'tempo-simplest-gateway:8090'
headers:
X-Scope-OrgID: tenantA
tls:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
insecure: true
insecure_skip_verify: true
server_name_override: tempo-simplest-gateway.tempostack.svc.cluster.local
otlp/tenantB:
auth:
authenticator: bearertokenauth
endpoint: 'tempo-simplest-gateway:8090'
headers:
X-Scope-OrgID: tenantB
tls:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
insecure: true
insecure_skip_verify: true
server_name_override: tempo-simplest-gateway.tempostack.svc.cluster.local
otlp/Default:
auth:
authenticator: bearertokenauth
endpoint: 'tempo-simplest-gateway:8090'
headers:
X-Scope-OrgID: dev
tls:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
insecure: true
insecure_skip_verify: true
server_name_override: tempo-simplest-gateway.tempostack.svc.cluster.local
extensions:
bearertokenauth:
filename: /var/run/secrets/kubernetes.io/serviceaccount/token
service:
extensions:
- bearertokenauth
pipelines:
traces/Default:
exporters:
- otlp/Default
receivers:
- routing/traces
traces/in:
exporters:
- routing/traces
processors:
- resourcedetection
- k8sattributes
- memory_limiter
- batch
receivers:
- otlp
traces/tenantA:
exporters:
- otlp/tenantA
receivers:
- routing/traces
traces/tenantB:
exporters:
- otlp/tenantB
receivers:
- routing/traces

The Central OpenTelemetry Collector

With this deployment a single Pod (because we have set the replica count to 1) is running in the namespace tempostack with the name otel-collector.

oc get pods -n tempostack | grep otel-col
otel-collector-75f8794dc6-rbq26 1/1 Running 0 52m

Stay Tuned

The next article will cover deploying an example application (Mockbin) and configuring Local OpenTelemetry Collectors in application namespaces to send traces to this Central Collector. We’ll also demonstrate how the namespace-based routing automatically directs traces to the correct TempoStack tenants.

The Hitchhiker's Guide to Observability - Example Applications - Part 4

Wed, 26 Nov 2025 00:00:00 +0000

With the architecture defined, TempoStack deployed, and the Central Collector configured, we’re now ready to complete the distributed tracing pipeline. It’s time to deploy real applications and see traces flowing through the entire system!

In this fourth installment, we’ll focus on the application layer - deploying Local OpenTelemetry Collectors in team namespaces and configuring example applications to generate traces. You’ll see how applications automatically get enriched with Kubernetes metadata, how namespace-based routing directs traces to the correct TempoStack tenants, and how the entire two-tier architecture comes together.

We’ll deploy an example application (Mockbin) into two different namespaces. Together with the application, we will deploy a local OpenTelemetry Collector. One with sidecar mode, one with deployment mode.

You’ll learn how to configure local collectors to add namespace attributes, forward traces to the central collector, and verify that traces appear in the UI with full Kubernetes context.

By the end of this article, you’ll have a fully functional distributed tracing system with applications generating traces, local collectors forwarding them, and the central collector routing everything to the appropriate TempoStack tenants. Let’s bring this architecture to life!

Application Mockbin - Step-by-Step Implementation

Prerequisites

Before starting, ensure you have the following Systems or Operators installed (used Operator versions in this article):

OpenShift or Kubernetes cluster (OpenShift v4.20)
Red Hat build of OpenTelemetry installed (v0.135.0-1)
Tempo Operator installed (v0.18.0-2)
S3-compatible storage (for TempoStack, based on OpenShift Data Foundation)
Cluster Observability Operator (v1.3.0) for now this Operator is only used to extend the OpenShift UI with the tracing UI)

Mockbin Application Introduction

The Mockbin Application was created and provided by Michaela Lang. She is doing a lot of testings with Service Mesh and Observability. I forked the source code for the Mockbin application to Mockbin and created an images to test everything at: Mockbin Image. Mockbin is a simple, OpenTelemetry-instrumented HTTP testing service built with Python’s aiohttp async framework. It serves as both a demonstration tool and a practical testing service for distributed tracing infrastructure.

Why Mockbin?

Unlike simple "hello world" examples, Mockbin demonstrates real-world OpenTelemetry implementation patterns that you can apply to your own Python applications. It shows how to properly instrument an async Python web service with full observability capabilities.

Key Features

Full OpenTelemetry Integration: Implements the complete observability stack - traces, metrics, and logs
Automatic Instrumentation: Uses OpenTelemetry’s auto-instrumentation for aiohttp
Manual Span Creation: Demonstrates how to create custom spans with attributes and events
Trace Context Propagation: Supports both W3C Trace Context (traceparent) and Zipkin B3 (x-b3-*) formats
OTLP Export: Sends all telemetry data via OpenTelemetry Protocol (OTLP) to collectors
Prometheus Metrics with Exemplars: Links high-cardinality traces to aggregated metrics
Structured Logging via OTLP: Logs are automatically correlated with traces
Nested Spans: Creates parent-child span relationships for complex operations
Exception Recording: Captures and records exceptions with full stack traces

Technical Architecture

The application consists of several Python modules:

proxy.py: Main application with HTTP endpoint handlers
tracing.py: OpenTelemetry configuration and trace context management
promstats.py: Prometheus metrics collection with trace exemplars
logfilter.py: OTLP logging with automatic trace correlation

HTTP Endpoints for Testing

Endpoint

Purpose

Use Case

Echo service - returns headers and environment variables

Basic trace generation

/logging/*

Generates multiple log entries linked to trace

Demonstrate trace-to-log correlation

/proxy/*

Chains multiple HTTP requests with nested spans

Test distributed trace propagation

/exception/{status}

Intentionally raises exception and returns custom status code

Test error trace capture

/webhook/alert-receiver

Converts AlertManager webhooks to traces

Alert-to-trace correlation

/outlier

Enables/disables failure mode (PUT/DELETE)

Test Service Mesh outlier detection

/metrics

Exposes Prometheus metrics

Metrics scraping

/health

Health check endpoint

Kubernetes liveness/readiness probes

OpenTelemetry Implementation Highlights

The application demonstrates several critical OpenTelemetry patterns:

Middleware-Based Instrumentation: Every HTTP request is automatically wrapped in a trace span via aiohttp middleware, capturing request metadata (method, URL, headers, etc.) as span attributes.
Context Propagation: Incoming requests have their trace context extracted from HTTP headers, and outgoing requests inject the context to maintain trace continuity across service boundaries.
Resource Attributes: Service metadata (name, namespace, version) is attached to all telemetry data for proper identification in the backend.
Manual Span Creation: Shows how to create nested spans for complex operations, add custom attributes, record events, and set span status (OK/ERROR).

Configuration via Environment Variables

The application is configured entirely through environment variables, making it easy to adapt to different environments, for example:

# OpenTelemetry Configuration
OTEL_EXPORTER_OTLP_ENDPOINT=http://otelcol-collector:4317
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=http://otelcol-collector:4317
OTEL_SPAN_SERVICE=mockbin

We will test OpenTelemetry Collector in deployment mode and in sidecar mode. When using sidecar mode, then the sidecar will try to automatically inject the OpenTelemetry Collector into the pod.

Deploy Mockbin Application & OpenTelemetry Collector - team-a

Let’s start with the deployment of the Mockbin Application and the OpenTelemetry Collector in the team-a namespace. Here we will use the OpenTelemetry Collector in deployment mode.

The deployment in sidecar mode will be done in the team-b namespace and is almost the same, just with the difference that we will use a different mode in the OTC resource and that we do not need to take care of the environment variables for the OTC.

Step 1a: Use Kustomize Manually

You can deploy the Mockbin Application and the OpenTelemetry Collector manually using Kustomize or by using a GitOps tool like ArgoCD (preferred). In the repository Mockbin your will find the sub-folder k8s with the Kustomize files for the Mockbin Application and the OpenTelemetry Collector.

You can manually deploy the application:

Step 1a.1: Create Namespace team-a

First create the namespace team-a.

apiVersion: v1
kind: Namespace
metadata:
name: team-a (1)

1	Namespace name

Step 1a.2: Deploy the Mockbin Application

Use the following command to deploy the application (assuming you cloned the repository to your local machine):

kustomize build . | kubectl apply -f -

This will deploy the required resources:

ServiceAccount
Service
Route
Deployment using the image: quay.io/tjungbau/mockbin:1.8.1

It is possible to kustomize the deployment by modifying the kustomization.yaml file. For example, you can change the image, or change from Route to Ingress object.

Step 1b: Use ArgoCD

Above can be achieved by using ArgoCD, which allows a more declarative approach. Create the following ArgoCD Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: mockbin-team-a (1)
namespace: openshift-gitops (2)
spec:
destination:
namespace: team-a (3)
server: 'https://kubernetes.default.svc' (4)
info:
- name: Description
value: Deploy Mockbin in team-a namespace
project: in-cluster
source:
path: k8s/ (5)
repoURL: 'https://github.com/tjungbauer/mockbin' (6)
targetRevision: main (7)
syncPolicy:
syncOptions:
- CreateNamespace=true (8)

1	Name of the Argo CD Application resource.
2	Namespace of the ArgoCD Application resource, here it is openshift-gitops. In your environment it might be different.
3	Namespace of the target application
4	Kubernetes API server URL. Here it is the local cluster where Argo CD is hosted.
5	Path to the Kustomize files
6	URL to the Git repository
7	Target revision
8	Create the namespace if it does not exist

Argo CD will leverage Kustomize to render the resources. It is the same as you would do manually, with the exception that the Namespace will be created automatically. The approach above will deploy the required resources:

Namespace
ServiceAccount
Service
Route
Deployment

In the Argo CD UI you can see the application and the resources that have been deployed.

Step 2: Verify the Deployment

Verify the Pods, Service and Route of the Mockbin Application:

kubectl get pods -n team-a
NAME READY STATUS RESTARTS AGE
pod/mockbin-c4587558b-ftvsd 2/2 Running 0 3d15h
pod/mockbin-c4587558b-zrxgc 2/2 Running 0 3d15h
kubectl get services -n team-a
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/mockbin ClusterIP 172.30.202.223 <none> 8080/TCP 3d16h
kubectl get routes -n team-a
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
route.route.openshift.io/mockbin mockbin-team-a.apps.ocp.aws.ispworld.at mockbin http edge/Redirect None

Access the route URL to see the Mockbin Application:

curl -k https://mockbin-team-a.apps.ocp.aws.ispworld.at

This will return a response looking like this:

{"headers": {"User-Agent": "curl/8.7.1", "Accept": "*/*", "Host": "mockbin-team-a.apps.ocp.aws.ispworld.at", .... lot's of header stuff"}

Step 3: Deploy the Local OpenTelemetry Collector - Mode Deployment

All steps below can be applied using GitOps again, using the Chart rh-build-of-opentelemetry. An example of a GitOps configuration (for the Central Collector) can be found at Setup OTEL Operator.

Create a ServiceAccount for the OpenTelemetry Collector

Let us first create a ServiceAccount for the OpenTelemetry Collector.

apiVersion: v1
kind: ServiceAccount
metadata:
name: otelcol-agent
namespace: team-a

Create the OpenTelemetry Collector Resource

Create the following OpenTelemetry Collector Resource. The mode shall be "deployment" The serviceAccount shall be the one we created in the previous step, and the namespace is "team-a".

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
name: otelcol-agent (1)
namespace: team-a (2)
spec:
mode: deployment (3)
replicas: 1 (4)
serviceAccount: otelcol-agent (5)
config:
# Receivers - accept traces from applications
receivers: (6)
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
# Processors
processors: (7)
# Add namespace identifier
attributes:
actions:
- action: insert
key: k8s.namespace.name
value: team-a
# Batch for efficiency
batch: {}
# Exporters - forward to central collector
exporters:
otlp/central: (8)
endpoint: otel-collector.tempostack.svc.cluster.local:4317
tls:
insecure: true
# Service pipeline
service:
pipelines: (9)
traces:
receivers:
- otlp
processors:
- batch
- attributes
exporters:
- otlp/central

1	Name of the OpenTelemetry Collector Resource.
2	Namespace team-aof the OpenTelemetry Collector Resource.
3	Mode deployment of the OpenTelemetry Collector Resource.
4	Number of replicas of the OpenTelemetry Collector Resource. For HA setup increase the number of replicas.
5	ServiceAccount of the OpenTelemetry Collector Resource, that was created in the previous step.
6	Receivers of the OpenTelemetry Collector Resource. We will receive traces using the OTLP protocol on gRPC and HTTP.
7	Processors of the OpenTelemetry Collector Resource. We will add the namespace identifier as an attribute and batch the traces for efficiency.
8	Exporters of the OpenTelemetry Collector Resource. We will forward the traces to the Central Collector.
9	Service pipeline of the OpenTelemetry Collector Resource.

Since we choose the deployment mode, the OpenTelemetry Collector will be deployed as a Deployment resource.

oc deployment all -n team-a
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/otelcol-agent-collector 1/1 1 1 3d2h

Step 4: Cool, and now what? - Environment Variables

If you came that far, then you have deployment TempoStack, the Central Collector, the Mockbin Application and the Local Collector. However, you will not receive any traces yet. We need to tell the Mockbin Application WHERE to send the traces.

To do this, we need to set the environment variables for the Mockbin Deployment resource. Edit the Deployment resource and add the following environment variables:

 env:
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: 'http://otelcol-agent-collector:4317' (1)
- name: OTEL_EXPORTER_OTLP_PROTOCOL
value: grpc (2)
- name: OTEL_SERVICE_NAME
value: mockbin (3)

1	The endpoint (service) of the Local Collector.
2	The protocol we would like to use.
3	The service name of the Mockbin Application.

This will trigger a restart of the Deployment. Once done, the environment variables should show up:

curl -k https://mockbin-team-a.apps.ocp.aws.ispworld.at
{"headers": {"...., "OTEL_EXPORTER_OTLP_ENDPOINT": "http://otelcol-agent-collector:4317", "OTEL_EXPORTER_OTLP_PROTOCOL": "grpc", ...."}}

This will now start sending traces to the Local Collector and from there to the Central Collector.

Traces, Traces, Traces

Now we have a working setup. We have a Central Collector, a Local Collector and a Mockbin Application. Our application is permanently sending requests to the Local Collector.

We can review the traces in the OpenShift UI: Observe > Traces.

Select the tempostack instance and select the tenant tenantA. If everything is working, you should see traces from the Mockbin Application.

If no traces show up, then most probably the RBAC rules are not configured correctly or the environment variables are not set correctly. You can check the logs of the OTC pods for errors.

Deploy Mockbin Application & OpenTelemetry Collector - team-b

Let us now deploy the Mockbin Application and the OpenTelemetry Collector in the team-b namespace. Here we will use the OpenTelemetry Collector in sidecar mode.

Step 1: Deploy the Application using Argo CD

Step 1b: Use ArgoCD

Above can be achieved by using ArgoCD, which allows a more declarative approach. Create the following ArgoCD Application (This is the same approach as above)

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: mockbin-team-b (1)
namespace: openshift-gitops (2)
spec:
destination:
namespace: team-b (3)
server: 'https://kubernetes.default.svc' (4)
info:
- name: Description
value: Deploy Mockbin in team-b namespace
project: in-cluster
source:
path: k8s/ (5)
repoURL: 'https://github.com/tjungbauer/mockbin' (6)
targetRevision: main (7)
syncPolicy:
syncOptions:
- CreateNamespace=true (8)

1	Name of the Argo CD Application resource.
2	Namespace of the ArgoCD Application resource, here it is openshift-gitops. In your environment it might be different.
3	Namespace of the target application
4	Kubernetes API server URL. Here it is the local cluster where Argo CD is hosted.
5	Path to the Kustomize files
6	URL to the Git repository
7	Target revision
8	Create the namespace if it does not exist

Step 2: Deploy the Local OpenTelemetry Collector - Sidecar Deployment

All steps below can be applied using GitOps again, using the Chart rh-build-of-opentelemetry. An example of a GitOps configuration (for the Central Collector) can be found at Setup OTEL Operator.

Create a ServiceAccount for the OpenTelemetry Collector

Let us first create a ServiceAccount for the OpenTelemetry Collector.

apiVersion: v1
kind: ServiceAccount
metadata:
name: otelcol-agent
namespace: team-b

Create the OpenTelemetry Collector Resource

Create the following OpenTelemetry Collector Resource. The mode shall be "sidecar" The serviceAccount shall be the one we created in the previous step, and the namespace is "team-b".

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
name: otelcol-agent (1)
namespace: team-b (2)
spec:
mode: sidecar (3)
replicas: 1 (4)
serviceAccount: otelcol-agent (5)
config:
# Receivers - accept traces from applications
receivers: (6)
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
# Processors
processors: (7)
# Add namespace identifier
attributes:
actions:
- action: insert
key: k8s.namespace.name
value: team-b
# Batch for efficiency
batch: {}
# Exporters - forward to central collector
exporters:
otlp/central: (8)
endpoint: otel-collector.tempostack.svc.cluster.local:4317
tls:
insecure: true
# Service pipeline
service:
pipelines: (9)
traces:
receivers:
- otlp
processors:
- batch
- attributes
exporters:
- otlp/central

1	Name of the OpenTelemetry Collector Resource.
2	Namespace team-aof the OpenTelemetry Collector Resource.
3	Mode sidecar of the OpenTelemetry Collector Resource.
4	Number of replicas of the OpenTelemetry Collector Resource. For HA setup increase the number of replicas.
5	ServiceAccount of the OpenTelemetry Collector Resource, that was created in the previous step.
6	Receivers of the OpenTelemetry Collector Resource. We will receive traces using the OTLP protocol on gRPC and HTTP.
7	Processors of the OpenTelemetry Collector Resource. We will add the namespace identifier as an attribute and batch the traces for efficiency.
8	Exporters of the OpenTelemetry Collector Resource. We will forward the traces to the Central Collector.
9	Service pipeline of the OpenTelemetry Collector Resource.

Step 3: Enable Sidecar Injection for Team-B

The big advantage of the sidecar mode is that the OpenTelemetry Collector is deployed as a sidecar container in the same pod as the application. This happens fully automatically, means you do not need to set any extra environment variables or anything like that.

To enable the sidecar injection, we need to add the following annotation to the namespace. This will automatically inject the OpenTelemetry Collector as a sidecar container into the ALL pods of the namespace:

Do not forget this to add it into GitOps process :)

First verify the current state of the deployment:

oc deployment all -n team-b
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/otelcol-agent-collector 1/1 1 1 3d2h

As you can see one container out of 1 is running (Read 1/1)

Now lets modify the namespace to enable the sidecar injection:

apiVersion: v1
kind: Namespace
metadata:
name: team-b
annotations:
sidecar.opentelemetry.io/inject: "true"

This can also be done by annotate the Deployment resource. In this case, the OpenTelemetry Collector will be injected into the specific pod.

Restart the deployment to apply the changes:

oc rollout restart deployment/mockbin -n team-b
deployment.apps/mockbin restarted

Now, when you check again, the sidecar container should be injected into the pods. The Deployment has now 2/2 containers ready:

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/mockbin 2/2 1 2 3m53s

Traces, Traces, Traces - Again

Again, we can review the traces in the OpenShift UI: Observe > Traces.

Select the tempostack instance and select the tenant tenantB this time.

Argo CD Stays in Progressing State

If you have deployed the Mockbin Application and the OpenTelemetry Collector as a sidecar deployment in the team-b namespace, you might notice that the Argo CD Application stays in the Progressing state.

This is because the OpenTelemetry Collector does not really return a state. The status is simply:

status:
version: 0.140.0

Nothing else … thats too less for Argo CD to work with.

In order to fix this, we need to add a custom health check to the Argo CD instance.

The Argo CD instance knows the following part:

apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
name: openshift-gitops
namespace: openshift-gitops
spec:
[...]
resourceHealthChecks: (1)
- check: |
hs = {}
if obj.status ~= nil then
if obj.status.version ~= nil and obj.status.version ~= "" then
hs.status = "Healthy"
hs.message = "Running version " .. obj.status.version
else
hs.status = "Progressing"
hs.message = "Waiting for version report"
end
else
hs.status = "Progressing"
hs.message = "Status not available"
end
return hs
group: opentelemetry.io
kind: OpenTelemetryCollector
[...]

1	Custom health check for the OpenTelemetry Collector.

Above will end up in a ConfigMap that is managed by the OpenShift GitOps operator.

This will make Argo CD happy and the Application will be in the Progressing state.

1	The namespace where the OpenBao deployment is running.
2	The self-signed CA Issuer.

YAUB Yet Another Useless Blog on TechBlog about OpenShift/Ansible/Satellite and much more

Onboarding to Ansible Automation Platform with Configuration as Code

OKD Single node installation in a disconnected environment

OpenShift Virtualization Networking - The Overview

Who does what: NMState vs. Multus

NMState Operator Installation

NMState Operator Resource Types

Part 1: Building the Bridge (NMState)

Part 2: Defining the NetworkAttachmentDefinition (NAD)

1. The Linux Bridge (CNI-Plugin)

A note on IPAM (IP Address Management)

2. OVN Kubernetes L2 Overlay

3. OVN Kubernetes Secondary Localnet

Part 3: Connecting the VM

GitOps Catalog

The Guide to OpenBao - Authentication Methods - Part 7

Introduction

Authentication Workflow

Prerequisites for Authentication Methods

Using the CLI

Using the UI

Understanding Policies

Basic Policy Structure

Policy Capabilities

Default Policies

Creating Policies using the CLI

Kubernetes Authentication Method

Step 1: Enable Kubernetes Auth

Step 2: Configure Kubernetes Auth

Step 3: Create a Policy for the Application

Step 4: Create a Kubernetes Auth Role

Ensure the KV v2 secrets engine is mounted at secret/

Step 5: Store the database secret in OpenBao (one-time)

Step 6: Create the expense namespace and demo application

Step 7: Test Kubernetes authentication manually

Summary: Kubernetes auth method

LDAP Authentication Method

Step 1: Enable LDAP Auth

Step 2: Configure LDAP (Active Directory Example)

Step 3: Create the policies used by the group mapping

Step 4: Map LDAP groups to policies

Step 5: Test LDAP authentication

What jdoe can and cannot do (user-policy)

Common Practices for Authentication

Common Authentication Patterns

What is Coming Next?

Conclusion

Resources

The Guide to OpenBao - Initialisation, Unsealing, and Auto-Unseal - Part 6

What is happening?

Handling Initialisation and Unsealing

Approach 1: Manual Initialisation (simplest)

Approach 2: Kubernetes Job for initialisation

How the Secret is created

RBAC Configuration

Init Job that creates the Secret

Using the Secret in a Job (unseal)

Approach 3: Auto-unseal with Key Management Service KMS (recommended for production)

Auto-unseal options overview

Why use OpenBao when the seal is in KMS? Why not keep all secrets in KMS and skip OpenBao?

Option: AWS KMS

Create a KMS key

Key policy for OpenBao

Create an IAM user for the seal

Create the Secret

Helm / Argo CD configuration (AWS)

Approach 4: Init container with external secret store

Option: init Container example

Option: Transit seal

Dedicated unseal service (seal-only OpenBao)

Migration from Shamir to auto-unseal

Resources

The Guide to OpenBao - GitOps Deployment with Argo CD - Part 5

Introduction

Prerequisites

The (Wrapper) Helm Chart

Helm Chart for the CA Certificate

Helm Chart for the OpenBao Deployment

Creating Argo CD Applications

OpenBao is running—what next?

Ensure the KV v2 secrets engine is mounted at `secret/`

But wait there is more…