YAUB Yet Another Useless Blog on TechBlog about OpenShift/Ansible/Satellite and much more

Creating an auditd rule to monitor Ansible Automation Platform static secrets

Tue, 09 Jun 2026 00:00:00 +0000

Ansible Automation Platform (AAP) comes with an elaborate credential management solution. All credentials within AAP are stored encrypted in a PostgreSQL database. But the keys used for encrypting database fields are currently stored on the machine(s) running AAP. In this blog post, we show we created an auditd rule to get at least some kind of monitoring if those keys are accessed.

Ansible Automation Platform local SECRET_KEY

AAP comes with a credential management solution. Credentials are used to

access remote machines
provide configuration data for automation, like API keys
connect AAP to source code repositories and inventories

and much more. You can even integrate external credential management solutions like HashiCorp Vault or OpenBao.

All credentials stored within AAP are saved as encrypted database fields in a PostgreSQL database. There is a main key and AAP creates for every encrypted database field a derived AES-256 encryption key. For more details see the documentation.

But there is one caveat: the main secret key is stored on the local file system. There is currently no supported option to integrate something like a HSM (High Security Module) for storing the main secret securely.

To mitigate this issue we deployed a Linux audit rule that monitors read/write and attribute changes to the main secret.

Configuring Linux auditd

We use the containerized version of AAP 2.6. In the containerized version of AAP those secrets are stored as podman secrets as the user running the AAP containers.

To list AAP podman secrets execute

$ podman secret list
ID NAME DRIVER CREATED UPDATED
02a2eb397dc5cedefde6ae411 hub_resource_server file 2 hours ago 2 hours ago
0dd84dc7b808917c4efb6ea9f controller_channels file 2 hours ago 2 hours ago
85fb73ff57729250c019976ed hub_database_fields file 2 hours ago 2 hours ago
c482ff7c07f908e443d327293 controller_postgres file 2 hours ago 2 hours ago
e6750121a8997cce8185072bf gateway_admin_password file 2 hours ago 2 hours ago
2731b2d03806f88f35809de66 eda_admin_password file 2 hours ago 2 hours ago
31ca44577b4c5b61dd3f0a0b6 eda_secret_key file 2 hours ago 2 hours ago
856473626bad9ff37b1b979d4 eda_resource_server file 2 hours ago 2 hours ago
b20ec0897fc2b6533544c4441 gateway_redis_url file 2 hours ago 2 hours ago
1dfdd812847ed4c1c01b194e2 gateway_db_password file 2 hours ago 2 hours ago
32d04575881dd781be192ef49 eda_db_password file 2 hours ago 2 hours ago
5102f69350717c15177b80821 hub_secret_key file 2 hours ago 2 hours ago
d1faf92922182604cfa746318 controller_resource_server file 2 hours ago 2 hours ago
d29f0d8c2fc709655dd918f2c postgresql_admin_password file 2 hours ago 2 hours ago
1b3ee24261bdff50d2c70e3f0 gateway_secret_key file 2 hours ago 2 hours ago
ae6c9e78e91dea180d22db6ee hub_settings file 2 hours ago 2 hours ago
b348d89bfb7d3b18f33a67d07 controller_secret_key file 2 hours ago 2 hours ago (1)

1	the controller secret key

If we want to know where those secrets are stored we can execute

$ podman secret inspect controller_secret_key
[
{
"ID": "b348d89bfb7d3b18f33a67d07",
"CreatedAt": "2026-06-09T10:57:28.278138715Z",
"UpdatedAt": "2026-06-09T10:57:28.278138715Z",
"Spec": {
"Name": "controller_secret_key",
"Driver": {
"Name": "file",
"Options": {
"path": "/home/admin/.local/share/containers/storage/secrets/filedriver"
}
},
"Labels": {}
}
}
]

Let’s see what’s in the filedriver directory:

$ ls /home/admin/.local/share/containers/storage/secrets/filedriver
secretsdata.json secretsdata.lock(1)

1	we are mainly interested in the secretsdata.json file

So the secretsdata.json file stores all podman secrets that are using the filedriver. For more information see the section Secret Drivers in the podman-secrets-create(1) manpage.

Let’s create a Linux auditd rule to monitor access to this file. As we are using Red Hat Enterprise Linux 10 a more detailed introduction to the Linux audit system is available here.

We created the file /etc/audit/rules.d/aap_secrets.rules with the following content:

-w /home/admin/.local/share/containers/storage/secrets/filedriver/secretsdata.json -p rwa -k aap_secrets (1)

1	-p rwa means monitor read/write/attribute changes, -k tells the audit system log events with an additional key "aap_secrets"

To activate the rule we need to load the audit rule via:

# auditctl -R /etc/audit/rules.d/aap_secrets.rules

You can list loaded audit rules via

# auditctl -l
-w /home/admin/.local/share/containers/storage/secrets/filedriver/secretsdata.json -p rwa -k aap_secrets

As a test we can cat the secrets file and check if auditd logged an event:

# cat /home/admin/.local/share/containers/storage/secrets/filedriver/secretsdata.json
...
# grep aap_secrets /var/log/audit/audit.log
type=SYSCALL msg=audit(1781009319.993:204): arch=c000003e syscall=257 success=yes exit=3 a0=ffffff9c a1=7ffc5276b65c a2=0 a3=0 items=1 ppid=6186 pid=6695 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts2 ses=3 comm="cat" exe="/usr/bin/cat" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="aap_secrets"ARCH=x86_64 SYSCALL=openat AUID="admin" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root" (1)

1	We can see that the cat command (comm=cat) was used by the root user to display the file contents

Summary

In this blog post we demonstrated how to configure the Linux audit system to log events if the AAP database secret key is accessed. The audit subsystem can also monitor system calls, for a detailed introduction see Audit system architecture.

Using a local Renovate bot to manage Ansible collection dependencies

Tue, 26 May 2026 00:00:00 +0000

We wanted to use renovate bot to automatically update collection dependencies for one of our Ansible playbook repositories. This blog post describes the basics of renovate, how we configured it and how we actually run renovate locally.

Understanding renovate

As this was our first try of using renovate for dependency updates, we struggled to understand how renovate works and how it is configured.

Renovate is a bot that is able to scan multiple repositories for dependencies and create pull/merge requests with updated dependencies.

Configuration is split between a configuration for the bot and a local one for each repository.

So you basically have

One or more global configuration files that configure the bot (config.js) written in JavaScript or TypeScript
One configuration file per repository (renovate.json) written in JSON that contains local configuration options just for this repository.

Renovate is very flexible on how to structure the configuration. For more information see config-overview in the docs.

Renovate also supports multiple platforms, like GitHub, GitLab or Forgejo. For a complete list see renovate platforms.

The idea is that you run the bot once, it scans all configured repositories and creates pull requests in every repository with dependency updates.

If your projects are hosted on GitHub and you would like to use GitHub Actions for running renovate the straight forward solution is to follow Mend Renovate on GitHub Marketplace.

But we wanted more control and tried to

run renovate on a local machine. This should be migrated to a Job Template or Kubernetes CronJob later (might be worth another post)
use a private Automation Hub for hosting dependencies. So renovate should query collections on our Private Automation Hub for dependency updates
credentials should be made available via environment variables to the bot

Configuring and running renovate

As a first test we created a local bot configuration file config.js in our home directory. The GIT repository we used for testing is https://github.com/tosmi-ansible/aap-setup.

The first thing to be aware of is that the renovate bot by default always searches the repository configuration file (renovate.json) in the upstream repository. Changes to local files are ignored.

One way to avoid this is using --platform=local. This actually uses the local renovate.json file, but has the caveat that not all renovate functionality is available.

But in our case this was good enough as we just wanted to test AAP Hub authentication.

It is also helpful to run renovate with debugging enabled, this dumps the configuration currently used to STDOUT.

For example we had the following renovate.json config for testing:

{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": [
"config:best-practices"
],
"timezone": "Europe/London",
"schedule": [
"at any time"
],
"packageRules": [
{
"matchDatasources": ["galaxy-collection"]
},
{
"matchDatasources": ["github-tags"],
"enabled": "false"
}
],
"hostRules": [
{
"matchHost": "aap.apps.hub.aws.tntinfra.net",
"token": "Token <the token>",
"authType": "Token-Only"
}
]
}

But running renovate with LOG_LEVEL=debug set in the environment revealed the following settings:

DEBUG: Repository config (repository=tosmi-ansible/aap-setup)
"fileName": "renovate.json",
"config": {
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": ["config:recommended"]
}

The configuration above is the configuration available in the remote GIT repository and not our local configuration.

Renovate does not use a local renovate.json repository config by default. It always pulls the configuration from the remote repository except when you use _—platform=local.

Furthermore renovate heavily relies on local caching. We wanted to test if renovate is using our hub token for connecting to the Hub, but we could not see any log entry in the AAP Hub container.

Renovate creates its cache files in a directory called baseDir. You can specify the location (and later wipe the cache) with the --base-dir option.

As we are using a self signed certificate in our AAP Hub test installation we had to specify NODE_TLS_REJECT_UNAUTHORIZED=0 to disable TLS verification.

To summarize our test configuration:

We used the following config.js global renovate configuration file:

module.exports = {
token: '<github token>',
platform: 'github',
onboardingConfig: {
extends: ['config:recommended'],
},
//repositories: ['tosmi-ansible/aap-setup'], (1)
};

1	Repository listing is not allowed when using --platform=local

In the repository where we want to test automatic dependency updates we used the following renovate.json configuration:

{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": [
"config:best-practices"
],
"timezone": "Europe/London",
"schedule": [
"at any time"
],
"packageRules": [
{
"matchDatasources": ["galaxy-collection"]
},
{
"matchDatasources": ["github-tags"],
"enabled": "false" (1)
}
],
"hostRules": [
{
"matchHost": "aap.apps.hub.aws.tntinfra.net",
"token": "Token <the token>",
"authType": "Token-Only"
}
]
}

1	We disabled GitHub tag lookup explicitly as renovate would find some of our dependencies on GitHub

This is our collections/requirements.yaml file with a test dependency:

roles:
collections:
- name: kubernetes.core
version: 6.3.0
source: https://aap.apps.hub.aws.tntinfra.net/api/galaxy/content/rh-certified/

On our AAP Private Automation Hub we had kubernetes.core version 6.4.0 available.

So to run and test renovate locally we used the following command line:

NODE_TLS_REJECT_UNAUTHORIZED=0 LOG_LEVEL=debug RENOVATE_CONFIG_FILE=~/config.js renovate --platform=github --repository-cache=disabled --base-dir=/tmp/cache

Secret management and configuration cleanup

The next step is to remove the hard-coded tokens from our configuration. We wanted to use environment variables, as this allows us to run renovate as:

A Kubernetes CronJob with a Pod. We could inject the token via a Secret
As a job template within AAP with tokens stored as AAP credentials or
A normal Unix cron job with environment variables set

As the global renovate configuration config.js is written in JavaScript we could leverage process.env to retrieve required tokens from environment variables.

Here is our modified config.js:

module.exports = {
platform: 'github',
onboardingConfig: {
extends: ['config:recommended'],
},
// repositories: ['tosmi-ansible/aap-setup'], (1)
secrets: {
AAP_HUB_TOKEN: process.env.AAP_HUB_TOKEN, (2)
},
hostRules: [
{
matchHost: "aap.apps.hub.aws.tntinfra.net",
token: "{{ secrets.AAP_HUB_TOKEN }}", (3)
authType: "Token" (4)
},
],
};

1	We disable repository listing for --platform=local. Otherwise renovate will raise an error.
2	We define a new secret AAP_HUB_TOKEN and read the value from the environment variable AAP_HUB_TOKEN
3	We use the secret as our token for authenticating to AAP Hub
4	The authType setting is a little bit special, see the following section

For GitHub authentication renovate expects a GitHub token in the environment variable RENOVATE_TOKEN.

A few words about the authType setting. AAP Hub requires the following authorization header in the HTTP request for authentication:

Authorization: Token <the token>

With renovate we have two options to achieve this:

Either set authType to "Token-Only" and specify the token as "Token <the token>". So with token-only renovate takes the token string verbatim
Or set the authType to the string we would like to prepend to the token as we did

We had to dig into the source code of renovate to understand how it functions.

And here is the renovate.json repository configuration that we used:

{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": [
"config:best-practices"
],
"timezone": "Europe/London",
"schedule": [
"at any time"
],
"packageRules": [
{
"matchDatasources": ["galaxy-collection"]
},
{
"matchDatasources": ["github-tags"],
"enabled": "false"
}
]
}

You could also move the hostRules to the repository local renovate.json file and still use secrets.AAP_HUB_TOKEN, but as this is going to be our global configuration for Ansible code base covering multiple repositories our current idea is to store the authentication in the global configuration.

To set the required environment variables containing our secrets we execute

$ export AAP_HUB_TOKEN=$(pass show other/ansible/aws-aap-tokens | awk '/hub/ {print $2}') (1)
$ export RENOVATE_TOKEN=$(pass show other/github/tosmi-rw-token | awk '/token/ {print $2}')

1	We use the pass utility for secret management.

Integration with GitHub

So the last step is to store everything (the global and local configuration) on GitHub.

For the global repository we created the tosmi-ansible/renovate-config repository.

Here is our final config.js global configuration file in this repository:

module.exports = {
platform: 'github',
gitAuthor: "Renovate Bot <renovate-bot@stderr.at>",
dryRun: null,
onboardingConfig: {
extends: ['config:recommended'],
},
repositories: ['tosmi-ansible/aap-setup'], (1)
secrets: {
AAP_HUB_TOKEN: process.env.AAP_HUB_TOKEN,
},
hostRules: [
{
matchHost: "aap.apps.hub.aws.tntinfra.net",
token: "{{ secrets.AAP_HUB_TOKEN }}",
authType: "Token"
},
],
};

1	This time we tell renovate which repositories it should scan.

The default configuration default.json, also stored in the global repository:

{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": [
"config:best-practices"
],
"timezone": "Europe/London",
"schedule": [
"at any time"
],
"enabledManagers": ["ansible", "ansible-galaxy"], (1)
"packageRules": [
{
"matchDatasources": ["galaxy-collection"]
},
{
"matchDatasources": ["github-tags"],
"enabled": "false"
}
]
}

1	We only enable ansible and ansible-galaxy managers. Otherwise renovate would search for various other sources for dependencies.

tosmi-ansible/aap-setup contains our local configuration:

{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": [
"tosmi-ansible/renovate-config",
"config:best-practices"
]
}

This only contains a reference to our global configuration and we would like to use the best-practices preset.

We can now run renovate from the renovate-config repository and it will scan the configured repository and create a pull request with updated dependencies if required.

$ git clone git@github.com:tosmi-ansible/renovate-config.git
$ cd renovate-config
$ NODE_TLS_REJECT_UNAUTHORIZED=0 renovate --platform=github --base-dir=/tmp/renovate-cache (1)

1	We still need to set NODE_TLS_REJECT_UNAUTHORIZED to 0, because our AAP Hub instance uses a private certificate

One minor issue that we discovered is that if a pull request for an update already existed (even a closed one) Renovate would not open a new one. If you run renovate with LOG_LEVEL=debug you will see the following message:

DEBUG: Closed PR #19 already exists. Skipping branch. (repository=tosmi-ansible/aap-setup, branch=renovate/kubernetes.core-6.x)
"prTitle": "Update dependency kubernetes.core to v6.4.0"

To convince Renovate to create a new pull request just rename the closed pull request on GitHub. We just added the prefix "RENAMED". After renaming the pull request, renovate will happily create a new one.

Tips and tricks

This section list various things we learning while running renovate locally

Renovate does not honor new dependencies in requirements.yaml

We had an issues with new dependencies in requirements.yaml. Wiping the renovate cache (/tmp/renovate_cache) in our case fixed this and renovate created new pull requests for the added dependencies.

Renovate does not open new PR (pull requests) for updated dependencies

We double checked AAP and our renovate configuration and there was clearly a newer version of a collection but renovate just did not open a new PR.

Running renovate with LOG_LEVEL=debug revealed the following message (we tried to update the redhat.openshift_virtualization collection):

 {
"branchName": "renovate/redhat.openshift_virtualization-2.x",
"prNo": null,
"prTitle": "Update dependency redhat.openshift_virtualization to v2.2.4",
"result": "branch-limit-reached", (1)

1	renovate does not open a new branch because of "branch-limit-reached"

We check our upstream repository on GitHub and there was just the main branch, hm…

Looking further up in the log we found:

DEBUG: prHourlyLimit of the upgrades present in this branch (repository=tosmi-ansible/aap-setup, branch=renovate/infra.aap_utilities-3.x)
"limits": [{"depName": "infra.aap_utilities", "prHourlyLimit": 2}]
DEBUG: Calculated lowest prHourlyLimit among the upgrades present in this branch is 2. (repository=tosmi-ansible/aap-setup, branch=renovate/infra.aap_utilities-3.x)
DEBUG: Hourly PRs limit reached (repository=tosmi-ansible/aap-setup, branch=renovate/infra.aap_utilities-3.x) (1)
"hourlyPrCount": 2,
"hourlyPrLimit": 2
DEBUG: Reached branch limit - skipping branch creation (repository=tosmi-ansible/aap-setup, branch=renovate/infra.aap_utilities-3.x)

1	There is a hourly branch limit of two in place

As we just ran renovate to update another dependency, we seem to be hitting this hourly limit. The docs mention a default of 2.

The solution was to run renovate with the option --pr-hourly-limit=0:

NODE_TLS_REJECT_UNAUTHORIZED=0 renovate --platform=github --base-dir=/tmp/renovate-cache --pr-hourly-limit=0

Creating a customized RHEL 10 VM image with image-builder

Fri, 10 Apr 2026 00:00:00 +0000

In the blog post Creating a RHEL 10 VM on macOS with bootc-image-builder we described how to quickly create a bootc (image mode) base image on macOS. Now we would like to create a customized standard RHEL image.

For this exercise we re-use an already provisioned RHEL 10.1 machine, the reasons for this are

We are going to use the image-builder command instead of bootc image builder. If there is a container image providing this command, please let us know
We are going to install packages which requires a subscribed RHEL machine. A RHEL subscription for up to 16 machines is free of charge. You just need to register at the Red Hat developer site.

First we need to install required tools onto our RHEL 10.1 host:

$ sudo dnf install -y image-builder

Next we create a blueprint file to customize the resulting qcow2 image. We use the resulting image for a "services" machine, running Unifi OS, Kea for DHCP and ISC bind (dns server):

name = "services-rhel10"
description = "A VM running services"
version = "0.1"
[[packages]]
name = "podman"
version = "*" (1)
[customizations]
hostname = "services"
[[customizations.filesystem]]
mountpoint = "/"
minsize = "50 GiB" (2)
[customizations.kernel]
append = "ip=10.0.0.144::10.0.0.1:255.255.255.0:services:ens3:none:10.0.0.255" (3)
[[customizations.group]]
name = "pinhead"
gid = 1000
[[customizations.user]]
name = "pinhead"
password = "<password hash>" (4)
key = "<ssh public key>" (5)
home = "/home/pinhead/"
shell = "/usr/bin/bash"
groups = ["users", "wheel"]
uid = 1000
gid = 1000
[[customizations.sshkey]]
user = "root"
key = "<public key>" (6)

1	use the latest version of this package
2	resize the resulting image to 50GB
3	we configure a static ip address for this machine, our default interface is ens3
4	Enter the generated password hash here (see below)
5	SSH public key to access this account
6	SSH public key to access the root account

To create a password hash use:

python3 -c 'import crypt,getpass;pw=getpass.getpass();print(crypt.crypt(pw) if (pw==getpass.getpass("Confirm: ")) else exit())'

For a detailed list of all customizations supported by image-builder see the image builder customizations documentation.

As mentioned in the previous article image builder provides the option to configure the resulting image via kickstart. A builder for kickstart files is available here:

https://access.redhat.com/labs/kickstartconfig/

For a complete list of options see the kickstart documentation.

One important note from the documentation (quoted):

The following combined customizations are not supported: [customizations.user] and [customizations.installer.kickstart]. When you add a Kickstart, use a configuration file in the TOML format, because multi-line strings are prone to error.

Now we are ready to run image-builder:

image-builder build qcow2 --distro rhel-10.1 --arch x86_64 --blueprint services-blueprint.toml

The command above failed for us with the following error message:

Failed to open file "/sys/fs/selinux/checkreqprot": Read-only file system
imported gpg key
Signature check failed on sha256:9d0a71d87912a815f837f8427438936d6e9842834cc1f1062b90b2a41fbde594, lookup package name in manifest.
Traceback (most recent call last):
File "/run/osbuild/bin/org.osbuild.rpm", line 260, in <module>
r = main(args["tree"], args["inputs"], args["options"])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/run/osbuild/bin/org.osbuild.rpm", line 162, in main
subprocess.run([
File "/usr/lib64/python3.12/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['rpmkeys', '--root', '/run/osbuild/tree', '--checksig', 'sha256:9d0a71d87912a815f837f8427438936d6e9842834cc1f1062b90b2a41fbde594']' returned non-zero exit status 1.
Finished module org.osbuild.rpm
Finished pipeline build
manifest - failed
Output:
Failed

According to this knowledge base article this is a bug in the current (2026-04-10) version of image builder.

You can download an updated JSON file containing the missing RPM GPG keys from the article. Drop the RHEL-10.1 JSON file in a directory and run image-builder again:

image-builder --data-dir override/ <1> build qcow2 --distro rhel-10.1 --arch x86_64 --blueprint services-blueprint.toml

1	Use the --data-dir option and specify the folder with the RHEL-10.1.json file.

You can find the resulting qcow image in the output directory under rhel-10.1-qcow2-x86_64/rhel-10.1-qcow2-x86_64.qcow2.

A Makefile to streamline image creation can be found here.

Creating a RHEL 10 VM on macOS with bootc-image-builder

Thu, 09 Apr 2026 00:00:00 +0000

Yes, we have Apple machines in our lab because why not. So we needed a RHEL 10 VM to set up Ansible Automation Platform, which seems to support AARCH64 and Red Hat Enterprise Linux 10.

We need UTM installed on our Mac machine, either manually, via Homebrew or using a Nix flake (in order of increasing coolness).

Podman is also required, same rules as above apply:

We followed the RHEL documentation for creating a bootable qcow image from a bootc container image.

According to the upstream image builder docs, we need to make sure that our podman machine runs rootful. Otherwise image builder will not work. So let’s do this:

$ podman machine stop
$ podman machine set --rootful
$ podman machine start
$ podman machine info

Next we need to pull the bootc-image-builder image:

$ podman login registry.redhat.io (1)
$ podman pull registry.redhat.io/rhel10/bootc-image-builder

1	This requires a valid Red Hat account. Registration is free of charge.

Then we can pull the RHEL 10 bootc container, as bootc-image-builder is not able to pull container images:

podman pull registry.redhat.io/rhel10/rhel-bootc:latest

Image builder provides the option to configure the resulting image via kickstart. A builder for kickstart files is available here:

https://access.redhat.com/labs/kickstartconfig/

For a complete list of options see the kickstart documentation.

One important note from the documentation (quoted):

For running the image builder we created a toml config file to configure the final qcow image:

[[customizations.user]]
name = "pinhead"
password = "thepassword"
key = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIYhjnWzsArZVyyTa1E6sDbH06rUGDAhAF3bf3pmeBtm toni@stderr.at"
groups = ["wheel"]
[[customizations.filesystem]]
mountpoint = "/"
minsize = "50 GiB"

Now we are ready to trigger bootc-image-builder:

podman run \
--rm \
--privileged \
--pull=newer \
--security-opt label=type:unconfined_t \
-v /var/lib/containers/storage:/var/lib/containers/storage \ (1)
-v ./config.toml:/config.toml:ro \
-v ./output:/output \
registry.redhat.io/rhel10/bootc-image-builder:latest \
--type qcow2 \
--config /config.toml \
registry.redhat.io/rhel10/rhel-bootc:latest

1	We had to map this directory into the container, maybe this is required because we run podman in a VM on macOS (podman machine).

You can find the resulting qcow image in the output directory under output/qcow2/disk.qcow2. This image can be used to create a RHEL 10 VM in UTM on macOS.

It is also possible to customize the container image which is used as an input for bootc-image-builder. But this requires a valid RHEL subscription inside the container. The easiest way to achieve this is by running bootc-image-builder on an already registered RHEL machine.

A Makefile to streamline image creation can be found here.

Onboarding to Ansible Automation Platform with Configuration as Code

Wed, 04 Mar 2026 00:00:00 +0000

We had the honor of presenting at the 2nd Ansible Anwendertreffen in Austria. The topic was the onboarding of application teams to the Ansible Automation Platform.

We created an extensive demo that:

Onboards a new tenant into an AAP organization.
Each tenant gets a configuration as code repository, cloned from a template.
A push event to the config-as-code repository triggers an update of AAP objects.
Provides an example repository for each tenant to get started.
The example repository provides a webhook that creates an OpenShift Virtualization VM for testing code changes when a feature branch is created.
Contains a playbook to display objects that are not currently managed by the tenant’s configuration-as-code repository.

The source code for the demo is available on GitHub.

The README contains more details about the implementation, and the slide deck is also available on GitHub.

OKD Single node installation in a disconnected environment

Mon, 09 Feb 2026 00:00:00 +0000

Because of reasons we had to set up a disconnected single node OKD "cluster". This is our brain dump as the OKD documentation sometimes refers to OpenShift image locations, we had to read multiple sections to get it working and we also consulted the OpenShift documentation at https://docs.redhat.com.

Updated on 2026-03-03: Fixed install-config.yaml and create manifest command

First we need to download the oc command in the version we would like to install. OKD provides it on its release page:

https://github.com/okd-project/okd/releases/download/4.21.0-okd-scos.3/openshift-client-linux-amd64-rhel9-4.21.0-okd-scos.3.tar.gz

To get an overview of releases available for OKD see https://amd64.origin.releases.ci.openshift.org/.

Another option is to extract the oc command from the release image, if you have a version of oc already installed:

oc adm release extract --tools quay.io/okd/scos-release:4.21.0-okd-scos.3

This will also extract the openshift-install tar.gz which we need later to generate the installation manifests and ignition configs.

Next we need the oc-mirror plugin, which is used to mirror the release content to our local registry. We were only able to find the plugin on console.redhat.com. Another option might be to compile the plugin from source code (https://github.com/openshift/oc-mirror/).

We copied the oc command and the oc-mirror plugin to /usr/local/bin/ and made them executable. After that we ran oc mirror --v2 --help to test the installation.

The upstream oc-mirror plugin documentation was also helpful.

As our private registry required authentication, we created a .dockerconfigjson file with the registry credentials:

{
"auths": {
"internal.registry": {
"auth": "<credentials base64 encoded>",
"email": "you@example.com"
}
}
}

You can encode the credentials with:

echo -n "username:password" | base64 -w0

For mirroring all required OKD images to our private registry we created an ImageSetConfiguration:

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
platform:
channels:
- name: 4-scos-stable
minVersion: 4.21.0-okd-scos.3
maxVersion: 4.21.0-okd-scos.3
graph: false
operators:
- registry: quay.io/okderators/catalog-index:testing-4.20 (1)
packages:
- name: aws-load-balancer-operator
- name: 3scale-operator
- name: node-observability-operator
additionalImages:
- name: registry.redhat.io/ubi8/ubi:latest
- name: registry.redhat.io/ubi9/ubi@sha256:20f695d2a91352d4eaa25107535126727b5945bff38ed36a3e59590f495046f0

We used a minimal configuration without operators and additional images, because downloading additional images did not work for us:

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
platform:
channels:
- name: 4-scos-stable (1)
type: okd
minVersion: 4.21.0-okd-scos.3 (2)
maxVersion: 4.21.0-okd-scos.3
graph: false
operators: []
additionalImages: []

The hard part for us was finding the right values for channels and the min/max versions. Once again the OKD release status page was helpful.

Our private registry is based on Harbor for our experiments, but any Docker v2 compatible registry should work. The only thing required is a project within Harbor called openshift and a user/password for pulling/pushing images to this project. The project name might be configurable with oc mirror, but this requires further investigation.

oc mirror wants to verify signatures so we had to download a public key and provide the OCP_SIGNATURE_URL:

curl -LO https://raw.githubusercontent.com/openshift/cluster-update-keys/master/keys/verifier-public-key-openshift-ci-4
export OCP_SIGNATURE_URL="https://storage.googleapis.com/openshift-ci-release/releases/signatures/openshift/release/"
export OCP_SIGNATURE_VERIFICATION_PK="verifier-public-key-openshift-ci-4"
# for debugging add --log-level debug, but we had intermittent errors with this option.
# after removing --log-level debug oc mirror ran without problems
oc mirror -c ImageSetConfiguration.yaml --authfile docker-auth.json --workspace file:///workspace docker://internal.registry --v2

This will create the ImageDigestMirrorSource in /workspace/working-dir/cluster-resources/idms-oc-mirror.yaml and ImageTagMirrorSource custom resources in /workspace/working-dir/cluster-resources/itms-oc-mirror.yaml.

The content of ImageDigestMirrorSource will be reused in the install-config.yaml. This is required to redirect image pulls done by the node from quay.io to our private registry.

Next we need DNS records for our cluster:

api.sno.internal
api-int.sno.internal
*.apps.sno.internal (wildcard DNS entry for applications deployed on the cluster)

Now we prepared the required install-config.yaml file to trigger the openshift-install command. We basically followed Installing a Single Node OpenShift Cluster.

apiVersion: v1
baseDomain: internal
compute:
- hyperthreading: Enabled
name: worker
replicas: 0 (1)
controlPlane:
hyperthreading: Enabled
name: master
replicas: 1 (2)
metadata:
name: okd-sno
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
networkType: OVNKubernetes
serviceNetwork:
- 172.30.0.0/16
platform:
none: {}
fips: false
sshKey: 'the key'
bootstrapInPlace: (3)
installationDisk: /dev/vda
pullSecret: |
{
"auths": {
"registry.internal": {
"auth": "<username:password base64 encoded>",
"email": "some@email"
}
}
}
additionalTrustBundle: | (4)
-----BEGIN CERTIFICATE-----
ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ
-----END CERTIFICATE-----
ImageDigestSources: (5)
- mirrors:
- registry.internal/openshift/release
source: quay.io/okd/scos-content
- mirrors:
- registry.internal/openshift/release-images
source: quay.io/okd/scos-release

1	We set the number of workers to zero (0). Because this is a SNO installation. Worker nodes could be added later, even for a SNO cluster
2	We set the number of master nodes to one (1)
3	This tells the installer that we do not have a bootstrap node and it should create one large ignition config file
4	If required the CA certificate of our registry. This is required if the registry uses an internal certificate
5	ImageDigestSources configures CRIO to redirect requests to quay.io to our private registry.

It’s time to create the installation manifests and ignition configs. We created a directory install, copied the install-config.yaml into this directory and triggered the openshift-install command:

$ mkdir install
$ cp install-config.yaml install/
$ openshift-install --dir=install create single-node-ignition-config

We are now ready to download the installation ISO and to embed our ignition config into it. We also want to set kernel boot arguments to configure the network interface with a static IP address.

curl -L $( ./openshift-install coreos print-stream-json |jq -r ".architectures.x86_64.artifacts.metal.formats.iso.disk.location" ) -o fhcos-live.iso

The next step is to modify the ISO. We basically followed the instructions in the OKD documentation: https://docs.okd.io/4.14/installing/installing_sno/install-sno-installing-sno.html#generating-the-install-iso-manually_install-sno-installing-sno-with-the-assisted-installer

alias coreos-installer='podman run --privileged --pull always --rm -v /dev:/dev -v /run/udev:/run/udev -v $PWD:/data -w /data quay.io/coreos/coreos-installer:release'
coreos-installer iso ignition embed -fi install/bootstrap-in-place-for-live-iso.ign fcos-live.iso
coreos-installer iso kargs modify --append 'ip=10.0.0.99::10.0.0.1:255.255.255.0:sno.internal:ens33:none:10.0.0.255' fcos-live.iso

We can take a look at the ISO bootstrap config and the kernel arguments with:

coreos-installer iso ignition show fcos-live.iso
coreos-installer iso kargs show fcos-live.iso

For understanding the kernel boot arguments see https://access.redhat.com/solutions/5499911.

The final step is to boot the modified ISO on the target machine and wait for the installation to complete.

OpenShift Virtualization Networking - The Overview

Mon, 29 Dec 2025 00:00:00 +0000

It’s time to dig into OpenShift Virtualization. You read that right, OpenShift Virtualization, based on kubevirt allows you to run Virtual Machines on top of OpenShift, next to Pods. If you come from a pure Kubernetes background, OpenShift Virtualization can feel like stumbling into a different dimension. In the world of Pods, we rarely care about Layer 2, MAC addresses, or VLANs. The SDN (Software Defined Network) handles the magic and we are happy.

But Virtual Machines are different….

They are needy creatures. They often refuse to live in the bubble of the default Pod network. They want to talk to the external Oracle database on the bare metal server next door, they need a static IP from the corporate range, or they need to be reachable via a specific VLAN. Networking is a key part of the Virtualization story and we need to master it.

To achieve this, we need to understand two tools: NMState Operator and Multus.

In this guide, I will try to discuss some of the basics for "Multihomed Nodes" and look at the three main ways to connect your VMs to the outside world.

Who does what: NMState vs. Multus

Before we look at YAML, we need to clear up a common confusion. Who does what?

Think of your OpenShift Node as a house.

NMState (The Electrician): This operator works on the Node (Host) level. It drills holes in the walls, lays the physical cables, and installs the wall sockets. In technical terms: It configures Bridges, Bonds, and VLANs on the Linux OS of the worker nodes.
Multus (The Extension Cord): This CNI plugin works on the VM/Pod level. It plugs the VM into the socket that NMState created. It allows a VM to have more than one network interface (eth0, eth1, etc.).

You usually need both. First, NMState builds the bridge. Then, Multus connects the VM to it.

NMState Operator Installation

While Multus is installed by default, NMState is not. You need to install the Operator first.

To install the NMState Operator, you can use the Web UI or the CLI. For the Web UI, you can use the following steps:

In the OpenShift Web Console, navigate to Operators → OperatorHub (or Ecosystem → Software Catalog in version 4.20+).
In the OperatorHub, search for NMState and select the Operator from the list.
Click on the NMState Operator and click on the Install button.
Select the Install button to install the Operator.
After the Operator is installed, you can check the status of the Operator in the Installed Operators view.

Once done you will need to create an instance of the NMState Operator. Simply create the following resource:

apiVersion: nmstate.io/v1
kind: NMState
metadata:
name: nmstate
spec:
probeConfiguration:
dns:
host: root-servers.net

NMState Operator Resource Types

The NMState Operator provides the three custom resource types to manage the node networking:

NodeNetworkState (NNS) → The "As-Is" State:
- This is an automatically generated report of the current status. It effectively tells you: "On Server 1, I currently see Network Card A and B, and IP address X is configured."
- Purpose: To verify the current reality on the ground before you make changes.
NodeNetworkConfigurationPolicy (NNCP) → The "To-Be" State (The Blueprint):
- This is the most critical resource. This resource is used to configure the network on the node. Here you tell the cluster that "I want a bridge named br1 to exist on all servers, and it must be connected to port ens4."
- The operator will try to configure this configuration across your nodes.
NodeNetworkConfigurationEnactment (NNCE) → The Result report:
- This resource is created autoamtically after the NNCP (the blueprint) has been created.
- It resports back: This resource is used to enact the network configuration on the node.
- It reports back on the execution: "Success! The bridge has been built," or "Failure! Cable not found."

With NMState in place, we can now start to configure the network on the node.

Part 1: Building the Bridge (NMState)

By default, Virtual Machines (VMs) are connected to the default pod network of the cluster as any other Pod. Using this network, the VM can communicate with other resources inside the cluster, or any resources that are reachable through the node network. Sometimes the VM needs a connection to a different network. In such a case you must connect the VM to an additional network. The VM will be configured with an additional network interface and is considered as multihomed VM.

To connect a VM to the physical network, we first need a bridge on the worker nodes. A bridge forwards packets between connected interfaces, similar to a network switch. We have two choices: Linux Bridge or OVS (Open vSwitch) Bridge.

Linux Bridge: The sturdy, wooden bridge. Simple, robust, standard Linux kernel tech. Use this for 90% of your use cases (Static IPs, simple VLAN access).
OVS Bridge: The magical, floating bridge. Complex, programmable, supports SDN logic. Use this only if you need advanced tunneling or integration with OVN policies on the physical interface.

Let’s create a standard Linux Bridge called br1 on all our worker nodes, attached to physical interface ens4.

We do this by applying a NodeNetworkConfigurationPolicy (NNCP).

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: br1-policy
spec:
nodeSelector:
node-role.kubernetes.io/worker: "" (1)
desiredState:
interfaces:
- name: br1 (2)
description: Linux bridge linked to ens4 (3)
type: linux-bridge (4)
state: up (5)
ipv4:
dhcp: true (6)
enabled: true
bridge: (7)
options:
stp:
enabled: false
port: (8)
- name: ens4

1	Apply to all worker nodes. If omitted, the policy will be applied to all nodes.
2	Name of the bridge
3	Description of the bridge
4	Type of the bridge (linux-bridge)
5	State of the bridge (up)
6	DHCP is enabled for the bridge (true)
7	Options for the bridge
8	Port of the bridge (ens4)

Setting an interface to absent or deleting an NNCP resource does not restore the previous configuration. A cluster administrator must define a policy with the previous configuration to restore settings.

Part 2: Defining the NetworkAttachmentDefinition (NAD)

Now that the bridge br1 exists on the nodes, we need to tell OpenShift how to use it. We do this with a NetworkAttachmentDefinition (NAD).

There are three distinct "Types" of networks you can define in a NAD.

1. The Linux Bridge (CNI-Plugin)

Linux bridges and OVS bridges both connect VMs to additional networks. A Linux bridge offers a simple, stable and sturdy solution and is well established. The OVS bridge on the other hand provides an advanced feature set for software-defined networks and is more challenging to troubleshoot.

Use Case: Direct Layer 2 connection to the physical network.

Behavior: The VM is practically "on the wire" of your datacenter. It can do DHCP with your corporate router.

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: bridge-network
namespace: my-vm-namespace
spec:
config: '{
"cniVersion": "0.3.1",
"name": "bridge-network",
"type": "bridge", (1)
"bridge": "br1", (2)
"ipam": {
"type": "dhcp"
}
}'

1	Type of the network (bridge)
2	Name of the bridge (br1)

type: "bridge" refers to the CNI plugin that utilizes the Linux Bridge we created earlier.

A note on IPAM (IP Address Management)

You might have noticed the "ipam" section in the NAD examples. You have three main choices there:

dhcp: Passes DHCP requests to the physical network (requires an external DHCP server).
static: You manually define IPs in the VM config (hard work).
whereabouts: A "Cluster-wide DHCP" for private networks. Perfect for the OVN L2 Overlay scenario where no external DHCP exists.

2. OVN Kubernetes L2 Overlay

A switched (layer 2) topology network interconnects workloads through a cluster-wide logical switch. This configuration allows East-West traffic only (packets between Pods within a cluster).

Use Case: You need a private network between your VMs (East-West traffic only), but you don’t want them to talk to the physical network.

Behavior: It creates a Geneve tunnel (like VXLAN) over the network.

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: private-overlay
namespace: my-vm-namespace
spec:
config: '{
"cniVersion": "0.4.0",
"name": "private-overlay",
"type": "ovn-k8s-cni-overlay",
"topology": "layer2", (1)
"netAttachDefName": "my-vm-namespace/private-overlay" (2)
}'

1	Topology of the network (layer2)
2	NetworkAttachmentDefinition name (my-vm-namespace/private-overlay)

A pod with the annotation k8s.v1.cni.cncf.io/networks: private-overlay will be connected to the private-overlay network.

3. OVN Kubernetes Secondary Localnet

OVN-Kubernetes also supports a localnet topology for secondary networks. This type creates a connection between a secondary network and a physical network, allowing VMs to communicate with destinations that are outside of the cluster. You must map the secondary network to the OVS bridge on the node. This is done by using a NodeNetworkConfigurationPolicy resource.

Use Case: You want to connect to the physical network (like the Linux Bridge), but you want to leverage OVN features (like NetworkPolicies) on that traffic.

Behavior: Uses Open vSwitch mapping to connect the OVN logic to the physical port.

Let’s create the bridge mapping by applying the following NodeNetworkConfigurationPolicy resource:

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: br0-ovs
spec:
nodeSelector:
node-role.kubernetes.io/worker: "" (1)
desiredState:
interfaces:
- name: br0-ovs (2)
description: OVS bridge with ens4 as a port
type: ovs-bridge (3)
state: up
ipv4:
dhcp: true
enabled: true
bridge:
options:
stp: true
port:
- name: ens4 (4)
ovn:
bridge-mappings:
- localnet: br0-network (5)
bridge: br0-ovs (6)
state: present

1	Apply to all worker nodes. If omitted, the policy will be applied to all nodes.
2	Name of the bridge
3	Type of the bridge (ovs-bridge)
4	Port of the bridge (ens4)
5	Localnet network name (br0-network)
6	OVS bridge name (br0-ovs)

Now let’s create the NetworkAttachmentDefinition for the localnet network so that VMs can connect to it. This NAD is created in the namespace of the VM.

If the NAD is created in the default namespace, the configuration will be available to all namespaces.

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: ovn-localnet
namespace: my-vm-namespace
spec:
config: '{
"cniVersion": "0.4.0",
"name": "ovn-localnet", (1)
"type": "ovn-k8s-cni-overlay", (2)
"topology": "localnet", (3)
"netAttachDefName": "my-vm-namespace/br0-network" (4)
}'

1	Name of the network (ovn-localnet)
2	Type of the network (ovn-k8s-cni-overlay)
3	Topology of the network (localnet)
4	Reference to the localnet mapping name defined in the NNCP (my-vm-namespace/br0-network)

Part 3: Connecting the VM

Finally, we have our "socket" (the NAD). Let’s plug the VM in. In your VirtualMachine manifest, you add the network to the spec.

This can be done via the Web Console or via the CLI.

Via the Web Console:

In the OpenShift Web Console, navigate to Virtualization → VirtualMachines.
Click on the VM you want to connect to the network.
Select the "Configuration" tab
Select "Network"

Click "Add network interface"
In the Popup enter a name for the interface
Model keep as "virtio"
On the "Network" select the NAD you created in the previous step.

Click "Save"

The VM will now have a Pending state. This is because the VM is waiting for a migration to activate the new network interface.

In the action menu you can select "Migrate" > "Compute" to start the migration process. After a while the new NIC will be active.

Via the CLI:

Here is a VM connected to the Linux Bridge NAD we created in step 1:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: helloworld-vm
namespace: my-vm-namespace
spec:
template:
spec:
domain:
devices:
interfaces:
- name: default
masquerade: {} (1)
- name: br0-network
bridge: {} (2)
networks:
- name: default
pod: {}
- name: br0-network (3)
multus:
networkName: br0-network (4)

1	The standard Pod Network
2	Our new connection
3	The NAD name
4	The NAD name

GitOps Catalog

Mon, 22 Dec 2025 00:00:00 +0000

The GitOps Catalog page provides an interactive visualization of all available ArgoCD applications from the openshift-clusterconfig-gitops repository. Check out the page GitOps Catalog for more details.

The Guide to OpenBao - Secrets Engines KV - Part 8

Wed, 25 Mar 2026 00:00:00 +0000

Secrets engines are the heart of OpenBao’s functionality. They store, generate, or encrypt data. This article covers the most commonly used secrets engine: KV for static secrets. It will demonstrate how to use the KV secrets engine to store and retrieve secrets. Upcoming articles will cover the PKI secrets engine and the integration with cert-manager.

Introduction

Secrets engines are components which store, generate, or encrypt data. Some engines simply store and read data, others connect to other services and other engines provide encryption as a service.

They are enabled at a path in OpenBao (as everything is done as a path in OpenBao). This means each secrets engine defines its very own path including properties.

The path is case-sensitive. For example, the KV secrets engine enabled at kv/ and KV/ are treated as two distinct instances of KV secrets engine.

The management of the secret engines is done via the CLI or API or, if enabled, the UI. Engines can be enabled, disabled, moved or tuned.

Enable - enables a secrets engine at the given path.
Disable - disables an existing secrets engine. All existing secrets are revoked and the data is deleted.
Move - moves the path for an existing secrets engine.
Tune - tunes global configuration for the secrets engine such as the TTLs.

List of Secrets Engines

To list all enabled secrets engines, we can use the following command:

bao secrets list

Which will output something like:

Path Type Accessor Description
---- ---- -------- -----------
cubbyhole/ cubbyhole cubbyhole_f8b1e784 per-token private secret storage
identity/ identity identity_76940581 identity store
secret/ kv kv_ccb47c73 n/a
sys/ system system_cb0d64b7 system endpoints used for control, policy and debugging

As a reminder: To login to the OpenBao CLI, we can use the following command (assuming that you are using HTTPS and have the CA certificate ready):

export BAO_ADDR='https://127.0.0.1:8200'
export BAO_CACERT="$PWD/openbao-ca.crt"
bao login

KV (Key-Value) Secrets Engine

The KV secrets engine is probably the most commonly used engine. It is a generic key-value store. When you work with KV secrets engine you will note very soon that there are two versions of the engine: KV v1 and KV v2.

KV Version 1

KV v1 is the older version of the KV secrets engine. It does not support versioning and only stores the recently written value for a key. Requests to this engine will be more performant because the storage requests will be fewer and no locking is required. The storage as such will be much smaller, since it does not need to retain any history.

KV Version 2

KV v2 on the other hand supports versioning of secrets. When a version is deleted, the underlying data is not removed, instead you must destroy it intentionally. Storing the history of secrets will increase the required storage space. However, I would recommend using KV v2 in most cases as the benefit of versioning is too great to ignore.

The following example will use KV v2.

KV Version 1 vs Version 2

Feature	Version 1	Version 2
Versioning	No	Yes (up to 10 versions by default)
Check-and-Set	No	Yes (prevents concurrent writes)
Soft delete	No	Yes (recoverable)
Metadata	No	Yes (custom metadata)

Feature

Version 1

Version 2

Versioning

Yes (up to 10 versions by default)

Check-and-Set

Yes (prevents concurrent writes)

Soft delete

Yes (recoverable)

Metadata

Yes (custom metadata)

KV v1 can be upgraded to KV v2 by using the following command as described in the documentation: Upgrading from version 1

Enabling KV Engine

Let’s enable the KV v2 engine at the path secret/ and give it the description "Application secrets".

This path was used in Part 7 of this series already. So you might have to remove or rename the engine first if you have already enabled it.

bao secrets enable \
-path=secret \ (1)
-version=2 \ (2)
-description="Application secrets" \
kv (3)

1	Path where the secrets will be stored
2	Version of the engine in case of KV
3	Name of the engine

Storing Secrets

With the enabled engine, we can now store secrets at the path:

# Put a secret (KV v2)
bao kv put secret/myapp/database \ (1)
username="dbadmin" \ (2)
password="supersecret123" \
host="db.example.com" \
port="5432"

1	Path where the secret will be stored
2	Secret data

Which will output something like:

======= Secret Path =======
secret/data/myapp/database (1)
======= Metadata =======
Key Value
--- -----
created_time 2026-03-13T05:56:52.670296216Z
custom_metadata <nil>
deletion_time n/a
destroyed false
version 1 (2)

1	Path where the secret will be stored
2	Version of the secret

Reading Secrets

To read the secret, we can use one of the following commands:

# Read a secret
bao kv get secret/myapp/database

Which will output something like:

======= Secret Path =======
secret/data/myapp/database (1)
[... more metadata ...]
====== Data ====== (2)
Key Value
--- -----
host db.example.com
password supersecret123
port 5432
username dbadmin

1	Path where the secret will be stored
2	Secret data

As an alternative, we can use the following command to get the secret as JSON:

# Get as JSON
bao kv get -format=json secret/myapp/database

We can also read a specific existing version of the secret by using the following command:

# Read specific version
bao kv get -version=2 secret/myapp/database

Or simply, a specific field:

# Get specific field
bao kv get -field=password secret/myapp/database

If the UI is enabled, the same can be achieved via the browser:

Figure 1. OpenBao Secret Retrieval

Managing Versions

KV v2 keeps a version history. This is a big advantage over KV v1. To try version management, ensure the secret has at least two versions. Create a second version by writing to the same path again (e.g. with updated values):

bao kv put secret/myapp/database username=dbadmin password=newpassword456 port=5432 (1)

1	Updating an existing secret with a new password, which will create a new version in KV v2.

Now you can list versions, soft-delete, undelete, or destroy a specific version:

I will only paste the output of the first command. The rest will be similar … you’ll get the idea.

# List all versions and metadata
bao kv metadata get secret/myapp/database

Which will output something like:

====== Version 1 ====== (1)
Key Value
--- -----
created_time 2026-03-13T05:56:52.670296216Z
deletion_time n/a
destroyed false
====== Version 2 ====== (2)
Key Value
--- -----
created_time 2026-03-13T06:13:18.539285992Z
deletion_time n/a
destroyed false

1	Version 1
2	Version 2

Specific versions can now be (soft-) deleted, permanently destroyed etc.

The last command of the list will completely delete all versions and metadata for a given path aka the secret is gone.

# Delete specific version (soft delete)
bao kv delete -versions=2 secret/myapp/database
# Undelete a version
bao kv undelete -versions=2 secret/myapp/database
# Permanently destroy a version
bao kv destroy -versions=2 secret/myapp/database
# Delete all versions and metadata
bao kv metadata delete secret/myapp/database

When you define a secret you can set the maximum number of versions to keep using the flag -max-versions=5. This is the number of versions that will be kept in the history.

Custom Metadata

A secret’s key metadata can contain multiple custom metadata that can be used to describe the secret. The data will be stored as key-value pairs. Imagine them like labels in Kubernetes.

Let’s extend our secret with some custom metadata:

# Add custom metadata to a secret
bao kv metadata put \
-custom-metadata="owner=team-a" \ (1)
-custom-metadata="environment=production" \ (2)
secret/myapp/database

1	First custom metadata to set an owner label
2	Second custom metadata to set an environment label

Policies for KV v2

OpenBao policies control which authenticated clients may access which paths and with what actions. For the KV v2 secrets engine, any read or write via the API is checked against the token’s policies; without a policy granting the right capabilities, requests return 403. This section shows example policy rules you can attach to roles (e.g. Kubernetes auth roles) or tokens so that applications and users have the minimum access they need.

KV v2 exposes two logical path trees under the mount path (here secret/):

secret/data/<path> — Secret values. Use capabilities such as read, create, update, and delete for reading and writing secret data.
secret/metadata/<path> — Version and metadata. Use list to enumerate keys, and read/delete for version metadata and destroying versions.

The following example grants read-only access to all secrets under myapp/, plus full read/write (including create, update, delete) for the single path myapp/database. Replace myapp with your application or team prefix as needed.

# Note: KV v2 uses /data/ for values and /metadata/ for metadata
# Read secret values under myapp/
path "secret/data/myapp/*" {
capabilities = ["read"]
}
# List secret keys under myapp/ (required for listing, e.g. bao kv list secret/myapp)
path "secret/metadata/myapp/*" {
capabilities = ["list"]
}
# Full access to a specific secret (create, read, update, delete)
path "secret/data/myapp/database" {
capabilities = ["create", "read", "update", "delete"]
}
# Read and delete version metadata for this secret (e.g. destroy version)
path "secret/metadata/myapp/database" {
capabilities = ["read", "list", "delete"]
}

After writing the policy to OpenBao (e.g. bao policy write myapp-kv policy.hcl), attach it to the appropriate auth method role or token so that clients receive a token with this policy. For token renewal, ensure the role or token also has update on auth/token/renew-self if you use renewable tokens.

Part 7 of this series (Authentication Methods) explains the authentication methods and policies in more detail.

Conclusion

This part focused on the KV secrets engine: enabling a mount, storing and reading secrets, version lifecycle (list, soft delete, destroy), custom metadata, and policies that distinguish KV v2’s data/ and metadata/ paths.

Key takeaways:

Prefer KV v2 when you need versioning, check-and-set, soft delete, and richer metadata; use KV v1 only when you explicitly want the simpler, lower-overhead path.
Grant secret/data/… for secret values and secret/metadata/… for listing and version operations—policies must match how the CLI and API resolve KV v2 paths.
Attach policies to tokens or auth roles (for example Kubernetes auth) so applications receive least-privilege access; include auth/token/renew-self if tokens should be renewable.

The PKI secrets engine and integration with cert-manager are covered in the next article in this series.

Resources

KV Secrets Engine v2

The Guide to OpenBao - Secrets Engines PKI - Part 9

Thu, 26 Mar 2026 00:00:00 +0000

Secrets engines are one of the most important concepts in OpenBao. Part 8 covered the KV secrets engine; this article covers the PKI secrets engine and its integration with cert-manager on Kubernetes and OpenShift.

Quick Navigation

Navigate directly to the main sections of this article:

PKI secrets engine
Integration with cert-manager

Introduction

Part 8 of this series explained the basics of secrets engines and the KV secrets engine. This article focuses on the PKI secrets engine and on integrating OpenBao with cert-manager.

As a reminder: to log in to the OpenBao CLI, use the following (assuming HTTPS and a CA certificate available locally):

export BAO_ADDR='https://127.0.0.1:8200'
export BAO_CACERT="$PWD/openbao-ca.crt"
bao login

PKI Secrets Engine

The PKI engine dynamically generates X.509 certificates, acting as a Certificate Authority (CA). You can obtain a certificate without manually generating a private key and CSR on the client: OpenBao provisions the key material and signed certificate according to the role.

The time to live (TTL) of each generated certificate can be kept short to reduce reliance on revocation. It is possible to store the certificates in memory during the startup of an application and discard them when the application shuts down. This way, a certificate is only valid during the application runtime, is kept in memory and is never written to disk.

Setting Up PKI Secrets Engine (Root CA)

Before we can use the PKI secrets engine, we need to enable and configure it. To enable it, we can simply use the following command:
```
bao secrets enable -path=pki pki
```
This will enable the PKI secrets engine at the path pki/. If the path parameter is not provided, the engine will use the name of the engine as path.

The default value of the TTL is 30 days. To increase the maximum lease time for certificates issued under this mount, use -max-lease-ttl=87600h (or another duration).

Note that individual roles can restrict this value to be shorter on a per-certificate basis. This just configures the global maximum for this secrets engine. Source: PKI secrets engine - setup and usage

Next we need to provide a CA certificate and a private key. OpenBao can either generate its own self-signed root certificate, or can use an existing one. Using an existing one is recommended for production environments. This is managed outside of OpenBao and OpenBao is provided with a signed intermediate CA.

We can simply generate a self-signed root certificate by using the following command:

bao write pki/root/generate/internal \
common_name="Example Root CA" \ (1)
issuer_name="root-ca" \ (2)
ttl=8760h \ (3)
key_bits=4096 \ (4)

1 Common name of the root certificate

2 Issuer name of the root certificate

3 Time to live of the root certificate, here 8760 hours (365 days)

Key bits of the root certificate, here 4096 bits (default is 2048 bits)

The response will look something like:

Key Value
--- -----
certificate -----BEGIN CERTIFICATE-----...
expiration 1536807433
issuing_ca -----BEGIN CERTIFICATE-----...
serial_number 7c:f1:fb:2c:6e:4d:99:0e:82:1b:08:0a:81:ed:61:3e:1d:fa:f5:29

The certificate key is stored in OpenBao. The returned certificates are purely informational.

Set URL configuration

The location of the Certificate Revocation List (CRL) and the issuing certificate are stored in the configuration of the PKI secrets engine and must be updated manually.
```
# Configure CA and CRL URLs
bao write pki/config/urls \
issuing_certificates="https://openbao.example.com/v1/pki/ca" \
crl_distribution_points="https://openbao.example.com/v1/pki/crl"
```
Configure a Role

Now we need to define a role for the certificates. Roles define the allowed domains and the maximum time to live for the certificates. The following example creates a role for server certificates. It allows us to issue certificates for the domains example.com and internal.example.com with a maximum time to live of 720 hours (30 days) and a key size of 2048 bits.
```
bao write pki/roles/server-cert \
allowed_domains="example.com,internal.example.com" \ (1)
allow_subdomains=true \ (2)
max_ttl=720h \
key_bits=2048 \ (3)
key_type=rsa \ (4)
require_cn=false \
allow_any_name=false
```
1 Allowed domains

2 Allow subdomains

3 Key size

4 Key type, typically rsa or ec (elliptic curve)

Issuing Certificates

OpenBao is now ready to issue certificates. We will use the issue endpoint to request a new certificate. The following example requests a new certificate for the domain myapp.example.com and the subdomain myapp.internal.example.com with a maximum time to live of 168 hours (7 days) and a key size of 2048 bits.

bao write pki/issue/server-cert \
common_name="myapp.example.com" \
alt_names="myapp.internal.example.com" \
ttl=168h \

This will return the full certificate chain including the root CA certificate and the key. For better readability, the output was shortened to show the certificate only:

Key Value
--- -----
certificate -----BEGIN CERTIFICATE-----
MIIE7zCCAtegAwIBAgIUH0eTVpXCP7iu0Y75dWAvjktbyxcwDQYJKoZIhvcNAQEL
BQAwGjEYMBYGA1UEAxMPRXhhbXBsZSBSb290IENBMB4XDTI2MDMyMjA1MjEyNVoX
DTI2MDMyOTA1MjE1NFowHDEaMBgGA1UEAxMRbXlhcHAuZXhhbXBsZS5jb20wggEi
MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDliY0K3qb8TwNBXOHdfBygs5+Y
Pwh83vtS0Nph6ZsPdI9JTuvI6AbzEz6M283aY7OeJ3m3G2fRwt7ZdgsAcyt+jnL8
Izx/MS/4ztsdZJ/wxs3M42/iFehSofqihgorY1+SGaTdveCFg9H7Egjn1eJFnGB/
iqANzxg5NohE4gf8zpAKDjl4CIPDSmNJX2xQ+3wXJgx1eKeVOeB0jFnMSYskS0hD
qwM+QyYRKodc/akI6Xu7OqWQLMW7qmNS+6TBHsGOopve9EP6ym65Jss6de4X9zIP
BYeWQ0gvG18i22eJrkISRWlO2uw+Fbho0tBhqfRfjsVpxrUrTKwIgxg+F0+5AgMB
AAGjggEpMIIBJTAOBgNVHQ8BAf8EBAMCA6gwHQYDVR0lBBYwFAYIKwYBBQUHAwEG
CCsGAQUFBwMCMB0GA1UdDgQWBBRbFm9oqRl//tc+Z0ert4DKp9KjkDAfBgNVHSME
GDAWgBSL03nIIcEY+IAz/1g2q5NBbufx9DBBBggrBgEFBQcBAQQ1MDMwMQYIKwYB
BQUHMAKGJWh0dHBzOi8vb3BlbmJhby5leGFtcGxlLmNvbS92MS9wa2kvY2EwOAYD
VR0RBDEwL4IRbXlhcHAuZXhhbXBsZS5jb22CGm15YXBwLmludGVybmFsLmV4YW1w
bGUuY29tMDcGA1UdHwQwMC4wLKAqoCiGJmh0dHBzOi8vb3BlbmJhby5leGFtcGxl
LmNvbS92MS9wa2kvY3JsMA0GCSqGSIb3DQEBCwUAA4ICAQBUoLwWgpTaodRKpL3c
zp/yPgHNagZA8SQRLqKttywbHzfPNUSsC56XqfZnOHaq/gobNKrmirUAHPnXRUjO
Jd2+wXBO9aKqAhiy5is2ueNRo+C8IdPQpFNQ+5MW+G4FXJqSkvDUlT47mAoRkKUJ
nsjoV/xzHYKgw/YF7QZtmL8moM9W9Bhg3dB6KEaUvscpNn/9USfPquVLXOuQ/8Bs
6zQGijJUhb5gpRgGjmJhs0Oo/MkABxBUq2jFmvGWP0NklcJO7smtetx/RSrtyFZ0
X7zgMlD1yUy4DaHX13LhszAfercTFNodV/mAfEaJUrW+U9I97MzVg8QJvn9/AsaM
mWjfKHczhd9AzupOlogX8L3cdZxx9hkdD4NcR8tRb4QR5sOqzoBN7NfNpQ699b0F
kxESrHIysqBcx4dYjXLFsUqzRGm2sGZ4j1N8w1OZY/s0kfsG7RXpXJjwtQ2RwPo1
iSjbJ/VLjji2pz5BtVRtHiXnkoVtSD8QBpjCYqyHekXUArJCVkaDPzkB2MWNZXpk
2koi2fYa0U4x4ITqDsKeMpLvGAl2vqkp20lphXXqRwaAdNuEFd69YZ6qXzw/6GdI
TC/Ki7zApSvO0kJvMs9NILcHgDjpndCA/sMpG9iv4wFMpgYet9wc+dzPMXnHcj4o
PGbyb/WJomcxQMDuV1x1S1X2iA==
-----END CERTIFICATE-----
expiration 1774761714
issuing_ca -----BEGIN CERTIFICATE-----
MIIFJTCCAw2gAwIBAgIUJde0eccC1TpbufxVsUBhZhq9hWwwDQYJKoZIhvcNAQEL
BQAwGjEYMBYGA1UEAxMPRXhhbXBsZSBSb290IENBMB4XDTI2MDMyMjA0NDI1OVoX
DTI2MDQyMzA0NDMyOVowGjEYMBYGA1UEAxMPRXhhbXBsZSBSb290IENBMIICIjAN
BgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA1XGanJKZZYySc9+WCXuGp9MLMo/O
MCRAR0u+BQbmxaSBcFLsxfNdkB8Jcunx7RnaGZIJwYXNIy+tNejqwQfiah7pTwJq
SYnXcYqZAETJUFvdYygjs4MCeaebvogmBQcZRzxJNNPzUblYpVOhmSGLvZZ7Vs5F
zY9wsKUEc3c3fMhwbxe/n/KQzvT/VzXSrLOw54LTsG7GNnUIhkuKwNqkhXuVItDy
ps+3z7RG/hOAWN9vktqLybi/E/uTmUqpsMrHSg7EMjqJ2jP8FMyH9EARLRL9gUG3
XFFNhFzjjDKjFrMsTUyVr+9tp4ucCZaakP9ZhjN9R/6FCXXR1K/XTVhTgYkawPHd
+PJaXn2T7NOie+tChiZx5Sii2XZtDm7WMFfwb4xhOBV67pWCSWSjkKl/xIDSrdRj
qLtzVfP/wiSL4sL9fy7Pya2OBtF2d/bdXXQ54JbbRlw3eRP+y+ej63UD+OP602EE
bbzrGB+B7ozwnfCmNNBnF23GgVYtJPWfDvgjXicprvbgjpmQAjeFohA8IIC322w5
sFsYuDL46KEs/KREoHjQNo1pxHKnmITU5GIMmWWoPyidTZuNNc4qWM6cH5RAbJup
IBaJWcWAr4eZ8p7pZd1dMkWEld4tieLdoJv2ENK77rDhwrq8kz66Xw9/xDJ/Nuio
sw5sd3py6Ayz+ekCAwEAAaNjMGEwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQF
MAMBAf8wHQYDVR0OBBYEFIvTecghwRj4gDP/WDark0Fu5/H0MB8GA1UdIwQYMBaA
FIvTecghwRj4gDP/WDark0Fu5/H0MA0GCSqGSIb3DQEBCwUAA4ICAQAJuRiReD46
H85vhc1wzg7JUy6k+q5aUA9iKOupGyYBtkvzniQzhouZGEP47AcWkbPNGqo/3yl+
SCUItkBfPjunBmzcAhhZqyxJJ1q/SgY21DBf3W7WB0yzPx+saCwEi74rf44HbslY
Gp9W9pv0NFDT1+lQGerp978ErGQZh2M8rrGFwhcHmx71/umZ9ju6xyaGRaoHba+Y
8bxhR11j/Zx/UGRQBpjdAe2+VuAHDGpHjd9ieQ45YgoyJYkjBbecXJFttcVnShwu
7rxu8HNV3fMjoa4M6TDaY4z8KDgzHChzQSRB085TJIZzcG+/4zpJredzLe+jEq0P
RAuFEx4FjpRqYQu08HUmepO0Vd8a2ut7WyIBOC5b+pM+55LaGDfDF6rt8H7g6G84
w/RYFP/w/elxp3T1cFgDXsuZ60mfK6SX5ZbH3ZGPN0+xF/VahncUWKSfxSaSjzRI
Eml/Aac22kgtbbtxf8Bn4tgtY30sxqQRVnldHKi02FxS+DNg3SZB1hIpw6myyUo0
XH/ZtyumDLkcCvKzz/KvQGDOWDBmaPfPOPfMgiFXryB+YFMfy75Bhm0+aE+GM8fP
sextO8kawYI62aCHiOClkm2qIBiFtbsNG2HMTL4HKaNDOh9vxesPD5EteVzxTvg4
Oyk8YPB4kGtloH569v2Fj2U24D6KRyqxvQ==
-----END CERTIFICATE-----
not_before 1774156885

That is it. We have just created our first certificate.

Setting Up an Intermediate CA

The test above demonstrated the usage of the PKI secrets engine to issue a certificate. However, the certificate was issued by the root CA. This is not practical or recommended for production environments. Instead, we should use an intermediate CA. The setup is very similar to the root CA, but with a few key differences: Instead of directly signing with the root CA, we will create a CSR (Certificate Signing Request) and sign it with the root CA. This can then be used to issue certificates with the intermediate CA.

First we enable the PKI secrets engine for the intermediate CA and tune it a little bit.
```
# Enable PKI engine for intermediate CA
bao secrets enable -path=pki_int pki
# Configure maximum lease time
bao secrets tune -max-lease-ttl=43800h pki_int
```
We set the maximum TTL of the intermediate CA to 43800 hours (5 years). It should be equal to or less than the root CA.

Generate a CSR (Certificate Signing Request)

# Generate intermediate CSR
bao write -format=json pki_int/intermediate/generate/internal \
common_name="Example Intermediate CA" \
issuer_name="intermediate-ca" \
key_bits=4096 \
| jq -r '.data.csr' > pki_int.csr

This will create the CSR and store it at pki_int.csr, which can be signed by the root CA and then used to issue certificates with the intermediate CA. As an example, the following CSR is generated:

Sign the CSR with the root CA

The CSR must now be signed using the root CA that was generated previously. Use:

bao write -format=json pki/root/sign-intermediate \
csr=@pki_int.csr \ (1)
format=pem_bundle \
ttl=43800h \
| jq -r '.data.certificate' > cert_bundle.pem (2)

1	The CSR file created in the previous step
2	The signed certificate is stored in the file `cert_bundle.pem` This will sign the CSR with the root CA and store the signed certificate at `cert_bundle.pem`.

Configure the set-signed certificate

We need to configure the intermediate CA with the signed certificate, so we can issue new certificates:
```
bao write pki_int/intermediate/set-signed \
certificate=@cert_bundle.pem
```

Creating Certificate Roles

Before we can issue certificates, we need to create roles. The following defines a role for server certificates and a role for client certificates, including allowed domains, maximum TTL, and key type.

# Create a role for issuing server certificates
bao write pki_int/roles/server-cert \
allowed_domains="example.com,internal.example.com" \ (1)
allow_subdomains=true \ (2)
max_ttl=720h \
key_bits=2048 \ (3)
key_type=rsa \
require_cn=false \
allow_any_name=false
# Create a role for client certificates
bao write pki_int/roles/client-cert \
allowed_domains="example.com" \ (1)
allow_subdomains=true \
max_ttl=168h \
key_usage="DigitalSignature,KeyEncipherment" \
ext_key_usage="ClientAuth"

The allowed domains, comma separated list

Allow subdomains, true or false

The key bits, 2048 or 4096

Check out the official documentation for more details of possible options: https://openbao.org/docs/

Issuing Certificates

Finally we can issue certificates. The example below issues a server certificate via the intermediate mount (and the article’s role setup supports a separate client role if you need mutual TLS).

# Issue a server certificate
bao write pki_int/issue/server-cert \
common_name="myapp.example.com" \ (1)
alt_names="myapp.internal.example.com" \ (2)
ttl=168h

1	The common name, the subdomain under example.com is allowed.
2	The alternative names This will print a large amount of detail. Typical fields include: `certificate` `issuing_ca` `private_key` `serial_number`

The output contains the full chain (intermediate and root CA), the leaf certificate, and the private key, which you can use to configure TLS in an application.

+ TIP: It can also be printed out in JSON format which is easier to process later.

+ The following example creates the JSON file with all the components.

bao write -format=json pki_int/issue/server-cert \
common_name="api.example.com" \
ttl=168h > api-cert.json

+ This can then be extracted to the individual components using the jq command.

cat /tmp/api-cert.json | jq -r '.data.certificate'
cat /tmp/api-cert.json | jq -r '.data.private_key'
cat /tmp/api-cert.json | jq -r '.data.ca_chain[]'

PKI Policies

Every OpenBao request is checked against the token’s policies; without an explicit allow, paths are denied. PKI is split into several API paths (issue, sign, ca, crl, …), so you grant only what each client needs.

For example: operators using pki_int/issue/… need the issue path; cert-manager sending a CSR needs the sign path; workloads or validators that fetch trust material need read on the CA (and often the CRL) paths etc.

The example below is a single policy you can attach to a role or token; split into narrower policies in production if different actors should not share the same privileges.

set -e
cat <<'EOF' | bao policy write pki-int-workload -
# Allow issuing certificates with the server-cert role (CLI / API issue endpoint)
path "pki_int/issue/server-cert" {
capabilities = ["create", "update"]
}
# Allow cert-manager (Vault issuer) to sign CSRs via the sign endpoint for the same role
path "pki_int/sign/server-cert" {
capabilities = ["create", "update"]
}
# Allow reading CA certificate
path "pki_int/ca/pem" {
capabilities = ["read"]
}
# Allow reading CRL
path "pki_int/crl/pem" {
capabilities = ["read"]
}
EOF
bao policy read pki-int-workload

Integration with cert-manager

cert-manager requests TLS certificates inside Kubernetes and stores them in Secrets. It includes a Vault issuer that talks to HashiCorp Vault’s HTTP API; OpenBao exposes the same API for PKI and auth, so you can point that issuer at OpenBao instead of Vault.

cert-manager generates a private key and a CSR in the cluster, then calls the PKI sign endpoint for your role (not the issue endpoint you used from the CLI earlier). The path in the issuer must therefore follow mount/sign/role-name, for example pki_int/sign/server-cert, matching the intermediate mount and role from this article.

Before the ClusterIssuer can log in to OpenBao, you need Kubernetes auth enabled, a policy that allows signing on pki_int (see PKI policies), and a role that binds the cert-manager controller’s service account to that policy. Part 7 of this series walks through Kubernetes authentication in detail; the commands below are a minimal summary for cert-manager.

Enable the Kubernetes auth method

Skip this step if kubernetes is already listed under bao auth list.

bao auth enable --description="Kubernetes/OpenShift service account auth" kubernetes

Configure the Kubernetes auth method

OpenBao’s server validates incoming service-account JWTs by calling the Kubernetes TokenReview API. If OpenBao runs inside the cluster (the setup assumed here), configure auth/kubernetes with:

kubernetes_host — API URL reachable from the OpenBao pods (typically https://kubernetes.default.svc:443).
kubernetes_ca_cert — PEM of the cluster CA that signs the API server certificate (same material as in the service-account token Secret).
token_reviewer_jwt — long-lived JWT for a dedicated service account that may create TokenReview objects (via system:auth-delegator).

If OpenBao runs outside the cluster, use your public API URL for kubernetes_host (for example $(oc whoami --show-server)), the same CA and reviewer JWT extracted from the cluster, and ensure the OpenBao process can reach that URL on the network.

In-cluster example: reviewer service account and Secret

Create a service account in the OpenBao namespace (here openbao), bind system:auth-delegator, and request a long-lived token Secret (required from Kubernetes 1.24 onward unless you use another token source).

---
apiVersion: v1
kind: ServiceAccount
metadata:
name: openbao-kube-auth
namespace: openbao (1)
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: openbao-kube-auth-delegator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator (2)
subjects:
- kind: ServiceAccount
name: openbao-kube-auth
namespace: openbao
---
apiVersion: v1
kind: Secret
metadata:
name: openbao-kube-auth-token
namespace: openbao
annotations:
kubernetes.io/service-account.name: openbao-kube-auth
type: kubernetes.io/service-account-token (3)

1	The namespace of the OpenBao service account
2	The name of the ClusterRole that binds the service account to the system:auth-delegator role
3	The type of the Secret, which is a long-lived token Secret

Save the YAML manifest to a file (for example openbao-kube-auth.yaml), apply it, then wait until the Secret contains a token (the control plane fills data.token—often within a few seconds).

oc apply -n openbao -f openbao-kube-auth.yaml
until oc get secret openbao-kube-auth-token -n openbao -o jsonpath='{.data.token}' | grep -q .
do
sleep 2
done

You can manage the reviewer service account, ClusterRoleBinding, and token Secret through GitOps so the configuration stays reproducible.

Configure auth/kubernetes/config

Dump the reviewer JWT and CA to files and configure the Kubernetes auth backend. Use the in-cluster API URL so OpenBao pods reach the Kubernetes API without relying on the external load balancer.

TMP=$(mktemp -d) (1)
trap 'rm -rf "$TMP"' EXIT
oc get secret openbao-kube-auth-token -n openbao -o jsonpath='{.data.token}' | base64 -d > "$TMP/reviewer.jwt" (2)
oc get secret openbao-kube-auth-token -n openbao -o jsonpath='{.data.ca\.crt}' | base64 -d > "$TMP/k8s-ca.crt" (3)
bao write auth/kubernetes/config \ (4)
kubernetes_host="https://kubernetes.default.svc:443" \
kubernetes_ca_cert=@"$TMP/k8s-ca.crt" \
token_reviewer_jwt=@"$TMP/reviewer.jwt"

1	Create a temporary directory to store the reviewer JWT and CA and remove it after the script exits
2	Get the reviewer JWT from the Secret
3	Get the CA from the Secret
4	Write the configuration to the auth/kubernetes/config path using the internal API URL of the Kubernetes cluster.

On plain Kubernetes, replace oc with kubectl and the same Secret paths.

If logins fail after cluster upgrades or with projected service-account tokens, check whether you must set issuer or related options; see OpenBao Kubernetes auth and Part 7.

To inspect the stored Kubernetes auth backend configuration:

bao read auth/kubernetes/config

You should see kubernetes_host and kubernetes_ca_cert; token_reviewer_jwt is often omitted or redacted on read.

Policy and role for cert-manager

Attach a policy (here cert-manager-pki) that matches the PKI paths cert-manager uses—aligned with PKI policies—then map the controller’s service account to that policy.

Adjust bound_service_account_names and bound_service_account_namespaces if your cert-manager installation does not use the cert-manager service account in the cert-manager namespace.

bao policy write cert-manager-pki - <<'EOF'
path "pki_int/sign/server-cert" {
capabilities = ["create", "update"]
}
path "pki_int/ca/pem" {
capabilities = ["read"]
}
path "pki_int/crl/pem" {
capabilities = ["read"]
}
path "auth/token/renew-self" {
capabilities = ["update"]
}
EOF
bao write auth/kubernetes/role/cert-manager \
bound_service_account_names=cert-manager \
bound_service_account_namespaces=cert-manager \
policies=cert-manager-pki \
ttl=1h \
max_ttl=24h

Verify with a Kubernetes login (optional)

After the cert-manager auth role exists, prove that OpenBao accepts the same service-account identity cert-manager will use. Request a short-lived projected token for that service account, then log in to the Kubernetes auth method:

JWT=$(oc create token cert-manager -n cert-manager --duration=15m)
bao write -format=json auth/kubernetes/login role=cert-manager jwt="$JWT" \
| jq -r '.auth.client_token'

A printed client token means TokenReview and role binding succeeded.

Configure the (Cluster)Issuer for cert-manager

To integrate cert-manager with OpenBao we need to configure either a ClusterIssuer (cluster-wide) or an Issuer (namespace-scoped). The setup is very similar.

The ClusterIssuer manifest below must use the same OpenBao role name in spec.vault.auth.kubernetes.role (here cert-manager).

The server URL in the issuer must be reachable from the cert-manager pods (in-cluster Service DNS is typical), for example https://openbao.openbao.svc:8200.

The following is an example of a ClusterIssuer manifest.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: openbao-cluster-issuer
spec:
vault:
path: pki_int/sign/server-cert (1)
server: https://openbao.openbao.svc:8200 (2)
caBundle: LS0tLS1CRUdJTi... (3)
auth:
kubernetes:
role: cert-manager
mountPath: /v1/auth/kubernetes (4)
serviceAccountRef:
name: cert-manager (5)

1	PKI sign path for the OpenBao role (mount/sign/role-name).
2	OpenBao Service URL as seen from cert-manager pods.
3	Required for private / self-signed OpenBao TLS: single-line base64 of the PEM CA (see the command block below). Omit only if the server uses a public CA trusted by default in the cert-manager image.
4	Kubernetes auth mount path in OpenBao (default is `/v1/auth/kubernetes`).
5	Name of the cert-manager controller service account only. Do not add `namespace` under `serviceAccountRef`; it is not part of the cert-manager API.

If OpenBao’s HTTPS certificate is not signed by a CA already trusted inside the cert-manager pods, you must supply trust material; otherwise TLS verification fails with x509: certificate signed by unknown authority.

Use caBundle (base64-encoded PEM of the CA that signed OpenBao’s server certificate) or caBundleSecretRef pointing to a Secret in the cert-manager namespace.

Fetch the certificate OpenBao is using:

oc get secret openbao-server-tls -n openbao -o jsonpath='{.data.ca\.crt}{"\n"}'

This secret was created in a previous article, when we were setting up the OpenBao server.

The returned string must be added to the ClusterIssuer manifest as the value of the caBundle field.

Alternatively, copy the CA PEM into a Secret in the cert-manager namespace and reference it (no giant line in the issuer):

# spec.vault fragment — use instead of caBundle if you prefer
caBundleSecretRef:
name: openbao-ca
key: ca.crt

However, this requires that the Secret exists in the appropriate namespace. For ClusterIssuer, the Secret must be in the cert-manager namespace.

Once the ClusterIssuer is created and is ready, we can issue a certificate.

Issue a Certificate

With the ClusterIssuer in place, create a Certificate in the workload namespace (here myapp, this namespace must exist). cert-manager will create a TLS Secret (myapp-tls below) containing tls.crt and tls.key, and renew the cert before it expires according to renewBefore.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: myapp-tls
namespace: myapp (1)
spec:
secretName: myapp-tls (2)
issuerRef: (3)
name: openbao-cluster-issuer
kind: ClusterIssuer
commonName: myapp.example.com (4)
dnsNames:
- myapp.example.com
duration: 168h (5)
renewBefore: 48h (6)
privateKey:
rotationPolicy: Always (7)

1	The namespace of the workload
2	The name of the TLS Secret
3	The issuer of the certificate
4	The common name of the certificate
5	The duration of the certificate
6	The duration before the certificate is renewed
7	Explicit rotation policy: `Always` matches cert-manager ≥ v1.18 default (new key each renewal). Set `Never` only if you rely on a stable key across renewals.

The values you set in commonName and dnsNames must be allowed by the OpenBao PKI role (allowed_domains, allow_subdomains, allow_bare_domains, and related settings).

If the Certificate never reaches Ready=True, check kubectl describe certificate / kubectl describe certificaterequest (or the same with oc describe on OpenShift) in that namespace, the cert-manager controller logs, and the OpenBao audit log for 403 policy denials or CSR rejection messages from the PKI engine.

When everything is working, a Secret named myapp-tls is created in the myapp namespace containing tls.crt and tls.key.

oc describe secret/myapp-tls -n myapp
[source,bash]
----
Name: myapp-tls
Namespace: myapp
[...output omitted...]
Type: kubernetes.io/tls
Data
====
tls.key: 1679 bytes
ca.crt: 1846 bytes
tls.crt: 3631 bytes

Conclusion

This part showed how to turn OpenBao into a usable CA for cluster workloads and how to automate TLS with cert-manager.

You should now be able to:

Enable the PKI engine, set CA and CRL URLs, define roles, and issue certificates via the issue API (CLI or automation).
Stand up an intermediate CA signed by the root, and keep day-to-day issuance on the intermediate mount (pki_int).
Express least-privilege with policies that separate issue from sign, and allow CA/CRL reads where validators or clients need them.
Wire cert-manager to OpenBao’s Vault-compatible API: Kubernetes auth in-cluster (reviewer JWT and CA), a ClusterIssuer with caBundle (or caBundleSecretRef) for private OpenBao TLS, serviceAccountRef.name only (no namespace field), and a Certificate with an explicit privateKey.rotationPolicy for cert-manager ≥ v1.18.

Resources

PKI Secrets Engine

The Guide to OpenBao - Authentication Methods - Part 7

Mon, 09 Mar 2026 00:00:00 +0000

With OpenBao deployed and running, the next critical step is configuring authentication. Ultimately, you want to limit access to only authorised people. This article covers two common authentication methods: Kubernetes for pods and LDAP for enterprise directories (in a simplified example). There are many more methods, but we cannot cover them all in this article.

Introduction

Authentication in OpenBao verifies the identity of clients before granting access to secrets. OpenBao supports multiple authentication methods, each suited for different use cases. Among others it supports:

Method

Best For

Key Features

Kubernetes

Pods running in K8s/OpenShift

Service account tokens, automatic

OIDC

Human users, SSO

OpenShift OAuth, Keycloak, Azure AD

LDAP

Enterprise directories

Active Directory, OpenLDAP

AppRole

CI/CD, automation

Role ID + Secret ID

Token

Direct access, bootstrap

Simple but less secure

The full documentation of the authentication methods and the list of available methods (for example Radius) can be found here: OpenBao Authentication Methods.

Authentication Workflow

The authentication workflow in OpenBao from a user perspective is as follows:

A client wants to authenticate to OpenBao and provides credentials.
OpenBao validates the credentials against the authentication method. For example: LDAP
If the authentication method (e.g. LDAP) is successful, it will return the required information about the client.
OpenBao maps this result to policies that are mapped to the authentication method.
OpenBao generates a token that is associated with the policies and returns it to the client.
The client can then use this token for further operations.

Prerequisites for Authentication Methods

Before configuring authentication:

OpenBao is deployed and unsealed (Parts 2-6)
You have admin access to OpenBao (root token)
Access to the required authentication backend:
- For Kubernetes auth: Access to the OpenShift cluster
- For LDAP: Access to the LDAP server
- etc.

Set up your environment:

You can set up your environment by using either the CLI or the UI.

Using the CLI

You can use the CLI to interact with OpenBao. In this example we will forward the port locally to the OpenBao server and set the environment variable BAO_ADDR to the local address. With boa login and the root token you can authenticate against OpenBao with full access.

# Port forward (if needed)
oc port-forward svc/openbao 8200:8200 -n openbao &
# Set environment
export BAO_ADDR='http://127.0.0.1:8200'
# You may need the SSL CA too.
# export BAO_CACERT="$PWD/openbao-ca.crt"
# Login with root token
bao login
# List all authentication methods
bao auth list

The last command returns a list of the available authentication methods. Currently there is only one authentication method enabled: token.

Path Type Accessor Description Version
---- ---- -------- ----------- -------
token/ token auth_token_1020fa7b token based credentials n/a

Using the UI

The UI, if enabled, is accessible via the OpenShift Route (if running on OpenShift and if it was created in Part 3). The password/token is the root token from the initialisation.

Figure 1. OpenBao UI Login Form

The UI has the advantage that it offers more visibility into the configuration and possibilities. For example, at the beginning there is one authentication method token enabled. This is required for the root token authentication.

Figure 2. OpenBao UI Authentication Methods

We can enable the other authentication methods by clicking on the Enable new method link to see what options are available.

Figure 3. OpenBao UI Configuring New Authentication Method

Understanding Policies

Before you can actually use an authentication method, it is important to understand what policies do. A policy is a way to declaratively define what (which paths) authenticated users can access or not. All paths are denied by default. A policy is mapped to an authentication method. For example, when we look at LDAP, we could configure something like this:

Every member of group "dev" is mapped to a policy named "dev-policy". The policy then allows the user to read the secrets under the path "secret/data/dev/*".

Basic Policy Structure

A policy is a set of path rules. Each rule says: “for this path (or path prefix), the bearer of this policy may use these capabilities.” Paths are tied to OpenBao’s internal API: for example, secrets in the KV v2 engine live under secret/data/<mount-path>/<key>, so a path like secret/data/myapp/* means “any key under the myapp prefix in the KV v2 store mounted at secret/.”

The example below does two things that are typical for an application policy:

Secrets access: It allows read and list on secret/data/myapp/*.
- read lets the client fetch the value of a secret at a path (e.g. secret/data/myapp/database).
- list is required to list keys under a path (e.g. to discover myapp/database, myapp/api-key); without it, the client would need to know every path in advance.
Token renewal: It allows update on auth/token/renew-self. Tokens often have a limited lifetime; this path lets the holder extend their own token without needing the root token or extra permissions. Including it avoids applications losing access when the token expires.

Policies can be written in JSON or HCL.

# Example policy: myapp-read-policy.hcl
# Allow reading secrets from specific path (KV v2 engine at mount "secret/")
path "secret/data/myapp/*" {
capabilities = ["read", "list"]
}
# Allow renewing own token so the app can extend its token before expiry
path "auth/token/renew-self" {
capabilities = ["update"]
}

The path secret/data/myapp/* assumes you have a KV v2 secrets engine mounted at secret/ and will store application secrets under keys like myapp/database, myapp/api-key, etc. Adjust the mount path and prefix to match your setup.

The use of globs (*) may result in surprising or unexpected behaviour. Use them with caution.

Policy Capabilities

Each path defined in a policy must have at least one capability. This provides control over the allowed or denied operations a user may perform. There are several capabilities available:

create - Create new data at the given path
read - Read data
update - Update existing data
patch - Patch existing data (Partial update)
delete - Delete data
list - List keys at a path
scan - Scan or browse the path for keys
sudo - Allows access to paths that are root-protected.
deny - Explicitly deny (overrides other grants)

Default Policies

When OpenBao is deployed the first time, there are no additional authentication methods enabled, except for token. This is required for the root access itself. To allow the administrator to access using the root token, there is one default policy called root. This policy is somewhat of a catch-all policy; it allows the administrator to access everything.

In addition to the root policy, there is one other default policy called default. This policy is used for all authenticated users. It allows them, for example, to look up their own properties and to renew or revoke their own token.

Let’s look at the default policies in the CLI:

A quick reminder of how to log in to OpenBao using the CLI when running it against Kubernetes: first open a port-forward, set the variable BAO_ADDR to the local address; if you use HTTPS you may need to set the SSL CA certificate, then log in with the root token.

You need to have the CA certificate in the current directory, which you might fetch with:

oc get secret openbao-ca-secret -n openbao -o jsonpath='{.data.ca\.crt}' | base64 -d > openbao-ca.crt

Port forward, set the environment variables and login with the root token:

# Port forward (if needed)
oc port-forward svc/openbao 8200:8200 -n openbao &
# Set environment
export BAO_ADDR='https://127.0.0.1:8200'
# You may need the SSL CA too.
export BAO_CACERT="$PWD/openbao-ca.crt"
# Login with root token
bao login

Now to list and fetch the current policies you can use the following commands:

# List policies
bao policy list
# Returns something like this:
#default
#root
# Read a policy
bao policy read default

The second command returns the policy content:

# Allow tokens to look up their own properties
path "auth/token/lookup-self" {
capabilities = ["read"]
}
# Allow tokens to renew themselves
path "auth/token/renew-self" {
capabilities = ["update"]
}
# Allow tokens to revoke themselves
path "auth/token/revoke-self" {
capabilities = ["update"]
}
# Allow a token to look up its own capabilities on a path
path "sys/capabilities-self" {
[...]

Again you will see the same policies in the UI. You can also manage them or create new ones using the UI.

Figure 4. OpenBao UI Listing Policies

Creating Policies using the CLI

Creating a policy, for example using the CLI, is straightforward. You can create a policy from a file or inline. The syntax is either JSON or HCL. The example below creates a policy called "myapp-read" from a file called "myapp-read-policy.hcl". The policy allows reading and listing secrets under the path "secret/data/myapp/*".

# Create a policy from file
bao policy write myapp-read myapp-read-policy.hcl
# Or inline
bao policy write myapp-read - <<EOF
path "secret/data/myapp/*" {
capabilities = ["read", "list"]
}
EOF
# List policies
bao policy list
# Read a policy
bao policy read myapp-read

Kubernetes Authentication Method

As we work a lot with OpenShift/Kubernetes, this method of authentication will be the first to try. The Kubernetes auth method can be used to authenticate against OpenBao using Kubernetes Service Account tokens. It is essential for pods to access secrets.

Let us try a simple example.

Step 1: Enable Kubernetes Auth

Assuming you are still logged into OpenBao, you can enable the kubernetes auth method with the following command:

bao auth enable --description="Authentication method for Kubernetes/OpenShift cluster" kubernetes

This simply enabled the authentication method and set a description. However, we need to configure it further.

Step 2: Configure Kubernetes Auth

To configure the kubernetes auth method, you need to get the Kubernetes API server address. You can get it with the following command (assuming you are logged into the OpenShift cluster):

The command oc can be replaced by kubectl if you are not using an OpenShift cluster.

# Get the Kubernetes API server address
KUBERNETES_HOST=$(oc whoami --show-server) (1)
# Configure the auth method
bao write auth/kubernetes/config \ (2)
kubernetes_host="$KUBERNETES_HOST"

1	Get the Kubernetes API server address. This will use something like https://api.cluster.example.com:6443
2	Configure the auth method with the Kubernetes API server address.

When running inside Kubernetes, OpenBao automatically uses the pod’s service account to communicate with the Kubernetes API.

Step 3: Create a Policy for the Application

Now we need to associate a policy to the authentication method. We could also assign the default policy but we want to provide access to the path secret/data/expense/database and secret/metadata/expense/config for the application.

# Create policy for an example application
bao policy write expense-app - <<EOF
# Read database credentials
path "secret/data/expense/database" {
capabilities = ["read"]
}
# Read application config
path "secret/data/expense/config" {
capabilities = ["read", "list"]
}
# Allow token renewal
path "auth/token/renew-self" {
capabilities = ["update"]
}
EOF

Step 4: Create a Kubernetes Auth Role

As the next step we map the policy to the authentication method. We will further limit the access to the namespace expense and the service account expense-app.

# Create a role that maps Kubernetes service accounts to OpenBao policies
bao write auth/kubernetes/role/expense-app \
bound_service_account_names=expense-app \ (1)
bound_service_account_namespaces=expense \ (2)
policies=expense-app \ (3)
ttl=1h \ (4)
max_ttl=24h (5)

1	`bound_service_account_names`: K8s service account name(s)
2	`bound_service_account_namespaces`: K8s namespace(s)
3	`policies`: OpenBao policies to attach
4	`ttl`: Token time-to-live
5	`max_ttl`: Maximum token lifetime

All of the configuration above, created using the CLI, can also be done using the UI of OpenBao.

Ensure the KV v2 secrets engine is mounted at `secret/`

We will discuss secret engines in more detail in the next part of the guide.

The policy and demo commands in this guide use the path secret/data/expense/database, which assumes a KV v2 secrets engine mounted at secret/. To check and enable it:

Check existing mounts: List enabled secrets engines and look for secret/ with type kv (version 2):
```
# List all secrets engine mounts (requires a token with read on sys/mounts)
bao secrets list
```
Look for an entry like secret/ with type kv or kv-v2. If you see secret/ and it is KV v2, you can skip to Step 5.
Enable KV v2 at secret/ if missing: If secret/ is not listed, or you want to use a fresh mount, enable the KV v2 engine at the path secret:
```
# Enable KV v2 secrets engine at path "secret" (requires root or policy with sys/mounts capability)
bao secrets enable -path=secret -version=2 kv
```
If secret/ already exists as KV v1, you cannot change it in place to v2; use a different path (e.g. secretv2/) and update the policy and demo paths accordingly (secretv2/data/expense/database, etc.).

Step 5: Store the database secret in OpenBao (one-time)

Before the application can read it, store the database password (and any other keys) at the path the policy allows. Run this once from a machine that has OpenBao access (e.g. bao CLI and a token).

You must use a token that has create and update capability on the KV path. The expense-app policy is read-only (for the application). Use the root token from OpenBao initialisation, or log in with bao login and use a token that has a policy granting create and update on secret/data/expense/*. If you use a read-only token (e.g. one that only has the expense-app policy), you will get a 403 "preflight capability check returned 403".

# Store the database secret at secret/data/expense/database (KV v2 path)
bao kv put secret/expense/database password="my-super-secret-db-password" username="expense_db_user"

This creates (or overwrites) the secret at secret/data/expense/database. The policy grants the expense-app role read on this path only; the application cannot create or update it.

Step 6: Create the expense namespace and demo application

Apply the following manifests to create the namespace, service account, optional long-lived token (Kubernetes 1.24+), and a demo deployment that authenticates to OpenBao and fetches the database secret. Adjust the OpenBao URL if your OpenBao is in another namespace or uses HTTPS (e.g. https://openbao.openbao.svc:8200).

Remember: We have created a policy that allows the service account with the name expense-app in the namespace expense to read the secret at secret/data/expense/database. If you want to use different paths or namespaces, you need to adjust the policy accordingly.

# full-demo-expense-app.yaml
# 1. Namespace
apiVersion: v1
kind: Namespace (1)
metadata:
name: expense
---
# 2. Service account used by the demo app to authenticate to OpenBao
apiVersion: v1
kind: ServiceAccount (2)
metadata:
name: expense-app
namespace: expense
---
# 3. Long-lived token for Kubernetes 1.24+ (optional; allows pods to use the SA token)
apiVersion: v1
kind: Secret (3)
metadata:
name: expense-app-token
namespace: expense
annotations:
kubernetes.io/service-account.name: expense-app
type: kubernetes.io/service-account-token
---
# 4. Demo deployment: authenticates to OpenBao and fetches secret/data/expense/database
apiVersion: apps/v1
kind: Deployment
metadata:
name: expense-demo
namespace: expense
labels:
app: expense-demo
spec:
replicas: 1
selector:
matchLabels:
app: expense-demo
template:
metadata:
labels:
app: expense-demo
spec:
serviceAccountName: expense-app
containers:
- name: demo
image: registry.access.redhat.com/ubi9/ubi-minimal:latest
command:
- /bin/sh
- -c
- |
echo "=== Installing curl ==="
microdnf install -y curl 2>/dev/null || true
echo "=== Fetching database secret from OpenBao ==="
JWT=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
OPENBAO_URL="${OPENBAO_URL:-http://openbao.openbao:8200}"
RESP=$(curl -s -k -X POST -H "Content-Type: application/json" \ (4)
-d "{\"jwt\": \"$JWT\", \"role\": \"expense-app\"}" \
"$OPENBAO_URL/v1/auth/kubernetes/login")
TOKEN=$(echo "$RESP" | sed -n 's/.*"client_token":"\([^"]*\)".*/\1/p')
if [ -z "$TOKEN" ]; then
echo "ERROR: Failed to get OpenBao token. Check OpenBao URL ($OPENBAO_URL), role expense-app, and network."
echo "Response: $RESP"
exit 1
fi
echo "OpenBao token obtained. Reading secret/data/expense/database ..."
SECRET=$(curl -s -k -H "X-Vault-Token: $TOKEN" "$OPENBAO_URL/v1/secret/data/expense/database") (5)
echo "$SECRET"
PASSWORD=$(echo "$SECRET" | sed -n 's/.*"password":"\([^"]*\)".*/\1/p')
echo "Database password (data.data.password): ${PASSWORD:-<not set>}"
echo "=== Demo complete. Pod stays running; exec in to run more curl commands. ==="
exec sleep infinity
env:
# Override if OpenBao is in another namespace or uses HTTPS
- name: OPENBAO_URL
value: "https://openbao.openbao:8200"
resources:
requests:
memory: "64Mi"
cpu: "10m"

1	Create the namespace
2	Create the service account
3	Create the secret (long-lived token for Kubernetes 1.24+)
4	Authenticate to OpenBao and get a client token (using the Kubernetes JWT)
5	Fetch the database secret

The examples above use curl -k to bypass certificate verification. For production, you should adjust this and mount the CA certificate in the pod (see the note earlier in this section on using the openbao-ca Secret).

Apply the manifests and wait for the pod to be ready:

oc apply -f full-demo-expense-app.yaml
# Wait for the demo pod to be running
oc -n expense rollout status deployment/expense-demo
# View the log: you should see the OpenBao token response and the fetched secret (including the password)
oc -n expense logs -l app=expense-demo -f

The log output shows the application authenticating with the Kubernetes JWT, receiving an OpenBao token, and reading the secret at secret/data/expense/database. The field data.data.password is the database password you stored in Step 5.

Step 7: Test Kubernetes authentication manually

From any pod that uses the same service account (e.g. the demo pod above), you can run the same steps by hand:

# From within a pod using the service account (e.g. oc exec -it deployment/expense-demo -n expense -- /bin/sh)
JWT=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
# 1. Authenticate to OpenBao and get a client token (sed works on UBI minimal; use jq -r .auth.client_token if jq is available)
LOGIN=$(curl -s -k --request POST \
--data "{\"jwt\": \"$JWT\", \"role\": \"expense-app\"}" \
https://openbao.openbao:8200/v1/auth/kubernetes/login)
TOKEN=$(echo "$LOGIN" | sed -n 's/.*"client_token":"\([^"]*\)".*/\1/p')
# 2. Fetch the database secret (same path the policy allows)
curl -s -k -H "X-Vault-Token: $TOKEN" https://openbao.openbao:8200/v1/secret/data/expense/database

Summary: Kubernetes auth method

A lot of steps were involved to configure the Kubernetes auth method. Let us summarize them:

In OpenBao: Enabled the Kubernetes auth method, configured it with the cluster API server address, created the expense-app policy (read on secret/data/expense/database and secret/data/expense/config, update on auth/token/renew-self), and created the expense-app role that binds the Kubernetes service account expense-app in namespace expense to that policy.
Secrets engine: Ensured the KV v2 engine is mounted at secret/ and stored the database secret at secret/expense/database (one-time, using the root token).
In the cluster: Created the expense namespace, the expense-app service account, an optional long-lived token Secret (for Kubernetes 1.24+), and the expense-demo deployment that uses that service account to authenticate to OpenBao and fetch the secret. For HTTPS, created the openbao-ca Secret in expense with the OpenBao server CA.
Verification: Applied the manifests, confirmed the demo pod starts and logs show a successful login and the fetched secret (including the database password). Optionally ran the same login and read steps manually via oc exec into the pod.

LDAP Authentication Method

LDAP auth integrates with enterprise directories like Active Directory.

This is a very simplified example. As a prerequisite you need to have an LDAP server and a service account with the necessary permissions to read the users and groups. This is not part of this guide.

Step 1: Enable LDAP Auth

To enable LDAP authentication, you need to use the following command:

bao auth enable ldap

Step 2: Configure LDAP (Active Directory Example)

In this step you need to configure the LDAP server address, the user and group attributes, the bind DN and the bind password. The example below configures the LDAP server address to ldaps://ad.example.com:636, the user attribute to sAMAccountName, the user DN to OU=Users,DC=example,DC=com, the group DN to OU=Groups,DC=example,DC=com, the group attribute to cn, the group filter to (&(objectClass=group)(member:1.2.840.113556.1.4.1941:={{.UserDN}})), the bind DN to CN=openbao-svc,OU=ServiceAccounts,DC=example,DC=com and the bind password to service-account-password.

bao write auth/ldap/config \
url="ldaps://ad.example.com:636" \
userattr="sAMAccountName" \
userdn="OU=Users,DC=example,DC=com" \
groupdn="OU=Groups,DC=example,DC=com" \
groupattr="cn" \
groupfilter="(&(objectClass=group)(member:1.2.840.113556.1.4.1941:={{.UserDN}}))" \
binddn="CN=openbao-svc,OU=ServiceAccounts,DC=example,DC=com" \
bindpass="service-account-password" \
certificate=@/path/to/ldap-ca.pem \
insecure_tls=false \
starttls=false

Step 3: Create the policies used by the group mapping

Before mapping LDAP groups to policy names, create those policies. Below are example definitions: admin-policy grants read/list/update on secrets (e.g. for operators), and user-policy grants read-only access to a limited path.

admin-policy — for members of openbao-admins (e.g. operators who may read and update secrets)

user-policy — for members of openbao-users (e.g. developers with read-only access to a subset of secrets)

Create them with the CLI (run before the group mapping in Step 4):

# Create admin-policy (e.g. from file admin-policy.hcl)
bao policy write admin-policy - <<EOF
path "secret/data/*" {
capabilities = ["read", "list", "update", "create", "delete"]
}
path "secret/metadata/*" {
capabilities = ["read", "list"]
}
path "auth/token/renew-self" {
capabilities = ["update"]
}
EOF
# Create user-policy
bao policy write user-policy - <<EOF
path "secret/data/myapp/*" {
capabilities = ["read", "list"]
}
path "auth/token/renew-self" {
capabilities = ["update"]
}
EOF

Adjust paths (e.g. secret/data/myapp/) to match your KV v2 mount and the paths you want each role to access. The default policy is built-in and grants token lookup/renew/revoke-self. Attaching it in addition to admin-policy or user-policy is a typical pattern.

Step 4: Map LDAP groups to policies

Now map the LDAP groups to the policy names. The example below maps the LDAP group openbao-admins to the policies admin-policy and default and the LDAP group openbao-users to the policies user-policy and default.

# Map an LDAP group to policies
bao write auth/ldap/groups/openbao-admins \
policies="admin-policy,default"
bao write auth/ldap/groups/openbao-users \
policies="user-policy,default"

Step 5: Test LDAP authentication

Assuming you have a user jdoe in the LDAP group openbao-users, you can test the LDAP authentication with the following commands:

# Login with LDAP credentials
bao login -method=ldap username=jdoe
# Or via API
curl --request POST \
--data '{"password": "user-password"}' \
$BAO_ADDR/v1/auth/ldap/login/jdoe

What jdoe can and cannot do (user-policy)

jdoe holds user-policy (and default), so their token has only the capabilities defined there.

What jdoe can do:

Get a secret under secret/data/myapp/*: after logging in, jdoe can read and list secrets in that path. For example:

# After bao login -method=ldap username=jdoe (and entering the password)
bao kv get secret/myapp/database
# Or via API (use the client_token from the login response)
curl -s -H "X-Vault-Token: $TOKEN" $BAO_ADDR/v1/secret/data/myapp/database

List keys under secret/data/myapp/ to discover available secrets.
Renew or revoke their own token (from the default policy) and look up their own token properties.

What jdoe cannot do:

Read secrets outside myapp — e.g. secret/data/expense/database or secret/data/other-app/* returns permission denied.
Create, update, or delete any secret — user-policy grants only read and list on secret/data/myapp/*, so write operations are denied.
Access admin or system paths — no access to sys/, other auth methods, or policies. Only the paths explicitly granted in *user-policy and default are allowed.

Common Practices for Authentication

Use the principle of least privilege - Do not grant more permissions than necessary.

# Bad: Too broad
path "secret/*" {
capabilities = ["read", "list"]
}
# Good: Specific paths
path "secret/data/myapp/database" {
capabilities = ["read"]
}

Set Appropriate Token TTLs - Set the appropriate TTLs for different use cases.

# Short-lived for automated systems
bao write auth/kubernetes/role/batch-job \
ttl=5m \
max_ttl=15m
# Longer for interactive users
bao write auth/oidc/role/human-user \
ttl=8h \
max_ttl=24h

Enable Audit Logging - Enable audit logging to track authentication attempts and other activities.

# Enable file audit
bao audit enable file file_path=/var/log/openbao/audit.log
# All authentication attempts are logged

Regularly Rotate Credentials - Rotate the credentials periodically to reduce the risk of compromise.

# Rotate AppRole Secret IDs periodically
bao write -f auth/approle/role/cicd-role/secret-id
# Revoke old tokens
bao token revoke <old-token>

Common Authentication Patterns

Pattern 1: Application Pods

Pod → Service Account Token → Kubernetes Auth → Policy → Secret

Pattern 2: Human Users

User → LDAP/OIDC Login → Group Mapping → Policy → Secret

Pattern 3: CI/CD Pipeline

Pipeline → AppRole (Role ID + Secret ID) → Policy → Secret

What is Coming Next?

In Part 8, we will explore secrets engines, like:

KV (Key-Value) for static secrets

In upcoming parts, I will try to cover common practices for OpenBao operation and maintenance. If there is time and an opportunity to test other authentication methods, I will do so.

Conclusion

Proper authentication configuration is crucial for OpenBao security. Key takeaways:

Use Kubernetes auth for pods
Use LDAP/OIDC for human users (SSO)
Use AppRole for automation
Create specific policies with least privilege
Set appropriate TTLs for different use cases
Enable audit logging for compliance

Resources

The Guide to OpenBao - Initialisation, Unsealing, and Auto-Unseal - Part 6

Thu, 05 Mar 2026 00:00:00 +0000

After deploying OpenBao via GitOps (Part 5), OpenBao must be initialised and then unsealed before it becomes functional. You usually do not want to do this unsealing manually, since this is not scalable especially in bigger, productive environments. This article explains how to handle initialisation and unsealing, and possible options to configure an auto-unseal process so that OpenBao unseals itself on every restart without manual key entry.

What is happening?

OpenBao starts in a sealed state. You must:

Initialise (once): run bao operator init to generate unseal keys (or recovery keys when using auto-unseal).
Unseal (after each restart): provide a threshold of unseal keys so each node can decrypt the root key. In the previous articles, we used a threshold of 3 (out of 5). This means that you need to run bao operator unseal three times with different keys on every node after every start before the service becomes available.

Unseal workflow (assuming a threshold of 3 and 5 nodes and initialisation already happened):

---
title: "Unseal Workflow"
config:
theme: 'dark'
---
flowchart LR
A["openbao-0 starts → 3× unseal using different keys"]
A --> B["openbao-1 starts → 3× unseal using different keys"]
B --> C["openbao-2 starts → 3× unseal using different keys"]
C --> D[OpenBao is ready]

This is a chicken-egg problem: you need to start somewhere, but you cannot start without the keys.

Manual unsealing does not scale very well and for production, auto-unseal is recommended.

This article covers:

Practical approaches to initialisation and unsealing (manual, Jobs, init containers).
Potential auto-unseal options: with AWS KMS, static key, and transit seal.

Handling Initialisation and Unsealing

The challenge with GitOps is that initialisation and unsealing are one-off or stateful operations. Below are several approaches. The problem is always the same: where do you start? You must have the unseal keys available to start the process. Do you keep them locally, do you keep them in a Kubernetes Secret, do you use an external Key Management System (KMS)? The most robust for production in my opinion is auto-unseal using a KMS (see [_approach_4_auto_unseal_with_kms_recommended_for_production] and the following sections). It uses an external Key Management System (KMS) to store the keys. This way, the keys are not stored in the cluster and are not exposed to the risk of being compromised.

Let’s have a look at the different options.

Approach 1: Manual Initialisation (simplest)

While manual initialisation and unsealing is the simplest approach, it is not recommended for production. Still, I would like to mention it here for completeness and at least provide a simple for loop to unseal the OpenBao service (on Kubernetes/OpenShift).

Manual unsealing is still a valid option when you have a small cluster and assume that it is not required often.

After Argo CD deploys OpenBao, the first pod, openbao-0, will not become ready until it is initialised and unsealed. Once done, the next pod, openbao-1, will try to start and wait until it is unsealed, and then the third pod, and so on.

The following, simplified script will go through the pods and unseal them one by one. It initialises the first pod and stores the unseal keys in the file openbao-init.json locally on your machine. From there it takes three different keys and unseals the OpenBao service. Since it takes a while until other pods are pulling the image and starting, we will use a sleep timer of 30 seconds between each pod. This may or may not be enough depending on your network and cluster speed. As said, this is a very simplified script:

The CLI tool oc can be replaced by kubectl in case you are using a different cluster. In addition, you will need the jq command line tool to parse the JSON file.

If your OpenBao was initialised previously, you can remove the first three commands and start with the unseal commands.

# Initialise the first pod, assuming the Pod openbao-0 is already running.
# You can skip this if OpenBao was initialised previously.
echo "Initialising OpenBao on openbao-0 (keys will be saved to openbao-init.json)..."
oc exec -it openbao-0 -n openbao -- bao operator init \
-key-shares=5 -key-threshold=3 -format=json > openbao-init.json (1)
echo "Init complete. Unseal keys and root token are in openbao-init.json."
# Unseal each pod (three times each with different keys)
echo "Unsealing pods openbao-0, openbao-1, openbao-2 (3 key steps each)..."
for i in 0 1 2; do
sleep_timer=30
SLEEPER_TMP=1
echo "Unsealing openbao-$i..."
for j in 1 2 3; do (2)
KEY=$(cat openbao-init.json | jq -r ".unseal_keys_b64[$((j-1))]")
oc exec -it openbao-$i -n openbao -- bao operator unseal $KEY
done
echo "openbao-$i unsealed. Waiting ${sleep_timer}s for next pod to be ready..."
while [[ $SLEEPER_TMP -le "$sleep_timer" ]]; do (3)
if (( SLEEPER_TMP % 10 == 0 )); then
echo -n "$SLEEPER_TMP"
else
echo -n "."
fi
sleep 1
SLEEPER_TMP=$((SLEEPER_TMP + 1))
done
echo ""
done
echo "All pods unsealed. Store init data securely (e.g. in a password manager)."
# Store init data securely (e.g. in a password manager)

1	Initialise the first pod, assuming the Pod openbao-0 is already running. This will store the keys (including the root token) in the file openbao-init.json.
2	Unseal each pod (three times each with different keys). The keys are taken from the openbao-init.json file.
3	Wait 30 seconds between each pod to give the other pods time to start and pull the image … super-fancy sleep timer.

The unseal keys as well as the root token are stored in the openbao-init.json file. This file is stored locally on your machine. Make sure to store it securely and do not share it with anyone.

Approach 2: Kubernetes Job for initialisation

A Job can run after OpenBao is deployed, initialise it if needed, and store the init output (unseal keys and root token) in a Kubernetes Secret. You can then unseal manually or with a second Job that reads from that Secret. This section assumes the init output is stored in a Kubernetes Secret and explains how to create that Secret from the Job and how to use it.

How the Secret is created

The init Job runs bao operator init and then creates the Secret using oc (or kubectl). The Secret name and key used here are openbao-init-data and init.json (the file contains the JSON output of bao operator init, including unseal_keys_b64 and root_token). One advantage of this approach is that you can integrate it into your GitOps pipeline. However, we can argue whether it is a good idea to store the unseal keys and the root token as a Secret in the same cluster where OpenBao is running.

Example structure of init.json:

{
"unseal_keys_b64": [
"key1",
"key2",
"key3",
"key4",
"key5"
],
"unseal_keys_hex": [
"hex1",
"hex2",
"hex3",
"hex4",
"hex5"
],
"unseal_shares": 5,
"unseal_threshold": 3,
"recovery_keys_b64": null,
"recovery_keys_hex": null,
"recovery_keys_shares": 0,
"recovery_keys_threshold": 0,
"root_token": "root.token"
}

You need at least unseal_threshold keys (e.g. 3) from unseal_keys_b64 to unseal each node. Store this file securely; anyone with access can unseal OpenBao and the root_token has full admin access.

RBAC Configuration

The Job’s service account (openbao-init) must be allowed to create (and optionally get/update) Secrets in the openbao namespace. Create a Role and RoleBinding:

apiVersion: v1
kind: ServiceAccount
metadata:
name: openbao-init (1)
namespace: openbao
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: openbao-init-secret-writer
namespace: openbao
rules: (2)
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create", "get", "update", "patch"]
- apiGroups: [""] (3)
resources: ["pods"]
verbs: ["get", "list"]
- apiGroups: [""] (4)
resources: ["pods/exec"]
verbs: ["create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding (5)
metadata:
name: openbao-init-secret-writer
namespace: openbao
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: openbao-init-secret-writer
subjects:
- kind: ServiceAccount
name: openbao-init
namespace: openbao

1	The ServiceAccount is used by the Job(s) to create the Secret and unseal the OpenBao pods.
2	The Role is used by the Job to create the Secret. The rules might be extended to include the permission to execute into the pods to unseal the OpenBao service (see below).
3	Permissions to get and list the Pods, required for the unseal process later
4	Permissions to execute into the Pods, required for the unseal process later
5	The RoleBinding is used by the Job to create the Secret.

The Job must also be able to reach the OpenBao API (e.g. openbao.openbao.svc:8200). This is usually possible when running in the same namespace as the OpenBao service.

The example below shows a working Job manifest that will perform the initialisation and the creation of the Secret in the openbao namespace. The manifest is already prepared for Argo CD with useful annotations. It runs an init container to initialise the OpenBao service and a main container to create the Secret. The init container writes (using the OpenBao CLI) the init.json file to a shared volume and the main container (using the kubectl command) creates the Secret from that file. In addition, the Secret containing the CA certificate is mounted as a volume and the BAO_CACERT environment variable is set to the path of the CA certificate.

Init Job that creates the Secret

apiVersion: batch/v1
kind: Job
metadata:
name: openbao-init
namespace: openbao
annotations: (1)
argocd.argoproj.io/sync-wave: "30"
argocd.argoproj.io/hook: PostSync
argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
template:
spec:
serviceAccountName: openbao-init (2)
initContainers: (3)
- name: openbao-init
image: ghcr.io/openbao/openbao:latest
command: (4)
- /bin/sh
- -c
- |
if [ -f /etc/openbao-ca/ca.crt ]; then
export BAO_CACERT=/etc/openbao-ca/ca.crt
export BAO_ADDR=https://openbao:8200
echo "Using TLS CA from /etc/openbao-ca/ca.crt"
else
export BAO_ADDR=http://openbao:8200
echo "Using plain HTTP"
fi
until bao status 2>&1 | grep -q "Initialized"; do
echo "Waiting for OpenBao..."
sleep 5
done
if bao status | grep -q "Initialized.*false"; then
echo "Initialising OpenBao..."
bao operator init -key-shares=5 -key-threshold=3 \
-format=json > /shared/init.json
echo "Init complete; init.json written to shared volume."
else
echo "OpenBao already initialised (skipping init)."
fi
volumeMounts:
- name: openbao-ca
mountPath: /etc/openbao-ca
readOnly: true
- name: shared
mountPath: /shared
containers: (5)
- name: create-secret
image: bitnami/kubectl:latest
command:
- /bin/sh
- -c
- |
if [ -f /shared/init.json ]; then
echo "Creating Secret openbao-init-data from init.json..."
kubectl create secret generic openbao-init-data \
--from-file=init.json=/shared/init.json \
-n openbao --dry-run=client -o yaml | kubectl apply -f -
echo "Initialisation complete!"
else
echo "No init.json (OpenBao already initialised or init skipped)."
fi
volumeMounts:
- name: shared
mountPath: /shared
readOnly: true
volumes:
- name: openbao-ca
secret:
secretName: openbao-ca-secret
optional: true
- name: shared (6)
emptyDir: {}
restartPolicy: Never
backoffLimit: 3

1	The annotations are used by Argo CD to trigger the Job after the OpenBao deployment.
2	The ServiceAccount is used by the Job to initialise the OpenBao service.
3	The init container runs the OpenBao CLI (wait for API, then operator init) and writes init.json to a shared volume.
4	The command is used to initialise the OpenBao service.
5	The main container runs kubectl to create the Secret from that file. This avoids needing a single image that bundles both `bao` and `kubectl`.
6	The shared volume is used to store the init.json file.

As a result the secret openbao-init-data is created in the openbao namespace. It contains the unseal keys and the root token.

Using the Secret in a Job (unseal)

A second Job can use the Secret openbao-init-data that was created by the init Job to unseal each OpenBao pod. A lot is happening here:

The init container extracts the unseal keys from the Secret using jq and writes them to a shared volume.
The main container is reading the unseal keys from the shared volume and unseals the first pod
Then the process is waiting for 30 seconds, to give the next pod time to pull the image and start
The process is repeated for the next pod and so on until all pods are unsealed.

First of all, the unseal Job needs permission to exec into the OpenBao pods. Add a Role that grants the Job’s service account create on pods/exec in the openbao namespace. To make it easier, we can extend the Role openbao-init-secret-writer that was created in the init Job.

These rules were already applied with the configuration for the init Job, but we will repeat them here for completeness.

rules: (1)
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"]

1	The role `openbao-init-secret-writer` must be extended to include the permission to execute into the pods to unseal the OpenBao service.

apiVersion: batch/v1
kind: Job
metadata:
name: openbao-unseal
namespace: openbao
annotations:
argocd.argoproj.io/sync-wave: "35"
argocd.argoproj.io/hook: PostSync
argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
template:
spec:
serviceAccountName: openbao-init (1)
initContainers:
- name: extract-keys
image: quay.io/codefreshplugins/curl-jq
command:
- /bin/sh
- -c
- |
if [ ! -f /secrets/init.json ]; then
echo "Secret not found; run init Job first."
exit 1
fi
for i in 0 1 2; do
jq -r ".unseal_keys_b64[$i]" /secrets/init.json | tr -d '\n' > "/shared/key$i" (2)
done
echo "Unseal keys extracted to shared volume."
volumeMounts:
- name: init-secret
mountPath: /secrets
readOnly: true
- name: shared
mountPath: /shared
containers: (3)
- name: unseal
image: bitnami/kubectl:latest
command: (4)
- /bin/sh
- -c
- |
if [ -f /etc/openbao-ca/ca.crt ]; then
export BAO_CACERT=/etc/openbao-ca/ca.crt
export BAO_ADDR=https://openbao:8200
echo "Using TLS CA from /etc/openbao-ca/ca.crt"
else
export BAO_ADDR=http://openbao:8200
echo "Using plain HTTP"
fi
sleep_timer=30
if [ ! -f /shared/key0 ]; then
echo "Keys not found; extract-keys init container may have failed."
exit 1
fi
echo "Unsealing pods openbao-0, openbao-1, openbao-2 (3 key steps each), ${sleep_timer}s delay between pods..."
for pod in openbao-0 openbao-1 openbao-2; do
echo "Unsealing $pod..."
for i in 0 1 2; do
key=$(cat "/shared/key$i" | tr -d '\n')
kubectl exec -n openbao "$pod" -- sh -c 'bao operator unseal "$1"' _ "$key" || true
done
echo "$pod unsealed."
if [ "$pod" != "openbao-2" ]; then
echo "Waiting ${sleep_timer}s for next pod to be ready..."
SLEEPER_TMP=1
while [ "$SLEEPER_TMP" -le "$sleep_timer" ]; do
if [ $(( SLEEPER_TMP % 10 )) -eq 0 ]; then
echo -n "$SLEEPER_TMP"
else
echo -n "."
fi
sleep 1
SLEEPER_TMP=$(( SLEEPER_TMP + 1 ))
done
echo ""
fi
done
echo "Unseal complete."
volumeMounts:
- name: shared
mountPath: /shared
readOnly: true
- name: openbao-ca
mountPath: /etc/openbao-ca
readOnly: true
volumes: (5)
- name: init-secret
secret:
secretName: openbao-init-data
- name: shared
emptyDir: {}
- name: openbao-ca
secret:
secretName: openbao-ca-secret
optional: true
restartPolicy: Never
backoffLimit: 2

1	The ServiceAccount is used by the Job to unseal the OpenBao pods. Here we are using the same ServiceAccount as the one used to initialise the OpenBao service.
2	The init container extracts unseal keys from init.json using jq and writes them to a shared volume.
3	The main container runs kubectl to exec into each OpenBao pod and run `bao operator unseal` (the OpenBao image is used inside the target pods).
4	The command is used to unseal the OpenBao service.
5	The volumes that are mounted: init-secret (contains the unseal keys), shared (contains the extracted unseal keys, shared between containers), openbao-ca (contains the CA certificate).

Both Jobs are using the same ServiceAccount and therefore the same RBAC rules. It is also possible to use a different ServiceAccount for the unseal Job.

Store init data securely. Keeping unseal keys in a cluster Secret is convenient but less secure than auto-unseal or an external secrets manager. Restrict access to the openbao-init-data Secret (e.g. RBAC and network policies) and consider rotating or moving keys to a more secure store after bootstrap.

Approach 3: Auto-unseal with Key Management Service KMS (recommended for production)

Configure a KMS (e.g. AWS KMS, Azure Key Vault, GCP Cloud KMS, Transit etc.) so that OpenBao unseals itself on every restart. You initialise once and thereafter no manual unseal step is needed.

The rest of this article will mention the different options. However, I can only demonstrate a real example using the AWS KMS option.

Auto-unseal options overview

Without auto-unseal, you must run bao operator unseal with a threshold of keys after each restart. With auto-unseal, OpenBao uses a seal backend to protect the root key and unseals itself on startup.

Supported options:

Option

Use case

Notes

1. AWS KMS

AWS (EKS, OpenShift on AWS)

IRSA or IAM user; no keys in cluster with IRSA.

2. Azure Key Vault

Azure (AKS, OpenShift on Azure)

Service principal or managed identity.

3. Google Cloud KMS

GCP (GKE)

Service account key or Workload Identity.

5. Transit

Existing Vault/OpenBao as root of trust; or a dedicated seal-only OpenBao that exists only to unseal production

External transit engine; token in a Secret.

Why use OpenBao when the seal is in KMS? Why not keep all secrets in KMS and skip OpenBao?

When using a cloud KMS (or Key Vault) for auto-unseal, a natural question is: why not store and retrieve all secrets from the KMS and run without OpenBao at all? OpenBao remains useful for operators and applications for several reasons:

KMS is for keys, not a full secrets platform. AWS KMS, Azure Key Vault keys, and GCP Cloud KMS are built to encrypt and decrypt small blobs (e.g. data encryption keys, or OpenBao’s seal blob). They are not a general-purpose secrets store with versioning, path-based access, dynamic credentials, and per-request audit. OpenBao provides that layer: applications request secrets by path, get short-lived tokens, and you get a single place to define who can read what.
Dynamic secrets. OpenBao can generate short-lived credentials on demand (e.g. database users, cloud IAM roles, PKI certificates). The KMS does not create or rotate such credentials; it only holds keys. If you need “give this pod a DB password that expires in 1 hour,” that is OpenBao’s domain, not the KMS.
Fine-grained, identity-aware access. OpenBao supports auth methods (Kubernetes, OIDC, AppRole, LDAP) and path-based policies. You can say “this service account may read only secret/data/myapp/*.” The KMS has IAM or Key Vault RBAC, but not the same model of “one API, many identities, many paths” that applications and operators use day to day.
Audit and compliance. OpenBao logs every secret read and write with identity and path. That gives a clear audit trail of who accessed which secret and when. KMS audit logs key usage, which is different from “which application read which secret at what time” in a unified way.
Abstraction and portability. Applications talk to OpenBao’s API. You can move the seal or the storage backend (or even the cloud) without changing how applications request secrets. If you put everything directly in a cloud KMS, you tie every app to that vendor’s API and limits.
Encryption as a service (Transit). OpenBao’s Transit secrets engine lets applications encrypt data with a key without ever seeing the key—useful for application-level encryption and key rotation. KMS can do something similar, but OpenBao integrates that with the same auth and audit as the rest of your secrets.

In short: the KMS is the root of trust for the seal (so OpenBao can unseal itself). OpenBao is the secrets management platform that applications and operators use for storing, retrieving, generating, and auditing secrets. Both layers complement each other.

General flow (same for all options):

Set up the seal backend (KMS key, Key Vault, static key, or transit server).
Add the appropriate seal "…" stanza to OpenBao configuration (e.g. in Helm server.ha.raft.config).
Deploy OpenBao; pods start and remain sealed until initialised.
Run one-time initialisation: bao operator init -recovery-shares=5 -recovery-threshold=3.
From then on, restarts are automatically unsealed.

Option: AWS KMS

Best for: EKS, OpenShift on AWS (ROSA), or any Kubernetes on AWS.

Create a KMS key

Create a symmetric KMS key used only for the OpenBao seal (do not use for application data).

AWS WebUI:
- search for: KMS → Create key → Symmetric, Encrypt/Decrypt
- Optionally set an alias (e.g. openbao-unseal).
- OPTIONAL: Define key administrative permissions
- OPTIONAL: Define key usage permissions
- Edit key policy

AWS CLI: (requires the AWS CLI to be installed and configured)

aws kms create-key \
--description "OpenBao auto-unseal key" \
--key-usage ENCRYPT_DECRYPT
aws kms create-alias \
--alias-name openbao-unseal \
--target-key-id <key-id-from-above>

Note the Key ID or Alias and the region (e.g. us-west-1) from the output of the command.

Key policy for OpenBao

The OpenBao process needs permission to use the key. Create an IAM policy. In the UI you can configure this directly when creating the key. When using the CLI, create a policy.json file with the following content:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": "arn:aws:kms:<REGION>:<ACCOUNT_ID>:key/<KEY_ID>" (1)
}
]
}

1	Replace `<REGION>`, `<ACCOUNT_ID>`, and `<KEY_ID>` with your values (or use the key ARN).

Create an IAM user for the seal

When using static credentials (e.g. on OpenShift or when IRSA is not available), create a dedicated IAM user that has permission only to use the KMS key above. Use that user’s access keys in the Kubernetes Secret.

AWS WebUI:
- search for: IAM → Users → Create user
- User name: e.g. openbao-seal
- DO NOT select "Provide user access to the AWS Management Console" (programmatic access only)
- Create user
- Open the created user → Permissions → Add permissions → Create inline policy (or Attach policies → Create policy)
- In the policy editor, choose JSON and paste the policy from Key policy for OpenBao, replacing <REGION>, <ACCOUNT_ID>, and <KEY_ID> with your KMS key ARN or alias
- Name the policy (e.g. OpenBaoSealKMS) and attach it to the user
- Security credentials → Create access key → Application running outside AWS (or "Third-party service") → Create access key
- Save the Access key ID and Secret access key; you will put them in the Kubernetes Secret openbao-aws-credentials.

AWS CLI: (requires the AWS CLI to be installed and configured)

Create a policy file (e.g. openbao-seal-policy.json) with the same JSON as in Key policy for OpenBao, then create the user, attach the policy, and create access keys:

# Replace REGION, ACCOUNT_ID, and KEY_ID in the policy (or use alias: arn:aws:kms:REGION:ACCOUNT_ID:alias/openbao-unseal)
aws iam create-policy \
--policy-name OpenBaoSealKMS \
--policy-document file://openbao-seal-policy.json
aws iam create-user --user-name openbao-seal
aws iam attach-user-policy \
--user-name openbao-seal \
--policy-arn arn:aws:iam::ACCOUNT_ID:policy/OpenBaoSealKMS
aws iam create-access-key --user-name openbao-seal

The last command returns AccessKeyId and SecretAccessKey; store them in the Secret openbao-aws-credentials with key AWS_REGION set to your region (e.g. us-west-1).

Create the Secret

Create the Secret openbao-aws-credentials in the openbao namespace. It contains the access key, secret access key, and region.

apiVersion: v1
kind: Secret
metadata:
name: openbao-aws-credentials
namespace: openbao
type: Opaque
stringData: (1)
AWS_ACCESS_KEY_ID: <your-access-key>
AWS_SECRET_ACCESS_KEY: <your-secret-key>
AWS_REGION: <your-region>

1	Replace `<your-access-key>`, `<your-secret-key>`, and `<your-region>` with the values from the previous steps.

Helm / Argo CD configuration (AWS)

Add the seal "awskms" block inside the same HCL config that contains listener, storage "raft". In Helm values this is typically under openbao.server.ha.raft.config.

Be sure to update the region and key ID.

The values file we used for Argo CD can be found at: OpenBao Argo CD values file

openbao:
server:
extraSecretEnvironmentVars: (1)
- envName: AWS_ACCESS_KEY_ID
secretName: openbao-aws-credentials
secretKey: AWS_ACCESS_KEY_ID
- envName: AWS_SECRET_ACCESS_KEY
secretName: openbao-aws-credentials
secretKey: AWS_SECRET_ACCESS_KEY
- envName: AWS_REGION
secretName: openbao-aws-credentials
secretKey: AWS_REGION
ha:
raft:
config: |
ui = true
seal "awskms" { (2)
region = "us-east-1"
kms_key_id = "alias/openbao-unseal"
}
listener "tcp" {
tls_disable = 0
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/openbao/tls/openbao-server-tls/tls.crt"
tls_key_file = "/openbao/tls/openbao-server-tls/tls.key"
tls_min_version = "tls12"
}
storage "raft" {
path = "/openbao/data"
}
service_registration "kubernetes" {}

1	If you use static credentials (e.g. for OpenShift), inject them via a Secret.
2	This must be added. Be sure to update your region and key ID.

The seal "awskms" block alone is not enough. OpenBao must have AWS credentials available at runtime. If you see an error such as NoCredentialProviders: no valid providers in chain or error fetching AWS KMS wrapping key information, the pod has no credentials. You must either inject static credentials from a Secret (see below) or use IRSA so the pod can assume an IAM role.

Approach 4: Init container with external secret store

Another option is to use an init container that retrieves secrets from an external source. Two use cases can be considered here:

Shamir unseal: The init container fetches the unseal keys from an external secret manager and then runs bao operator unseal (or writes keys to a file used by a sidecar/unseal script). This is the classic “external secret store” idea (e.g. using AWS Secrets Manager, HashiCorp Vault, another Kubernetes cluster).
Transit seal token: The init container fetches the token that production OpenBao uses to call the dedicated unseal service (see Dedicated unseal service (seal-only OpenBao)). That token is written to a file or shared volume so the main OpenBao process can use it in seal "transit". The token never lives in a Kubernetes Secret in Git or in plain YAML; it is fetched at pod start from the external store.

Below is a generic placeholder for the init container.

Option: init Container example

The following options are generic and more theoretical. Unlike the other options, I could not test them in a real environment yet. This may be covered in a future article.

Generic placeholder (implement according to your secret store):

# Additional values for the Helm chart
server:
extraInitContainers:
- name: fetch-secrets
image: registry.access.redhat.com/ubi9/ubi-minimal:latest
command:
- /bin/sh
- -c
- |
# Fetch unseal keys OR transit token from external secret manager
echo "Waiting for OpenBao to be ready..."
sleep 30
# Your fetch logic here (e.g. aws secretsmanager get-secret-value, vault kv get, curl to API) (1)
env:
- name: EXTERNAL_SECRET_ENDPOINT (2)
value: "https://your-external-secret-store.example.com"

1	Replace with your fetch logic.
2	Replace with your external secret store endpoint.

Option: Transit seal

When you already have a Vault or OpenBao cluster and want it to act as the root of trust (e.g. central Vault encrypts the seal key), you can use this option.

To prepare everything, you need to do the following:

Enable the Transit secrets engine on the external Vault/OpenBao and create a key (e.g. openbao-unseal).
Grant the token used by this OpenBao permission to encrypt/decrypt with that key.
Store the token in a Kubernetes Secret and inject via extraSecretEnvironmentVars

Do not put the token in the config file.

The following HCL configuration is used to configure the OpenBao cluster to use the transit seal.

seal "transit" {
address = "https://external-vault.example.com:8200"
token = "s.xxx"
disable_renewal = "false"
key_name = "openbao-unseal"
mount_path = "transit/"
}

See the OpenBao transit seal documentation for further details.

Dedicated unseal service (seal-only OpenBao)

A common and recommended pattern is to run a separate, standalone OpenBao instance whose only role is to provide the seal for your production OpenBao. In other words: the external OpenBao holds the unseal keys—via its Transit secrets engine—and the production OpenBao cluster uses the transit seal to auto-unseal by calling that external instance. No application secrets or other workloads run on the dedicated instance; it exists solely to unseal the production OpenBao.

Why use a dedicated unseal service?

Separation of concerns: The root of trust for unsealing lives outside the production cluster. If the production OpenBao is compromised or rebuilt, the seal keys remain in the dedicated instance.
Simpler operations: You unseal and manage only one small, locked-down OpenBao (the seal service). The production OpenBao unseals itself automatically via transit.
No cloud KMS required: On-premise or air-gapped environments can use this pattern instead of AWS KMS, Azure Key Vault, or GCP Cloud KMS.
Clear trust boundary: The dedicated instance can be hardened, network-isolated, and backed up independently.

Architecture (high level):

Dedicated unseal service: A standalone OpenBao (single node is often enough) with the Transit secrets engine enabled and a dedicated transit key (e.g. production-openbao-unseal). This instance is initialised and unsealed once; you keep its unseal keys and root token very secure. It does not store application secrets.
Production OpenBao: Your HA OpenBao cluster (e.g. in Kubernetes) is configured with seal "transit" pointing at the dedicated service URL and a token that has permission only to use that transit key. On startup, each production node asks the dedicated OpenBao to decrypt its seal blob and thus unseals automatically.

Migration from Shamir to auto-unseal

You might wonder if you can migrate from Shamir to auto-unseal. The answer is yes, you can.

If OpenBao was previously initialised with Shamir unseal keys and you want to switch to any auto-unseal backend, you can do the following:

Plan a short maintenance window. Raft HA will tolerate one node at a time.
Add the appropriate seal "…" block to the config and deploy (e.g. via Argo CD).

Do NOT re-run init.
Unseal the leader with the existing Shamir unseal keys.
Run seal migration on the leader:
```
bao operator unseal -migrate
```
Restart the leader. It should auto-unseal via the new seal.
Update and restart standby nodes. They should auto-unseal as well.

Resources

The Guide to OpenBao - GitOps Deployment with Argo CD - Part 5

Fri, 20 Feb 2026 00:00:00 +0000

Following the GitOps mantra "If it is not in Git, it does not exist", this article demonstrates how to deploy and manage OpenBao using Argo CD. This approach provides version control, audit trails, and declarative management for your secret management infrastructure.

Introduction

Deploying OpenBao via GitOps offers significant advantages:

Version Control: All configuration changes are tracked in Git
Audit Trail: Who changed what, when, and why
Declarative: Desired state is defined, not imperative commands
Reproducible: Same deployment process across environments
Self-healing: Argo CD ensures actual state matches desired state

However, there are challenges specific to secret management:

Initial unsealing requires manual intervention or automationxw
Root tokens and unseal keys must be handled carefully
Chicken-and-egg problem: How to store OpenBao secrets before OpenBao exists?

This article will focus on the deployment of the applicaiotns for the OpenBao deployment. The problem with the automatic unsealing will be addressed in the next article.

Prerequisites

Before you begin, ensure you have:

OpenShift GitOps (Argo CD) installed and configured
A Git repository for your cluster configuration
Understanding of the App-of-Apps pattern (see Configure App-of-Apps)
The official OpenBao Helm chart from Part 3

The (Wrapper) Helm Chart

The official OpenBao Helm Chart works well for deploying OpenBao itself. However, additional settings such as certificates are not covered there. Therefore, I created a (wrapper) Helm Chart that includes the official OpenBao Helm Chart, a Cert-Manager Helm Chart that I created a while ago, and allows you to add additional objects in the templates folder.

I will follow the same setup as discussed in Part 4. This means:

Create Issuer for Cert-Manager
Create CA Certificate for OpenBao
Create Certificate for OpenBao Server
Create Certificate for OpenBao Agent Injector
Install OpenBao Helm Chart

However, there is one significant issue with this approach: we need the certificate details (especially the CA certificate) before we can install the OpenBao Helm Chart, since the certificate must be referenced in the values file. This is a chicken-and-egg problem—particularly if you use self-signed CA certificates.

Since you typically create the CA certificate first and only once (or have it already), a separate Application for the CA certificate is a good approach. Once this is synchronised, the OpenBao Application can be updated and deployed.

This means we will need two Helm Charts: one for the CA certificate and one for OpenBao itself.

Helm Chart for the CA Certificate

Create a new Helm Chart for the CA certificate. This will use the Cert-Manager Helm Chart as a dependency and will create the:

Issuer for Cert-Manager
Request the CA certificate from the Issuer
Create the openbao namespace

The example I am using can be found at GitOps openbao-ca-certificate.

Please check the GitOps Repository Structure and subsequent articles for more information on how to structure your Git repository.

In the Chart.yaml file you can see two dependencies:

dependencies:
- name: tpl
version: ~1.0.0
repository: https://charts.stderr.at/
- name: cert-manager
version: ~2.0.3 (1)
repository: https://charts.stderr.at/
condition: cert-manager.enabled

1	The Cert-Manager Helm Chart version must be at least 2.0.3 to support the namespace parameter for the Issuer.

The tpl dependency is a library that contains templates for the namespace and other shared components (I keep this library in my Helm Charts repository). The cert-manager dependency is the Cert-Manager Helm Chart.

The values.yaml file below contains the configuration for the Cert-Manager:

All settings that are passed to a subchart (cert-manager) must be prefixed with the name of the subchart. In this case, cert-manager..

namespace:
create: true (1)
name: "openbao"
description: "OpenBao Namespace"
displayName: "OpenBao Namespace"
additionalLabels:
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/audit-version: latest
pod-security.kubernetes.io/warn: restricted
pod-security.kubernetes.io/warn-version: latest
cert-manager: (2)
enabled: true (3)
issuer:
# Name of issuer
- name: openbao-selfsigned (4)
# -- Syncwave to create this issuer
syncwave: 5 (5)
# -- Type can be either ClusterIssuer or Issuer
type: Issuer
# -- Enable this issuer.
# @default -- false
enabled: true
# -- Namespace for Issuer (ignored for ClusterIssuer). Defaults to the default namespace when not set.
namespace: openbao (6)
# -- Create a selfSigned issuer. The SelfSigned issuer doesn't represent a certificate authority as such, but instead denotes that certificates will "sign themselves" using a given private key.
selfSigned: true (7)
certificates: (8)
enabled: true
# List of certificates
certificate:
- name: openbao-ca
enabled: true
namespace: openbao
syncwave: "10"
secretName: openbao-ca-secret
duration: 87660h
dnsNames:
- openbao-ca
privateKey:
algorithm: ECDSA
size: 256
rotationPolicy: Always
isCA: true
# Reference to the issuer that shall be used.
issuerRef:
name: openbao-selfsigned
kind: Issuer

1	Create the openbao namespace with various settings.
2	Enable the Cert-Manager subchart. All settings below will be passed to the Cert-Manager subchart.
3	Enables Cert-Manager.
4	Name of the issuer.
5	Syncwave for this issuer; it must be lower than the syncwave of the certificate.
6	Namespace for the issuer; supported since version 2.0.3 of the Cert-Manager Helm Chart.
7	Creates a self-signed issuer.
8	The certificate section and all the settings. Verify the Part 4 for more information.

This chart creates the openbao namespace, which is required for the OpenBao deployment.

Helm Chart for the OpenBao Deployment

Create a new Helm Chart for the OpenBao deployment. This will use the official OpenBao Helm chart and the Cert-Manager Helm Chart as dependencies and will create the:

OpenBao deployment (including Route)
Issuer for the OpenBao server using the CA certificate
Required OpenBao certificates based on the CA certificate

I have added the full values.yaml file below. Please check the GitOps openbao/values.yaml to fetch the full chart.

Besides the two certificates added here in the values.yaml file, I made two further changes: fullnameOverride and nameOverride are both set to openbao. This is good practice for setting the name of the deployment when using Argo CD.

This values file contains the CA certificate in plain text. Whether that is acceptable is debatable—it is a public certificate, and here it is self-signed and used only in my demo environment.

# Full values.yaml file for the OpenBao deployment
########################################################
# Cert-Manager
########################################################
cert-manager:
enabled: true
issuer:
# Name of issuer
- name: openbao-ca-issuer (1)
# -- Syncwave to create this issuer
syncwave: '-1'
# -- Type can be either ClusterIssuer or Issuer
type: Issuer
# -- Enable this issuer.
# @default -- false
enabled: true
# -- Namespace for Issuer (ignored for ClusterIssuer). Defaults to the default namespace when not set.
namespace: openbao
ca:
secretName: openbao-ca-secret
certificates:
enabled: true
# List of certificates
certificate:
########################################################
# OpenBao Server TLS Certificate
########################################################
- name: openbao-server-tls (2)
enabled: true
namespace: openbao
syncwave: "0"
secretName: openbao-server-tls
duration: 8760h
renewBefore: 720h
dnsNames:
- openbao.openbao.svc
- openbao.apps.ocp.aws.ispworld.at # Route host (adjust to your domain)
- openbao
- openbao.openbao
- openbao.openbao.svc
- openbao.openbao.svc.cluster.local
- openbao-internal
- openbao-internal.openbao
- openbao-internal.openbao.svc
- openbao-internal.openbao.svc.cluster.local
- openbao-0.openbao-internal
- openbao-0.openbao-internal.openbao
- openbao-0.openbao-internal.openbao.svc
- openbao-0.openbao-internal.openbao.svc.cluster.local
- openbao-1.openbao-internal
- openbao-1.openbao-internal.openbao
- openbao-1.openbao-internal.openbao.svc
- openbao-2.openbao-internal
- openbao-2.openbao-internal.openbao
- openbao-2.openbao-internal.openbao.svc
ipAddresses:
- 127.0.0.1
- "::1"
# Reference to the issuer that shall be used.
issuerRef:
name: openbao-ca-issuer
kind: Issuer
########################################################
# Injector TLS Certificate
########################################################
- name: injector-certificate (3)
enabled: true
namespace: openbao
syncwave: "0"
secretName: injector-tls
duration: 24h
renewBefore: 144m
dnsNames:
- openbao-agent-injector-svc
- openbao-agent-injector-svc.openbao
- openbao-agent-injector-svc.openbao.svc
- openbao-agent-injector-svc.openbao.svc.cluster.local
# Reference to the issuer that shall be used.
issuerRef:
name: openbao-ca-issuer
kind: Issuer
########################################################
# OpenBao Deployment
########################################################
openbao: (4)
# Override the full name of the deployment via Argo CD
fullnameOverride: openbao
nameOverride: openbao
global:
# Enable OpenShift-specific settings
openshift: true
# -- The namespace to deploy to. Defaults to the `helm` installation namespace.
namespace: openbao
# Required when TLS is enabled: tells the chart to use HTTPS for readiness/liveness
# probes and for in-pod API_ADDR (127.0.0.1:8200). Otherwise you get "client sent
# an HTTP request to an HTTPS server" from the probes.
tlsDisable: false
server:
extraEnvironmentVars:
BAO_CACERT: /openbao/tls/openbao-server-tls/ca.crt
# High Availability configuration
ha:
enabled: true
replicas: 3
# Raft storage configuration
raft:
enabled: true
setNodeId: true
config: |
ui = true
listener "tcp" {
tls_disable = 0
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/openbao/tls/openbao-server-tls/tls.crt"
tls_key_file = "/openbao/tls/openbao-server-tls/tls.key"
tls_min_version = "tls12"
telemetry {
unauthenticated_metrics_access = "true"
}
}
storage "raft" {
path = "/openbao/data"
retry_join {
leader_api_addr = "https://openbao-0.openbao-internal:8200"
leader_tls_servername = "openbao-0.openbao-internal"
leader_ca_cert_file = "/openbao/tls/openbao-server-tls/ca.crt"
}
retry_join {
leader_api_addr = "https://openbao-1.openbao-internal:8200"
leader_tls_servername = "openbao-1.openbao-internal"
leader_ca_cert_file = "/openbao/tls/openbao-server-tls/ca.crt"
}
retry_join {
leader_api_addr = "https://openbao-2.openbao-internal:8200"
leader_tls_servername = "openbao-2.openbao-internal"
leader_ca_cert_file = "/openbao/tls/openbao-server-tls/ca.crt"
}
}
service_registration "kubernetes" {}
telemetry {
prometheus_retention_time = "30s"
disable_hostname = true
}
route:
enabled: true
host: openbao.apps.ocp.aws.ispworld.at
tls:
# Route terminates client TLS; backend can use reencrypt or passthrough
termination: reencrypt
insecureEdgeTerminationPolicy: Redirect
destinationCACertificate: |
-----BEGIN CERTIFICATE-----
MIIBWzCCAQCgAwIBAgIQNdbg4KIu9oi6dDClE8drmjAKBggqhkjOPQQDAjAAMB4X
DTI2MDIxODA5NDIxNFoXDTM2MDIxODIxNDIxNFowADBZMBMGByqGSM49AgEGCCqG
SM49AwEHA0IABIeYw35/kEHvyctLtOA5xMlyQNxUtXtfBbZMUfPh6AN5MFjIGuNS
cn07a3EpSpfY6/3DaPpu+4wYNFlc+/qDNYajXDBaMA4GA1UdDwEB/wQEAwICpDAP
BgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQgyRk5Vv9K0VULcCgga5mRg4O9kTAY
BgNVHREBAf8EDjAMggpvcGVuYmFvLWNhMAoGCCqGSM49BAMCA0kAMEYCIQCwQ7lZ
Q0jzUjJFzpTGkQjU2+OB159LIQMSbSQ7dz8nVQIhAIDa7f87tjQxDxbJio+/vJx2
awFaWnueGOOQpvwCcV/+
-----END CERTIFICATE-----
extraVolumes:
- type: secret
name: openbao-server-tls
path: /openbao/tls
readOnly: true
extraVolumeMounts:
- name: openbao-server-tls
mountPath: /openbao/tls
readOnly: true
# Resource requests and limits
resources:
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 1Gi
cpu: 1000m
# Persistent volume for data
dataStorage:
enabled: true
size: 10Gi
# storageClass: "gp3-csi"
# Injector configuration
injector:
enabled: true
replicas: 2 # HA for the injector too
certs:
secretName: injector-tls
# For a private CA: set caBundle to the CA cert (PEM) so the Kubernetes API server trusts the injector webhook. E.g. oc get secret openbao-ca-secret -n openbao -o jsonpath='{.data.ca\.crt}' | base64 -d
caBundle: "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJXekNDQVFDZ0F3SUJBZ0lRTmRiZzRLSXU5b2k2ZERDbEU4ZHJtakFLQmdncWhrak9QUVFEQWpBQU1CNFgKRFRJMk1ESXhPREE1TkRJeE5Gb1hEVE0yTURJeE9ESXhOREl4TkZvd0FEQlpNQk1HQnlxR1NNNDlBZ0VHQ0NxRwpTTTQ5QXdFSEEwSUFCSWVZdzM1L2tFSHZ5Y3RMdE9BNXhNbHlRTnhVdFh0ZkJiWk1VZlBoNkFONU1GaklHdU5TCmNuMDdhM0VwU3BmWTYvM0RhUHB1KzR3WU5GbGMrL3FETllhalhEQmFNQTRHQTFVZER3RUIvd1FFQXdJQ3BEQVAKQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlFneVJrNVZ2OUswVlVMY0NnZ2E1bVJnNE85a1RBWQpCZ05WSFJFQkFmOEVEakFNZ2dwdmNHVnVZbUZ2TFdOaE1Bb0dDQ3FHU000OUJBTUNBMGtBTUVZQ0lRQ3dRN2xaClEwanpVakpGenBUR2tRalUyK09CMTU5TElRTVNiU1E3ZHo4blZRSWhBSURhN2Y4N3RqUXhEeGJKaW8rL3ZKeDIKYXdGYVdudWVHT09RcHZ3Q2NWLysKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo="
certName: tls.crt
keyName: tls.key
# UI configuration
ui:
enabled: true

1	Issuer for OpenBao. The syncwave is set to '-1' to create the issuer before the certificate request and the OpenBao deployment.
2	Certificate for the OpenBao server, with all DNS names and IP addresses used to reach it. The syncwave is set to '0' so the certificate is created after the issuer.
3	Certificate for the OpenBao Agent Injector, with all DNS names used to reach it. The syncwave is set to '0' so the certificate is created after the issuer.
4	OpenBao deployment configuration; the two overrides fullnameOverride and nameOverride are both set to openbao. See Part 4 for more information.

Creating Argo CD Applications

Whilst the Helm Charts are ready, we need to create the Argo CD Applications for the CA certificate and the OpenBao deployment. If you read our blog carefully, you will notice that these Application resources are created automatically when I add something to the folder clusters/management-cluster :) This is done by leveraging ApplicationSet resources and is fully described in the GitOps Repository Structure and subsequent articles.

For the sake of simplicity, I will show the created Application resources below. The setup is the same for both; they simply target a different path in the Git repository.

# Full Application resource for the OpenBao CA Certificate
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: in-cluster-openbao-ca-certificate
namespace: openshift-gitops
spec:
destination:
name: in-cluster (1)
namespace: default (2)
info:
- name: Description
value: ApplicationSet that Deploys on Management Cluster Configuration (using Git Generator)
project: in-cluster (3)
source:
path: clusters/management-cluster/openbao-ca-certificate (4)
repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops' (5)
targetRevision: main
syncPolicy:
retry:
backoff:
duration: 5s
factor: 2
maxDuration: 3m
limit: 5
---
# Full Application resource for the OpenBao deployment
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: in-cluster-openbao
namespace: openshift-gitops
spec:
destination:
name: in-cluster (1)
namespace: openbao (2)
info:
- name: Description
value: ApplicationSet that Deploys on Management Cluster Configuration (using Git Generator)
project: in-cluster (3)
source:
path: clusters/management-cluster/openbao (6)
repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops' (5)
targetRevision: main
syncPolicy:
retry:
backoff:
duration: 5s
factor: 2
maxDuration: 3m
limit: 5

1	Target cluster; here, the local cluster
2	Namespace of the target cluster; here, the OpenBao deployment will be installed.
3	Argo CD Project (must exist)
4	Path to the Git repository for the CA certificate
5	URL of the Git repository
6	Path to the OpenBao deployment

This will create the following Applications: - in-cluster-openbao-ca-certificate - in-cluster-openbao

The ApplicationSet adds the prefix in-cluster- to each Application name so that they remain unique in Argo CD.

Figure 1. Argo CD: OpenBao Argo CD Applications

The first Application to synchronise is in-cluster-openbao-ca-certificate.

It creates the following:

Namespace
CA Issuer
CA Certificate

Then synchronise in-cluster-openbao. It creates:

OpenBao deployment
OpenBao Agent Injector
OpenBao Server TLS Certificate
OpenBao Agent Injector TLS Certificate

OpenBao is running—what next?

Deploying OpenBao via GitOps gives you version control and declarative management for your secret management infrastructure.

Whilst OpenBao is running and managed by Argo CD, the next step is to configure it: you will need to handle initialisation (for a new cluster) and unsealing (see Part 3 and Part 4). That manual approach does not scale. In the next article, I will discuss ways to automate initialisation and unsealing.

Key takeaways:

Use sync waves to control deployment order
Consider auto-unseal for production (Part 6)
Store initialisation data securely outside Git

Resources

The Guide to OpenBao - Enabling TLS on OpenShift - Part 4

Tue, 17 Feb 2026 00:00:00 +0000

In Part 3 we deployed OpenBao on OpenShift in HA mode with TLS disabled: the OpenShift Route terminates TLS at the edge, and traffic from the Route to the pods is plain HTTP. While this is ok for quick tests, for a production-ready deployment, you should consider TLS for the entire journey. This article explains why and how to enable TLS end-to-end using the cert-manager operator, what to consider, and the exact steps to achieve it.

Introduction

Part 3 uses tls_disable = 1 in the OpenBao listener and relies on the OpenShift Route for TLS. That gives encryption between the client and the Route, but not between the Route and the OpenBao pods or between pods (e.g. Raft). Enabling TLS on OpenBao itself adds encryption in transit everywhere and aligns with defense-in-depth and compliance requirements. After all, we are talking about a secrets management system here and should not compromise security.

This part assumes:

OpenBao is already deployed as in Part 3 (HA with Raft, Agent Injector enabled).
The cert-manager operator is installed and configured on your OpenShift cluster.

The article SSL Certificate Management for OpenShift describes the setup and usage of the cert-manager operator as an example.

Why Enable TLS?

Enabling TLS for OpenBao makes sense for several reasons:

Encryption in transit: Traffic between the Route and the pods, and between OpenBao peers (Raft), is encrypted. Secrets and tokens are never sent in plain text on the cluster network.
Defense in depth: Even if the Route or network is misconfigured, backend traffic remains protected.
Compliance: Many standards (e.g. PCI-DSS, SOC 2) require encryption in transit for sensitive data; TLS to the application (OpenBao) helps satisfy this.
Agent Injector: The injector webhook is called by the Kubernetes API server. Using TLS for the webhook (with a valid certificate) is required for production and when running multiple replicas.
Consistency: Using HTTPS everywhere simplifies client configuration and avoids mixing HTTP/HTTPS in the same environment.

What Must Be Considered?

Two TLS contexts: Each needs its own certificate and configuration.
1. OpenBao server (API and Raft)
2. Agent Injector (mutating webhook)
Certificate SANs: Server certificates must include all names used to reach OpenBao: Route host, internal service names (e.g. openbao.openbao.svc, openbao-0.openbao-internal.openbao.svc), 127.0.0.1 and ::1 for in-pod traffic, and any external DNS you use. The injector certificate must match the injector Service DNS name (e.g. openbao-agent-injector-svc.openbao.svc).
cert-manager: Using cert-manager gives automatic issuance and renewal. You need a ClusterIssuer (or an Issuer) in the OpenBao namespace.

In this example, we will use a self-signed CA for the OpenBao server and the Agent Injector. In a production environment, you should use a trusted CA.

OpenShift Route: With backend TLS enabled, you can keep the Route in reencrypt mode: the Route terminates TLS from the client and opens a new TLS connection to the pod. Alternatively, use passthrough if you want end-to-end TLS without re-encryption at the Route.
Raft join: After switching to TLS, retry_join and cluster addresses must use https:// and the correct hostnames. Existing unseal keys and root token are unchanged; only the transport is different.
Clients: CLI and applications must use https:// for BAO_ADDR and, if you use a private CA, BAO_CACERT (or the system trust store) so that the client trusts the server certificate.

Prerequisites

OpenBao HA deployment from Part 3 (namespace openbao, Helm release openbao).
cert-manager operator installed on OpenShift.
Sufficient rights to create Issuers, Certificates, and Secrets in the openbao namespace.

Overview of Steps

Create a Certificate Authority (CA) in the openbao namespace (or use an existing ClusterIssuer).
Issue a Certificate for the OpenBao server (API + Raft) and store it in a Secret.
Issue a Certificate for the Agent Injector and reference it in the Helm values.
Update Helm values to mount the server cert and CA, and configure the listener and Raft for TLS.
Upgrade the Helm release; initialize and unseal openbao-0 (new cluster) or re-unseal (existing).
Verify access via HTTPS and configure clients (BAO_ADDR, BAO_CACERT).

Step 1: Certificate Authority (CA) for OpenBao

If you already have a ClusterIssuer (e.g. Let’s Encrypt or an enterprise CA), you can use it for the server and injector certificates and skip this step. For a self-signed CA in the OpenBao namespace (typical for internal cluster TLS), create the following.

This self-signed CA is only for testing purposes. In a production environment, you should use a trusted CA, preferably your own.

Create a self-signed CA Issuer in the openbao namespace:

You might have your own CA or a ClusterIssuer already. You can use them, instead of creating a new one.
```
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: openbao-selfsigned
namespace: openbao (1)
spec:
selfSigned: {} (2)
```
1 The namespace where the OpenBao deployment is running.

2 The self-signed CA Issuer.

Create a self-signed CA Certificate in the openbao namespace:

This will now actually request the CA certificate from the self-signed CA Issuer. This process is fully automated by cert-manager because everything is self-signed. The requested certificate will be stored in the secret openbao-ca-secret and is available immediately.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: openbao-ca
namespace: openbao (1)
spec:
isCA: true (2)
commonName: OpenBao CA
secretName: openbao-ca-secret (3)
duration: 87660h (4)
privateKey: (5)
algorithm: ECDSA
size: 256 (6)
rotationPolicy: Always (6)
issuerRef:
name: openbao-selfsigned (7)
kind: Issuer (8)
group: cert-manager.io

1	The namespace where the OpenBao deployment is running.
2	The certificate is a CA certificate.
3	The name of the secret where the certificate and key will be stored.
4	The duration of the certificate. In this case 10 years.
5	The private key algorithm and size.
6	The rotation policy of the private key.
7	The issuer of the certificate.
8	The kind of the issuer. Can be Issuer or ClusterIssuer.

Step 2: Create an Issuer for the OpenBao Server and Agent Injector

Create an Issuer for the OpenBao Server and Agent Injector. This Issuer will reference the CA certificate and key stored in the secret openbao-ca-secret.

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: openbao-ca-issuer
namespace: openbao
spec:
ca:
secretName: openbao-ca-secret

The name of the secret where the CA certificate and key are stored.

Step 3: Certificate for the OpenBao Server

Now it is time to create the certificate for the OpenBao server. The server certificate must include every hostname used to reach OpenBao: the Route host, the headless service, and each Raft member. Adjust the dnsNames and optional uris to match your cluster and Route.

Create the following Certificate object:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: openbao-server-tls
namespace: openbao
spec:
secretName: openbao-server-tls (1)
duration: 8760h (2)
renewBefore: 720h (3)
commonName: openbao.openbao.svc (4)
dnsNames:
- openbao.apps.cluster.example.com # Route host (adjust to your domain) (5)
- openbao (6)
- openbao.openbao
- openbao.openbao.svc
- openbao.openbao.svc.cluster.local
- openbao-internal
- openbao-internal.openbao
- openbao-internal.openbao.svc
- openbao-internal.openbao.svc.cluster.local
- openbao-0.openbao-internal
- openbao-0.openbao-internal.openbao
- openbao-0.openbao-internal.openbao.svc
- openbao-0.openbao-internal.openbao.svc.cluster.local
- openbao-1.openbao-internal
- openbao-1.openbao-internal.openbao
- openbao-1.openbao-internal.openbao.svc
- openbao-2.openbao-internal
- openbao-2.openbao-internal.openbao
- openbao-2.openbao-internal.openbao.svc
ipAddresses: (7)
- 127.0.0.1
- "::1"
issuerRef:
name: openbao-ca-issuer (8)
kind: Issuer
group: cert-manager.io

1	The name of the secret where the certificate and key will be stored.
2	The duration of the certificate. In this case 1 year.
3	The duration before the certificate is renewed. In this case 30 days.
4	The common name of the certificate. This is the service name of the OpenBao server.
5	The Route host. Adjust to your domain.
6	The headless service names, all should be included.
7	Required for in-pod traffic: Readiness/liveness probes and local bao commands (e.g. raft join, operator unseal) connect to 127.0.0.1:8200. Without these IP SANs you get "tls: bad certificate" or "x509: cannot validate certificate for 127.0.0.1 because it doesn’t contain any IP SANs".
8	The issuer of the certificate, this time openbao-ca-issuer.

After a few moments the certificate will be ready and the secret will be created. The cert-manager will store the signed certificate and key in the Secret openbao-server-tls with keys tls.crt and tls.key. The Helm chart can mount this secret for the OpenBao listener.

Step 4: Certificate for the Agent Injector

The Agent Injector runs as a webhook; the Kubernetes API server calls it over TLS. The certificate must match the Service DNS name of the injector. Create a Certificate that references the same CA Issuer. In this example we use a short-lived certificate valid for 24 hours, renewed when 10% of the validity period remains.

Save as openbao-injector-cert.yaml:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: injector-certificate
namespace: openbao
spec:
secretName: injector-tls (1)
duration: 24h (2)
renewBefore: 144m
commonName: Agent Inject Cert (3)
dnsNames:
- openbao-agent-injector-svc (4)
- openbao-agent-injector-svc.openbao
- openbao-agent-injector-svc.openbao.svc
- openbao-agent-injector-svc.openbao.svc.cluster.local
issuerRef:
name: openbao-ca-issuer
kind: Issuer
group: cert-manager.io

1	The name of the secret where the certificate and key will be stored.
2	The duration of the certificate. In this case 24 hours. (renewal is done 10% before expiry)
3	The injector service name.
4	The injector service name is defined by the Helm chart. If you override the injector service name, adjust dnsNames accordingly.

Step 5: Helm Values for Server TLS

With all the certificates created, we can update the Helm values so that OpenBao uses the server certificate and listens with TLS. You need to:

Mount the Secret openbao-server-tls into the OpenBao pods.
Set the listener to use tls_cert_file and tls_key_file and disable tls_disable.
Switch Raft retry_join and cluster addresses to https://.
Add the environment variable BAO_CACERT to the OpenBao pods.

Before we start, we need to export the CA certificate from the secret openbao-ca-secret and save its value:

oc get secret openbao-ca-secret -n openbao -o jsonpath='{.data.ca\.crt}' | base64 -d

This will return the certificate like this:

-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

Save this, you will need it in the next step.

Now we need to create the Helm values file. This time we will enable TLS. Refer to Part 3 of this series to see what the initial values file looks like.

Create or update openbao-ha-values-tls.yaml (building on your Part 3 values):

global:
# Enable OpenShift-specific settings
openshift: true
# Required when TLS is enabled: tells the chart to use HTTPS for readiness/liveness
# probes and for in-pod API_ADDR (127.0.0.1:8200). Otherwise you get "client sent
# an HTTP request to an HTTPS server" from the probes.
tlsDisable: false (1)
server:
extraEnvironmentVars:
BAO_CACERT: /openbao/tls/openbao-server-tls/ca.crt (2)
# High Availability configuration
ha:
enabled: true
replicas: 3
# Raft storage configuration
raft:
enabled: true
setNodeId: true
config: |
ui = true
listener "tcp" {
tls_disable = 0 (3)
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/openbao/tls/openbao-server-tls/tls.crt" (4)
tls_key_file = "/openbao/tls/openbao-server-tls/tls.key" (5)
tls_min_version = "tls12" (6)
telemetry {
unauthenticated_metrics_access = "true"
}
}
storage "raft" {
path = "/openbao/data"
retry_join {
leader_api_addr = "https://openbao-0.openbao-internal:8200" (7)
leader_tls_servername = "openbao-0.openbao-internal"
leader_ca_cert_file = "/openbao/tls/openbao-server-tls/ca.crt" (8)
}
retry_join {
leader_api_addr = "https://openbao-1.openbao-internal:8200"
leader_tls_servername = "openbao-1.openbao-internal"
leader_ca_cert_file = "/openbao/tls/openbao-server-tls/ca.crt"
}
retry_join {
leader_api_addr = "https://openbao-2.openbao-internal:8200"
leader_tls_servername = "openbao-2.openbao-internal"
leader_ca_cert_file = "/openbao/tls/openbao-server-tls/ca.crt"
}
}
service_registration "kubernetes" {}
telemetry {
prometheus_retention_time = "30s"
disable_hostname = true
}
route:
enabled: true
host: openbao.apps.cluster.example.com
tls: (9)
# Route terminates client TLS; backend can use reencrypt or passthrough
termination: reencrypt
insecureEdgeTerminationPolicy: Redirect
destinationCACertificate: | (10)
-----BEGIN CERTIFICATE-----
# CA Certificate
-----END CERTIFICATE-----
extraVolumes: (11)
- type: secret
name: openbao-server-tls
path: /openbao/tls
readOnly: true
extraVolumeMounts: (12)
- name: openbao-server-tls
mountPath: /openbao/tls
readOnly: true
# Resource requests and limits
resources:
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 1Gi
cpu: 1000m
# Persistent volume for data
dataStorage:
enabled: true
size: 10Gi
# storageClass: "gp3-csi"
# Injector configuration
injector:
enabled: true
replicas: 2 # HA for the injector too
certs: (13)
secretName: injector-tls
# For a private CA: set caBundle to the CA cert (PEM) so the Kubernetes API server trusts the injector webhook. E.g. oc get secret openbao-ca-secret -n openbao -o jsonpath='{.data.ca\.crt}' | base64 -d
caBundle: "BASE64_ENCODED_CA_CERTIFICATE"
certName: tls.crt
keyName: tls.key
# UI configuration
ui:
enabled: true

1	global.tlsDisable: Set to false when the server listener uses TLS. This makes the chart use HTTPS for readiness/liveness probes and for the in-pod API_ADDR env var. If you leave it true (default), probes and local clients will use HTTP and you will see "client sent an HTTP request to an HTTPS server".
2	The environment variable BAO_CACERT is set to the CA certificate file path. This is helpful to execute the boa command inside the container.
3	The listener tls_disable is set to 0 to enable TLS.
4	The listener tls_cert_file is set to the certificate file path.
5	The listener tls_key_file is set to the key file path.
6	The listener tls_min_version is set to tls12.
7	The leader API address is set to the HTTPS address.
8	The leader CA certificate file is set to the CA certificate file path.
9	The Route tls termination is set to reencrypt.
10	The destination CA certificate is set to the CA certificate. Be sure not to add any extra lines or spaces.
11	The extra volumes are mounted at the /openbao/tls path.
12	The extra volume mounts are mounted at the /openbao/tls path.
13	The injector certs are set to the injector TLS secret name. The caBundle must be provided in base64 format.

The secret key names must match what cert-manager writes: tls.crt and tls.key. The OpenBao Helm chart mounts each entry in extraVolumes at path + name (e.g. with path: /openbao/tls and name: openbao-server-tls the secret is mounted at /openbao/tls/openbao-server-tls/). The listener tls_cert_file and tls_key_file must use that full path.

Be sure to change the Route host in the Helm values to the one you are using. In addition, make sure that the CA certificate is added correctly to the Route. Do not add any extra lines or spaces and do not forget the | after destinationCACertificate: and the -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- lines.

Step 6: Upgrade the Helm Release

Perform a Helm upgrade so the new volumes and configuration are applied. Pods will restart and pick up TLS; Raft will use HTTPS for join and replication.

helm upgrade openbao openbao/openbao \
--namespace openbao \
--values openbao-ha-values-tls.yaml

Watch the rollout. The first pod (openbao-0) will log "raft retry join initiated" and stay 0/1 Ready until it is initialized and unsealed (new cluster) or until it forms quorum (existing cluster).

New cluster: initialize and unseal openbao-0

openbao-0 will not become Ready until it is initialized and unsealed. Fetch the certificate, use port-forward and talk to OpenBao over HTTPS with the CA:

# 1. Save the CA cert (for BAO_CACERT)
oc get secret openbao-ca-secret -n openbao -o jsonpath='{.data.ca\.crt}' | base64 -d > openbao-ca.crt
# 2. Port-forward to openbao-0 (background)
oc port-forward openbao-0 8200:8200 -n openbao &
# 3. Use HTTPS and CA cert
export BAO_ADDR='https://127.0.0.1:8200'
export BAO_CACERT="$PWD/openbao-ca.crt"
# 4. Check status, then initialize (only once) and unseal 3 times
bao status
bao operator init -key-shares=5 -key-threshold=3 -format=json > openbao-init.json
bao operator unseal
bao operator unseal
bao operator unseal
bao status

After unsealing, openbao-0 becomes leader and goes 1/1 Ready. Then start openbao-1 and openbao-2 and join or unseal them as in Part 3 (use https://).

Perform the following steps for the other pods - join the raft cluster and unseal them (3 times):

oc exec -ti openbao-1 -- bao operator raft join https://openbao-0.openbao-internal:8200
# 3 times with 3 different unseal keys
oc exec -ti openbao-1 -- bao operator unseal
Unseal Key (will be hidden):

Repeat the same steps for openbao-2.

Existing cluster: re-unseal after restart

The existing cluster is already initialized. Re-unseal the pods (and re-join the raft cluster if necessary).

oc get pods -n openbao -w
oc exec -ti openbao-1 -- bao operator raft join https://openbao-0.openbao-internal:8200
oc exec -ti openbao-1 -- bao operator unseal # three times; same for openbao-2

Repeat the same steps for openbao-2.

Step 7: Verify and Use HTTPS from Clients

Verify server health over HTTPS (from inside the cluster):

curl -k https://openbao.apps.cluster.example.com/v1/sys/health | jq

Be sure to use the Route host in the Helm values.

From your workstation, use the Route URL with HTTPS. If the Route host is signed by your internal CA, set BAO_CACERT to the CA file:

export BAO_ADDR='https://openbao.apps.cluster.example.com'
export BAO_CACERT='/path/to/openbao-ca.crt'
bao status

Be sure to use the Route host in the Helm values.

Agent Injector: New pods that use the injector should start without webhook certificate errors. Check injector logs if you see TLS or certificate errors:

oc logs -n openbao -l app.kubernetes.io/name=openbao-agent-injector -f

Troubleshooting

MutatingWebhookConfiguration conflict (vault-k8s)

conflict occurred while applying object ... MutatingWebhookConfiguration: Apply failed with 1 conflict: conflict with "vault-k8s" using ... .webhooks[name="vault.hashicorp.com"].clientConfig.caBundle
this is a server-side apply (SSA) field ownership conflict. The OpenBao chart uses the same webhook name (vault.hashicorp.com) as the HashiCorp Vault agent injector for annotation compatibility. The clientConfig.caBundle field is still owned by a previous manager (e.g. a prior Vault Helm release or the Vault injector), so Helm cannot update it when you change the injector certificate.

Fix option 1 – Delete the webhook and re-upgrade (recommended)

Remove the MutatingWebhookConfiguration so Helm can recreate it and own all fields. There will be a short window where the injector webhook is missing (new pods requesting injection may fail until the upgrade completes).

# Replace RELEASE_NAME with your Helm release name (e.g. openbao)
RELEASE_NAME=openbao
oc delete mutatingwebhookconfiguration ${RELEASE_NAME}-agent-injector-cfg
# Re-run the upgrade
helm upgrade ${RELEASE_NAME} openbao/openbao \
--namespace openbao \
--values openbao-ha-values-tls.yaml

Fix option 2 – Force takeover of the conflicting field

If you cannot delete the webhook (e.g. in production), take over the caBundle field with server-side apply, then run the Helm upgrade again:

Export the MutatingWebhookConfiguration, set webhooks[0].clientConfig.caBundle to your injector CA (base64 PEM from openbao-ca-secret), then re-apply with --server-side --force-conflicts:

# Get the current object and the new CA bundle
oc get mutatingwebhookconfiguration openbao-agent-injector-cfg -o yaml > mwc.yaml
CA_BUNDLE=$(oc get secret openbao-ca-secret -n openbao -o jsonpath='{.data.ca\.crt}')
# Edit mwc.yaml: set .webhooks[0].clientConfig.caBundle to the value of CA_BUNDLE (no quotes in YAML).
# Then apply with force-conflicts so Helm can later manage it:
oc apply -f mwc.yaml --server-side --force-conflicts
# Re-run the Helm upgrade
helm upgrade openbao openbao/openbao --namespace openbao --values openbao-ha-values-tls.yaml

If you still have HashiCorp Vault’s agent injector installed on the same cluster, ensure only one injector is active for a given namespace (e.g. use namespace selectors) or uninstall the Vault injector to avoid two webhooks with the same name in different resources.

Route / UI – tls: bad record MAC

Pod logs show tls: bad record MAC from the router IP. The Route is likely using edge termination (HTTP to pod). Fix: Use reencrypt and set destinationCACertificate (Step 5).

oc get route openbao -n openbao -o jsonpath='{.spec.tls.termination}'

If the OpenBao UI does not load and the pod logs show:

http: TLS handshake error from 10.x.x.x:xxxxx: local error: tls: bad record MAC

the traffic is coming from the OpenShift router (the IP is typically a cluster pod IP).

Cause: The Route is likely using edge termination: the router terminates TLS at the edge and sends plain HTTP to the pod. The pod expects HTTPS, so the TLS layer receives non-TLS data and reports "bad record MAC".

Fix: Use reencrypt (or passthrough) and set destinationCACertificate so the router talks HTTPS to the pod. See Step 5 above. Quick check:

oc get route openbao -n openbao -o jsonpath='{.spec.tls.termination}'
# Must be "reencrypt" or "passthrough", not "edge"

If the output is edge, patch the Route to reencrypt and set spec.tls.destinationCACertificate to the CA PEM (Step 5).

TLS handshake / 127.0.0.1 certificate errors

If you see in the pod logs:

remote error: tls: bad certificate (from 127.0.0.1)

or when running:

oc exec -ti openbao-0 — bao operator raft join https://…;: x509: cannot validate certificate for 127.0.0.1 because it doesn’t contain any IP SANs

The server certificate does not include 127.0.0.1 (and optionally ::1) as Subject Alternative Names. Readiness/liveness probes and in-pod bao commands connect to the listener on 127.0.0.1, so the certificate must include these IP SANs.

Fix: Add ipAddresses to the OpenBao server Certificate and let cert-manager re-issue:

# Edit the Certificate (or re-apply the YAML from Step 3 with ipAddresses added)
oc edit certificate openbao-server-tls -n openbao
# Add under spec:
# ipAddresses:
# - 127.0.0.1
# - "::1"
# cert-manager will issue a new cert; wait until the secret is updated
oc get certificate openbao-server-tls -n openbao
# Restart OpenBao pods so they load the new cert
oc rollout restart statefulset/openbao -n openbao

Resources

The Guide to OpenBao - OpenShift Deployment with Helm - Part 3

Fri, 13 Feb 2026 00:00:00 +0000

After understanding standalone installation in Part 2, it is time to deploy OpenBao on OpenShift/Kubernetes using the official Helm chart. This approach provides high availability, Kubernetes-native management, and seamless integration with the OpenShift ecosystem.

Introduction

Deploying OpenBao on OpenShift/Kubernetes offers several advantages:

High Availability: Multiple replicas with automatic failover
Kubernetes-native: Managed by standard Kubernetes primitives
Persistent Storage: Data survives pod restarts via PVCs
Integration: Works with Kubernetes service accounts for authentication
Scalability: Easy to scale and manage

The official OpenBao Helm chart supports multiple deployment modes:

Dev: Single server, in-memory storage (testing only)
Standalone: Single server, persistent storage
HA: Multiple servers with Raft consensus (recommended)
External: Connect to external OpenBao cluster

Integrations

Currently, OpenBao supports the following integrations that help seamlessly load secrets into applications without the need to modify the application code:

Agent Injector: A mutating webhook that automatically injects a sidecar container that retrieves and renews secrets.
CSI Provider: A (vendor neutral) CSI driver that mounts secrets as volumes.

Prerequisites

Before deploying, ensure you have:

OpenShift 4.12+ or Kubernetes 1.30+
Helm 3.6+
oc or kubectl CLI configured
The OpenBao CLI (bao) for initialization and unsealing (see OpenBao installation)
Cluster-admin privileges (for initial setup)
A storage class that supports ReadWriteOnce PVCs

# Verify prerequisites
oc version
helm version
# Check available storage classes
oc get storageclass

Adding the Helm Repository

First, add the OpenBao Helm repository:

# Add the OpenBao Helm repository
helm repo add openbao https://openbao.github.io/openbao-helm
# Update repository cache
helm repo update
# Search for available charts
helm search repo openbao
# Expected output:
# NAME CHART VERSION APP VERSION DESCRIPTION
# openbao/openbao 0.x.x 2.x.x Official OpenBao Helm chart

According to the official documentation, the Helm chart is new and under significant development. It should always be run with --dry-run before any install or upgrade to verify changes.

Creating the Namespace

Create a dedicated namespace for OpenBao:

While I am using the oc CLI, you can also use the kubectl CLI as in-place replacement.

oc new-project openbao

Deployment Mode: High Availability (Recommended)

For production environments, deploy in HA mode with Raft. This is the recommended deployment mode for production and is straightforward to achieve on Kubernetes and OpenShift. The following values definitions will use OpenShift specific settings. I will mark them in the callouts.

We will first create a values file, deploy openbao and then discuss what must be done to activate all OpenBao pods.

Create HA Values File

Create openbao-ha-values.yaml:

The values file is based on the official values file from that chart, but only modified values or important changes are listed here.

global:
# Enable OpenShift-specific settings
openshift: true (1)
server:
# High Availability configuration
ha:
enabled: true (2)
replicas: 3 (3)
# Raft storage configuration
raft:
enabled: true (4)
setNodeId: true
config: | (5)
ui = true
listener "tcp" {
tls_disable = 1
address = "[::]:8200"
cluster_address = "[::]:8201"
telemetry {
unauthenticated_metrics_access = "true"
}
}
storage "raft" {
path = "/openbao/data"
retry_join {
leader_api_addr = "http://openbao-0.openbao-internal:8200"
}
retry_join {
leader_api_addr = "http://openbao-1.openbao-internal:8200"
}
retry_join {
leader_api_addr = "http://openbao-2.openbao-internal:8200"
}
}
service_registration "kubernetes" {}
telemetry {
prometheus_retention_time = "30s"
disable_hostname = true
}
route:
enabled: true (6)
host: openbao.apps.cluster.example.com (7)
tls:
termination: edge
# Resource requests and limits
resources:
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 1Gi
cpu: 1000m
# Persistent volume for data
dataStorage: (8)
enabled: true
size: 10Gi
# storageClass: "gp3-csi"
# Injector configuration
injector:
enabled: true
replicas: 2 # HA for the injector too (9)
# UI configuration
ui: (10)
enabled: true

1	OpenShift specific: Activate OpenShift Mode: Critical setting, if you install on OpenShift. It adjusts the Helm chart to use Routes instead of Ingress and modifies RoleBindings to work with OpenShift’s stricter authentication.
2	High Availability (HA): Deploys OpenBao as a StatefulSet rather than a Deployment.
3	Raft Consensus Quorum: Sets the cluster size to 3. Raft requires an odd number of nodes to handle leader elections and avoid split-brain scenarios. This cluster can survive the loss of 1 node.
4	Integrated Raft Storage: Enables the internal Raft storage backend, removing the need for external dependencies like Consul or Etcd.
5	Server Configuration (HCL): The actual OpenBao server configuration file. Note that tls_disable = 1 is used because the OpenShift Route handles TLS termination at the edge, passing unencrypted traffic to the pod.
6	OpenShift specific: Route: Tells Helm to create an OpenShift Route object automatically.
7	OpenShift specific: Host: The external DNS address where users and applications will access the OpenBao API and UI.
8	Persistent Storage: Allocates a 10Gi Persistent Volume Claim (PVC) for each of the 3 pods to store encrypted data and Raft logs.
9	Injector Redundancy: Runs 2 replicas of the sidecar injector. If the injector service is down, new application pods attempting to start with secrets will fail.
10	Web UI: Enables the graphical dashboard service.

Deploy HA Cluster

# Deploy with HA values
helm install openbao openbao/openbao \
--namespace openbao \
--values openbao-ha-values.yaml

Verify the Deployment

oc get pods -n openbao

This will result in the following output:

NAME READY STATUS RESTARTS AGE
openbao-0 0/1 Running 0 60s (1)
openbao-agent-injector-xxx 1/1 Running 0 60s
openbao-agent-injector-yyy 1/1 Running 0 60s

1	Only 1 openbao pod (instead of 3) is running, and the pod is not in "ready" state.

As you can see, only one OpenBao pod exists so far, and it is not in "ready" state. This is because OpenBao is not yet initialized and unsealed. Once that is done, the other pods will appear and join the cluster.

The OpenBao pods show 0/1 ready because they are sealed and need initialization.

This is also indicated in the logs of the openbao pod:

2026-02-13T14:44:52.587Z [ERROR] core: failed to get raft challenge: leader_addr=http://openbao-0.openbao-internal.openbao.svc:8200
error=
| error during raft bootstrap init call: Error making API request.
|
| URL: PUT http://openbao-0.openbao-internal.openbao.svc:8200/v1/sys/storage/raft/bootstrap/challenge
| Code: 503. Errors:
|
| * Vault is sealed (1)
2026-02-13T14:44:52.588Z [ERROR] core: failed to get raft challenge: leader_addr=http://openbao-2.openbao-internal.openbao.svc:8200 error="error during raft bootstrap init call: Put \"http://openbao-2.openbao-internal.openbao.svc:8200/v1/sys/storage/raft/bootstrap/challenge\": dial tcp: lookup openbao-2.openbao-internal.openbao.svc on 172.30.0.10:53: no such host" (2)
2026-02-13T14:44:52.589Z [ERROR] core: failed to get raft challenge: leader_addr=http://openbao-1.openbao-internal.openbao.svc:8200 error="error during raft bootstrap init call: Put \"http://openbao-1.openbao-internal.openbao.svc:8200/v1/sys/storage/raft/bootstrap/challenge\": dial tcp: lookup openbao-1.openbao-internal.openbao.svc on 172.30.0.10:53: no such host"
2026-02-13T14:44:52.589Z [ERROR] core: failed to retry join raft cluster: retry=2s err="failed to get raft challenge"

1	Vault is sealed: This means that the OpenBao service is not yet initialized and unsealed.
2	No such host: This means that the OpenBao service is not yet available.

Initializing and Unsealing OpenBao

After deployment, OpenBao needs to be initialized and unsealed. This is done on the first pod. Once this is done, the other pods will appear and can join the cluster. We will create a local portforwarding to the first pod to initialize it.

Initialize the Cluster

Create a local port forwarding to the first pod

oc port-forward openbao-0 8200:8200 -n openbao &

Set environment variable
```
export BAO_ADDR='http://127.0.0.1:8200'
```

Check status

bao status

This will result in the following output:

Key Value
--- -----
Seal Type shamir
Initialized false (1)
Sealed true (2)
Total Shares 0
Threshold 0
Unseal Progress 0/0
Unseal Nonce n/a
Version 2.5.0
Build Date 2026-02-04T16:19:33Z
Storage Type raft
HA Enabled true (3)

1	Not yet initialized
2	Vault is sealed
3	High Availability is enabled

Initialize the cluster with 5 key shares and 3 threshold
```
bao operator init -key-shares=5 -key-threshold=3 -format=json > openbao-init.json
```
This will create the file openbao-init.json with the unseal keys and root token.
```
cat openbao-init.json
```

Take care of the openbao-init.json file. It contains the unseal keys and root token!

Unseal Pod openbao-0

After initialization, we need to unseal the first pod. This is done by providing 3 different unseal keys. (threshold is 3)

# Unseal openbao-0 (3 times with different keys)
bao operator unseal # Enter first key
bao operator unseal # Enter second key
bao operator unseal # Enter third key

This unseals openbao-0, which can be verified with the command bao status.

Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false (1)
Total Shares 5
Threshold 3
Version 2.5.0
Build Date 2026-02-04T16:19:33Z
Storage Type raft
Cluster Name vault-cluster-80c01167
Cluster ID b81ecb85-9751-655a-95b7-69463dd13241
HA Enabled true
HA Cluster https://openbao-0.openbao-internal:8201
HA Mode active
Active Since 2026-02-13T14:46:13.292643992Z
Raft Committed Index 29
Raft Applied Index 29

1	Vault is unsealed

This makes openbao-0 ready and openbao-1 is trying to start:

oc get pods
NAME READY STATUS RESTARTS AGE
openbao-0 1/1 Running 0 2m10s (1)
openbao-1 0/1 Running 0 14s (2)
openbao-agent-injector-98769cf97-r4stk 1/1 Running 0 2m12s
openbao-agent-injector-98769cf97-xldgm 1/1 Running 0 2m12s

1	openbao-0 is ready
2	openbao-1 is trying to start and wants to join the Raft cluster

Activate Pod openbao-1

Since OpenBao is already initialized, we can skip the initialization step. However, we must join the Raft cluster.

oc exec -ti openbao-1 -- bao operator raft join http://openbao-0.openbao-internal:8200

Once done, we can unseal openbao-1. We need to provide 3 different unseal keys again. Execute the following command 3 times:

oc exec -ti openbao-1 -- bao operator unseal
Unseal Key (will be hidden):

Now openbao-1 is ready and openbao-2 is trying to start:

oc get pods
NAME READY STATUS RESTARTS AGE
openbao-0 1/1 Running 0 3m48s
openbao-1 1/1 Running 0 112s
openbao-2 0/1 Running 0 18s
openbao-agent-injector-98769cf97-r4stk 1/1 Running 0 3m50s
openbao-agent-injector-98769cf97-xldgm 1/1 Running 0 3m50s

Activate Pod openbao-2

Now we need to repeat the previous steps for openbao-2.

oc exec -ti openbao-2 -- bao operator raft join http://openbao-0.openbao-internal:8200
# Run 3 times (enter a different unseal key each time):
oc exec -ti openbao-2 -- bao operator unseal

Verify Raft Cluster

You can verify the Raft cluster by logging in with the root token and then checking the Raft peer list.

oc exec -ti openbao-0 -- bao login

Check the peer list:

# Check Raft peer list
oc exec -ti openbao-0 -- bao operator raft list-peers
Node Address State Voter
---- ------- ----- -----
openbao-0 openbao-0.openbao-internal:8201 leader true
openbao-1 openbao-1.openbao-internal:8201 follower true
openbao-2 openbao-2.openbao-internal:8201 follower true

Accessing the UI

Once unsealed, access the OpenBao UI:

Via Route (Production)

oc get route openbao -n openbao
# Open browser: https://openbao.apps.cluster.example.com

Figure 1. OpenBao Login Form

Upgrading OpenBao

Keep your secrets management up to date. To upgrade an existing deployment:

# Update Helm repository
helm repo update
# Check available versions
helm search repo openbao --versions
# Upgrade with your values file
helm upgrade openbao openbao/openbao \
--namespace openbao \
--values openbao-ha-values.yaml
# Watch the rolling update
oc get pods -n openbao -w

After upgrade, pods may need to be unsealed again if they restart.

Troubleshooting

Pods Not Starting

# Check pod status
oc describe pod openbao-0 -n openbao
# Check pod logs
oc logs openbao-0 -n openbao
# Check events
oc get events -n openbao --sort-by='.lastTimestamp'

PVC Issues

# Check PVC status
oc get pvc -n openbao
# If pending, check storage class
oc describe pvc data-openbao-0 -n openbao

Raft Join Failures

If pods cannot join the Raft cluster:

# Check internal DNS resolution
oc exec -it openbao-0 -n openbao -- nslookup openbao-internal
# Check connectivity between pods
oc exec -it openbao-0 -n openbao -- wget -O- http://openbao-1.openbao-internal:8200/v1/sys/health

What Should Be Considered Next?

Securely store the unseal keys and root token
Configure pod anti-affinity for true HA
Consider auto-unseal for operational ease (upcoming article)
Put everything into a GitOps pipeline

Conclusion

You now have OpenBao running on OpenShift in high-availability mode. This deployment:

Survives pod failures and restarts
Uses Raft for distributed consensus
Integrates with OpenShift security model
Is ready for production use (after unsealing automation)

Key points to remember:

Use HA mode for production
Store unseal keys securely
Configure pod anti-affinity for true HA
Consider auto-unseal for operational ease (upcoming article)

Resources

Hosted Control Planes behind a Proxy

Mon, 15 Dec 2025 00:00:00 +0000

Recently, I encountered a problem deploying a Hosted Control Plane (HCP) at a customer site. The installation started successfully—etcd came up fine—but then it just stopped. The virtual machines were created, but they never joined the cluster. No OVN or Multus pods ever started. The only meaningful message in the cluster-version-operator pod logs was:

I1204 08:22:35.473783 1 status.go:185] Synchronizing status errs=field.ErrorList(nil) status=&cvo.SyncWorkerStatus{Generation:1, Failure:(*payload.UpdateError)(0xc0006313b0), Done:575, Total:623,

This message did appear over and over again.

Problem Summary

etcd started successfully, but the installation stalled
API server was running
VMs started but never joined the cluster
No OVN or Multus pods ever started
The installation was stuck in a loop

Troubleshooting

Troubleshooting was not straightforward. The cluster-version-operator logs provided the only clue. However, the VMs were already running, so we could log in to them—and there it was, the reason for the stalled installation.

Connect to a VM

To connect to a VM, use the virtctl command. This connects you to the machine as the core user:

You can download virtctl from the OpenShift Downloads page in the OpenShift web console.

You need the SSH key for the core user.

virtctl ssh -n clusters-my-hosted-cluster core@vmi/my-node-pool-dsj7z-sss8w (1)

1	Replace my-hosted-cluster with the name of your hosted cluster and my-node-pool-dsj7z-sss8w with the name of your VM.

Verify the Problem

Once logged in, check the journalctl -xf output. In case of a proxy issue, you’ll see an error like this:

> Dec 10 06:23:09 my-node-pool2-gwvjh-rwtx8 sh[2143]: time="2025-12-10T06:23:09Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:500704de3ef374e61417cc14eda99585450c317d72f454dda0dadd5dda1ba57a: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 3.209.93.201:443: i/o timeout"

So we have a timeout when trying to pull the image from the container registry. This is a classic proxy issue. When you try curl or manual pulls you will get the same message.

Configuring the Proxy for the Hosted Control Plane

The big question is: How do we configure the proxy for the Hosted Control Plane?

This is not well documented yet—in fact, it’s barely documented at all.

The Hosted Control Plane is managed through a custom resource called HostedCluster, and that’s exactly where we configure the proxy. The upstream documentation at HyperShift Documentation explains that you can add a configuration section to the resource. Let’s do that:

Update the HostedCluster resource with the following configuration:

spec:
configuration:
proxy:
httpProxy: http://proxy.example.com:8080 (1)
httpsProxy: https://proxy.example.com:8080 (2)
noProxy: .cluster.local,.svc,10.128.0.0/14,127.0.0.1,localhost (3)
trustedCA:
name: user-ca-bundle (4)

1	HTTP proxy URL
2	HTTPS proxy URL
3	Comma-separated list of domains, IPs, or CIDRs to exclude from proxy routing
4	OPTIONAL: Name of a ConfigMap containing a custom CA certificate bundle

If you’re using a custom CA, create the ConfigMap beforehand or alongside the HostedCluster resource. The ConfigMap must contain a key named ca-bundle.crt with your CA certificate(s).

But wait there is more…

If you are using such a proxy, which is injecting it’s own certificate, then you probably saw the following: The Release Image is not available when you try to deploy a new Hosted Control Plane. The drop down in the UI is empty and you can’t select a release image.

This happens because the Pod that is trying to fetch the image is not able to connect to Github.

You can test this with the following commands:

oc rsh cluster-image-set-controller-XXXXX -n multicluster-engine (1)
curl -v https://github.com/stolostron/acm-hive-openshift-releases.git/info/refs?service=git-upload-pack (2)

1	Replace XXXXX with the name of the Pod cluster-image-set-controller in the multicluster-engine namespace
2	Try to execute the curl command to see if you can connect to Github

If this command fails with a certificate error, then you need to add the certificate to the Pod/Deployment. A Configmap called trusted-ca-bundle should already exist in the multicluster-engine namespace. If not, it must be created with the certificate chain of your Proxy (key: ca-bundle.crt)

The following command will add the ConfigMap to the deployment.

oc -n multicluster-engine set volume deployment/cluster-image-set-controller --add --type configmap --configmap-name trusted-ca-bundle --name trusted-ca-bundle --mount-path /etc/pki/tls/certs/ --overwrite

Wait a couple of minutes after the Pods has been restarted. It will try to download the release images from Github again.

You can check the logs of the Pod to see if the download was successful.

oc logs cluster-image-set-controller-XXXXX -n multicluster-engine (1)

1	Replace XXXXX with the name of the Pod cluster-image-set-controller in the multicluster-engine namespace

That should do it, you should now be able to select a release image and deploy a new Hosted Control Plane.

Figure 1. Drow Down with Release Images

Summary

That’s actually everything you need for proxy configuration with Hosted Control Planes. Hopefully, the official OpenShift documentation will be updated soon to include this information. red Hat created two tickets to track the progress of this:

Helm Charts Repository Updates

Fri, 12 Dec 2025 00:00:00 +0000

This page shows the latest updates to the stderr.at Helm Charts Repository. The charts are designed for OpenShift and Kubernetes deployments, with a focus on GitOps workflows using Argo CD.

The content below is dynamically loaded from the Helm repository and always shows the most recent changes.

Quick Links

Resource

Link

Helm Repository

https://charts.stderr.at/

GitHub Source

https://github.com/tjungbauer/helm-charts

ArtifactHub

https://artifacthub.io/packages/search?repo=tjungbauer

Example GitOps Repo

https://github.com/tjungbauer/openshift-clusterconfig-gitops

The Hitchhiker's Guide to Observability - Limit Read Access to Traces - Part 8

Sat, 06 Dec 2025 00:00:00 +0000

In the previous articles, we deployed a distributed tracing infrastructure with TempoStack and OpenTelemetry Collector. We also deployed a Grafana instance to visualize the traces. The configuration was done in a way that allows everybody to read the traces. Every system:authenticated user is able to read ALL traces. This is usually not what you want. You want to limit trace access to only the appropriate namespace.

In this article, we’ll limit the read access to traces. The users of the team-a namespace will only be able to see their own traces.

Prerequisites

Before we begin, make sure you have:

TempoStack deployed and configured (from Part 2)
Team-a namespace with traces flowing (from Part 4)
A separate user for the team-a namespace.

Verify Trace Access

Let’s verify what a user can see when they are authenticated. We have user1 who is a member of the team-a namespace. Let’s log in as this user and verify what they can see.

Navigate to Observability > Traces and select the tempostack/simplest datasource:

You should see the traces for the team-a namespace. This is fine—that’s what we want.

But now let’s change the tenant to tenantB and verify what the user can see.

As you can see, the user can see traces from the tenantB namespace, although they are a member of the team-a namespace. This is not what we want. We want to limit trace access to only the appropriate namespace.

The Original RBAC Configuration

In Part 2, we created a ClusterRoleBinding to grant read access to traces for everybody who is authenticated.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: tempostack-traces-reader
subjects:
- kind: Group
apiGroup: rbac.authorization.k8s.io
name: 'system:authenticated'
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: tempostack-traces-reader

This ClusterRoleBinding allows system:authenticated users to read all traces from all tenants and is bound to the ClusterRole tempostack-traces-reader.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tempostack-traces-reader
rules:
- verbs:
- get
apiGroups:
- tempo.grafana.com
resources:
- tenantA
- tenantB
resourceNames:
- traces

This is the configuration we want to change. We want to limit read access to traces to only the appropriate namespace.

Change the RBAC Configuration

We will change the RBAC configuration to limit read access to traces. To do so, we will first create a new ClusterRole for the team-a namespace and bind it to users of that namespace.

Create the new ClusterRole:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tempostack-traces-reader-team-a (1)
rules:
- verbs:
- get
apiGroups:
- tempo.grafana.com
resources:
- tenantA (2)
resourceNames:
- traces

1	Name of the ClusterRole, now with the prefix team-a.
2	The Tenant that is allowed to read the traces. This time it is only the tenantA tenant.

Create the new ClusterRoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: tempostack-traces-reader-team-a (1)
subjects:
- kind: User (2)
apiGroup: rbac.authorization.k8s.io
name: user1
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: tempostack-traces-reader-team-a (3)

1	Name of the ClusterRoleBinding, now with the prefix team-a.
2	The User that is allowed to read the traces. In this example: user1.
3	The ClusterRole that is allowed to read the traces.

In this example, we are using a single user. In a real-world scenario, you would most likely have a group of users. In that case, you would use a Group instead of a User.

Modify the Original ClusterRole:

As a final step, we need to modify the original ClusterRole to remove tenantB from the list of tenants that are allowed to read the traces.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tempostack-traces-reader
rules:
- verbs:
- get
apiGroups:
- tempo.grafana.com
resources:
- tenantB (1)
resourceNames:
- traces

1	Only tenantA remains. Remove tenantB from the list of resources.

This removes the permission to read traces from the tenantB namespace for the system:authenticated group.

Eventually, the original ClusterRoleBinding might be deleted once every user has been assigned to a separate ClusterRole.

Verify the Changes

Let’s see what these changes do. As user1, you should still be able to see the traces from the tenantA namespace.

But if you change the tenant to tenantB, you should see an error message like this:

The user will still see the list of tenants. Hopefully, this will be fixed in a future version of TempoStack.

The Guide to OpenBao - Standalone Installation - Part 2

Thu, 12 Feb 2026 00:00:00 +0000

In the previous article, we introduced OpenBao and its core concepts. Now it is time to get our hands dirty with a standalone installation. This approach is useful for testing, development environments, edge deployments, or scenarios where Kubernetes is not available.

Introduction

While OpenBao shines in Kubernetes environments, understanding the standalone installation helps you grasp the fundamentals. This knowledge is valuable whether you are:

Learning OpenBao before deploying to production
Running OpenBao outside of Kubernetes (edge, legacy systems)
Debugging issues in containerized deployments
Setting up a development environment

Installation Methods Overview

OpenBao can be installed through multiple methods:

Method

Best For

Complexity

Package managers (apt, dnf, brew)

Production Linux/macOS systems

Low

Container images (Podman/Docker)

Quick testing, isolated environments

Low

Binary download

Air-gapped environments

Low

Source compilation

Custom builds, development

Medium

For this article, we will focus on macOS for local testing with binary and container image (using Podman) and Red Hat Enterprise Linux to set up an example production-ready server. The deployment on the Kubernetes environment will be discussed in the next article.

Method 1: Package Manager Installation

RHEL/Fedora/CentOS

On RHEL or CentOS, before you can install OpenBao you need to install the EPEL repository. To do so, you first need to enable the Code Ready Repository. The following commands will do the trick. Be sure that your system is registered, in case of RHEL.

If you get the error "Repositories disabled by configuration." you need to tell the subscription manager that you want to manage the repositories. This can be done permanently or temporarily. You can use the command: sudo subscription-manager config --rhsm.manage_repos=1 to do so.

RHEL

Enable Code Ready Repository

sudo subscription-manager repos --enable codeready-builder-for-rhel-9-$(arch)-rpms

Install EPEL Repository

sudo dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm

Install OpenBao
```
sudo dnf install -y openbao
```

CentOS

Enable Code Ready Repository on CentOS

sudo dnf config-manager --set-enabled crb

Install EPEL Repository

sudo dnf install epel-release epel-next-release

Install OpenBao
```
sudo dnf install -y openbao
```

macOS (Homebrew)

Install OpenBao
```
brew install openbao
```

Verify Installation

After installation, verify OpenBao is available:

bao version
# Output:
OpenBao v2.5.0 (bcbb6036ec2b747bceb98c7706ce9b974faa1b23), built 2026-02-04T15:57:17Z (cgo)

The OpenBao CLI command is bao, not vault. This distinguishes it from HashiCorp Vault.

Start Development Server

To start a development environment, you can use the following command.

This is NOT suitable for production, it is just a test to evaluate the basic concepts of OpenBao.

bao server -dev -dev-root-token-id="dev-only-token"

This will start the server. The UI is accessible at http://localhost:8200 where you can login using the root token dev-only-token.

Method 2: Container Image

For quick testing or isolated environments, container images are ideal. Luckily, OpenBao offers several types of containers suitable for any environment. We will use the image hosted on quay.io, which is based on RHEL UBI and can be found at: quay.io/openbao/openbao-ubi

Using Podman

The following command will fetch the image and start the container.

This is NOT suitable for production, it is just a test to evaluate the basic concepts of OpenBao.

# Run in dev mode (for testing only!)
podman run --rm -d \
--name openbao-dev \
-p 8200:8200 \
-e 'BAO_DEV_ROOT_TOKEN_ID=dev-only-token' \
-e 'BAO_DEV_LISTEN_ADDRESS=0.0.0.0:8200' \
quay.io/openbao/openbao-ubi:latest server -dev

# Verify it is running
podman logs -f openbao-dev

This will start the server. The UI is accessible at http://localhost:8200 where you can login using the root token dev-only-token.

If you prefer to use Docker, simply replace podman with docker in the commands.

Dev Mode: Quick Start for Testing

Dev mode is the fastest way to start using OpenBao for learning and testing.

The characteristics in this mode are:

In-memory storage (data lost on restart)
Automatically initialized and unsealed
Root token printed to stdout
TLS disabled
Single server (no HA)

Let’s create an example secret and try to retrieve it again.

Authenticate against OpenBao

export VAULT_TOKEN="dev-only-token"
export BAO_ADDR='http://127.0.0.1:8200'

Create Secret

curl \
--header "X-Vault-Token: $VAULT_TOKEN" \
--header "Content-Type: application/json" \
--request POST \
--data '{"data": {"password": "OpenBao123"}}' \
$BAO_ADDR/v1/secret/data/my-secret-password &&
echo "Secret written successfully."

Retrieve Secret

curl --header "X-Vault-Token: $VAULT_TOKEN" \
$BAO_ADDR/v1/secret/data/my-secret-password | jq '.data.data'

You should see the password that was created before:

{
"password": "OpenBao123"
}

Check Status of OpenBao Server

bao status

This will give you the status of your running OpenBao instance.

Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 2.5.0
Build Date 2026-02-04T15:57:17Z
Storage Type inmem
Cluster Name vault-cluster-421b2431
Cluster ID 6d42dcd2-e399-211e-999a-49b1874cc8ce
HA Enabled false

Hardening your System

OpenBao does a good job to secure your secrets, however, memory paging (or swap) can undermine the protection. Your OS should either have swap disabled completely or encrypt the swap space.

As I am testing on macOS, the swap space is encrypted out of the box.

However, OpenBao has documented what must be done for various operating systems at Post-installation hardening

Production Standalone Setup

For a proper standalone installation, follow these steps:

Step 1: Create Configuration File

Create /etc/openbao.d/openbao.hcl:

# Full configuration for standalone OpenBao server
# Cluster name for identification
cluster_name = "openbao-standalone"
# Storage backend using integrated Raft
storage "raft" {
path = "/var/lib/openbao/data"
node_id = "node1"
}
# HTTP listener (for internal communication)
listener "tcp" {
address = "0.0.0.0:8200"
cluster_address = "0.0.0.0:8201"
tls_disable = false
tls_cert_file = "/etc/openbao.d/tls/tls.crt"
tls_key_file = "/etc/openbao.d/tls/tls.key"
}
# API address for clients
api_addr = "https://openbao.example.com:8200"
# Cluster address for raft communication
cluster_addr = "https://openbao.example.com:8201"
# UI enabled
ui = true
# Logging
log_level = "info"
log_file = "/var/log/openbao/openbao.log"
# Disable memory locking (enable in production if possible)
disable_mlock = true
# Telemetry (optional)
telemetry {
prometheus_retention_time = "30s"
disable_hostname = true
}

Step 2: Generate TLS Certificates

For production, you should use proper certificates. For testing, create self-signed ones:

This is only for testing purposes. In production, you should use proper certificates.

First, prepare the TLS directory if it does not exist
```
sudo mkdir -p /etc/openbao.d/tls
```

Then create a configuration file for the TLS certificates:

# Create TLS config
cat <<EOF > openbao.cnf
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no
[req_distinguished_name]
CN = openbao.example.com
[v3_req]
subjectAltName = @alt_names
[alt_names]
DNS.1 = openbao.example.com
IP.1 = 127.0.0.1 (1)
EOF

1	The IP address of the server. In this case we are using localhost, but you can add your IPs here.

Then generate the certificate:

# Generate private key
sudo openssl genrsa -out /etc/openbao.d/tls/tls.key 4096
# Generate certificate signing request
sudo openssl req -new -key /etc/openbao.d/tls/tls.key -out /etc/openbao.d/tls/tls.csr -subj "/CN=openbao.example.com"
# Generate self-signed certificate
sudo openssl x509 -req -days 365 -in /etc/openbao.d/tls/tls.csr -signkey /etc/openbao.d/tls/tls.key -out /etc/openbao.d/tls/tls.crt -extfile openbao.cnf -extensions v3_req
# Set permissions
sudo chown -R openbao:openbao /etc/openbao.d/tls
sudo chmod 600 /etc/openbao.d/tls/tls.key
sudo chmod 755 /etc/openbao.d/tls

Step 3: Create Systemd Service

Create /etc/systemd/system/openbao.service:

[Unit]
Description=OpenBao Secret Management
Documentation=https://openbao.org/docs
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/openbao.d/openbao.hcl
[Service]
User=openbao
Group=openbao
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/bin/bao server -config=/etc/openbao.d/openbao.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
LimitNOFILE=65536
LimitMEMLOCK=infinity
[Install]
WantedBy=multi-user.target

Step 4: Start the Service

# Reload systemd
sudo systemctl daemon-reload
# Enable and start OpenBao
sudo systemctl enable openbao
sudo systemctl start openbao
# Check status
sudo systemctl status openbao
# View logs
sudo journalctl -u openbao -f

Initialize and Unseal OpenBao

After starting OpenBao for the first time, it needs to be initialized and unsealed.

Set Environment Variables

We will do the following commands as root user.

Be sure that the hostname is resolvable. In this case I am using the test domain: openbao.example.com.

# Set the OpenBao address
export BAO_ADDR='https://openbao.example.com:8200'
# If using self-signed certificates
export BAO_CACERT='/etc/openbao.d/tls/tls.crt'

Check Status

You will see that OpenBao is running but not yet initialized. This will be the next step.

bao status
Key Value
--- -----
Seal Type shamir
Initialized false (1)
Sealed true
Total Shares 0
Threshold 0
Unseal Progress 0/0
Unseal Nonce n/a
Version 2.4.4-1.el9
Build Date 2025-11-24
Storage Type file
HA Enabled false

1	Not yet initialized

Initialize OpenBao

Before we use OpenBao, we need to initialize it. This will create the unseal keys and the root token. The unseal keys are used to … well, unseal the OpenBao service. The root token is used as a master key to authenticate against the OpenBao service. This token has access to ALL secrets. Treat this root token with care.

We will initialize OpenBao with default options of 5 key shares and a threshold of 3. This means that we need 3 different unseal keys to unseal the OpenBao service.

bao operator init -key-shares=5 -key-threshold=3 -format=json > /root/openbao-init.json

Store the unseal keys and root token securely! Anyone with these can access all secrets.

The output looks like:

{
"unseal_keys_b64": [
"key1...",
"key2...",
"key3...",
"key4...",
"key5..."
],
"unseal_keys_hex": [...],
"unseal_shares": 5,
"unseal_threshold": 3,
"recovery_keys_b64": [],
"recovery_keys_hex": [],
"root_token": "sbr.xxxxxxxxxxxx"
}

Unseal OpenBao

Now everything is set up and we can use the unseal keys to unseal the OpenBao service.

You need to provide 3 (threshold) different unseal keys:

# First key
bao operator unseal
# Enter first unseal key
# Second key
bao operator unseal
# Enter second unseal key
# Third key
bao operator unseal
# Enter third unseal key

After providing enough keys:

bao status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false (1)
Total Shares 5
Threshold 3
Version 2.4.4-1.el9
Build Date 2025-11-24
Storage Type file
Cluster Name openbao-standalone
Cluster ID d07874ef-df52-e45f-1723-ade333c6d609
HA Enabled false

1	Now unsealed and ready

Now we can login with the root token to the OpenBao service.

Login with Root Token

Since we have not created any users yet, we will use the root token to authenticate.

# Enter the root token from initialization
bao login

Basic Verification

Let us verify the installation works:

List enabled secrets engines

bao secrets list

This lists the enabled secrets engines.

Path Type Accessor Description
---- ---- -------- -----------
cubbyhole/ cubbyhole cubbyhole_84c3d2a5 per-token private secret storage
identity/ identity identity_61809171 identity store
sys/ system system_034ae772 system endpoints used for control, policy and debugging

List enabled auth methods

bao auth list

This lists the enabled authentication methods.

Path Type Accessor Description Version
---- ---- -------- ----------- -------
token/ token auth_token_c7feca17 token based credentials n/a

Create a test secret
```
bao secrets enable -path=secret kv-v2
```
This enables the KV (key/value) secrets engine at the path secret.
```
bao kv put secret/test message="Hello from OpenBao"
```
This creates a test secret at the path secret/test.
```
bao kv get secret/test
```
This retrieves the test secret from the path secret/test.

That’s it for the basic verification. You can now start to use OpenBao in your production environment. Let’s see what we can do in the UI.

Visiting the UI

If you watched the previous commands closely, you will have noticed that the UI has been enabled in the configuration file:

# UI enabled
ui = true

The UI is accessible at http://openbao.example.com:8200. You can login with the root token from the initialization.

Figure 1. OpenBao Login Form

Here you will see several options to explore. For now we are interested in the Secrets Engine section.

Figure 2. OpenBao Secrets Engine

We have created the path "secret" and inside it a secret called "test". We can retrieve it now:

Figure 3. OpenBao Secret Retrieval

Security Hardening Checklist

Before using OpenBao in production you should always consider the following checklist:

Item

Recommendation

TLS

Always enable TLS with valid certificates

Root Token

Revoke root token after initial setup and create admin users instead

Unseal Keys

Distribute to different people/locations, consider auto-unseal

Network

Restrict access with firewall rules

Audit

Enable audit logging

Backups

Regular Raft snapshots

Updates

Keep OpenBao updated

What is Coming Next?

In Part 3, we will deploy OpenBao on OpenShift/Kubernetes using the official Helm chart. This provides:

High availability out of the box
Kubernetes-native management
Integration with OpenShift security features
Persistent storage via PVCs

Conclusion

You now have a working standalone OpenBao installation. This forms the foundation for understanding how OpenBao operates. While standalone mode is useful for testing and edge cases, most production deployments will use Kubernetes, which we will cover next.

Key takeaways:

OpenBao can run standalone or in containers
Dev mode is for testing only
Production requires proper TLS, initialization, and security hardening
Unseal keys must be stored securely

Resources

[Ep.15] OpenShift GitOps - Argo CD Agent

Wed, 14 Jan 2026 00:00:00 +0000

OpenShift GitOps based on Argo CD is a powerful tool to manage the infrastructure and applications on an OpenShift cluster. Initially, there were two ways of deployment: centralized and decentralized (or distributed). Both methods had their own advantages and disadvantages. The choice was mainly between scalability and centralization. With OpenShift GitOps v1.19, the Argo CD Agent was finally generally available. This agent tries to solve this problem by bringing the best of both worlds together. In this quite long article, I will show you how to install and configure the Argo CD Agent with OpenShift GitOps using hub and spoke architecture.

Classic Deployment Models:

Prior to the Argo CD Agent, there were two classic and often used deployment models: centralized and decentralized.

Centralized Model

In a centralized deployment, all changes are applied to a single, central Argo CD instance, often installed on a management cluster. This is the traditional way of deploying Argo CD, at least in my opinion. With this model, you have a single pane of glass to manage all your clusters. You have a single UI and will see all your clusters in one place, which makes this model very convenient when you have multiple clusters. However, the scalability of this model is limited. Organizations with a huge number of clusters or Argo CD applications would hit some boundaries at some point. A sharding configuration would help, but only to a certain extent. The performance would degrade significantly. In addition, this model creates a Single Point of Failure. If this instance is down, the company loses the ability to manage their clusters through Argo CD.

I often saw or used this model for the cluster configuration.

Decentralized Model

In a decentralized deployment, multiple instances of Argo CD, often one for each cluster, are installed. With this approach, the issue with scalability is solved. Moreover, the Single Point of Failure is eliminated as well, since a broken instance will not affect the other instances. However, the disadvantages of this model are that the complexity for the operational teams will increase, since they need to manage multiple instances now. Also, the single pane of glass for management is lost, as there are multiple UIs to manage.

I often saw or used this model for the application deployment.

The - not so secret - Argo CD Agent

The Argo CD Agent, released as generally available in OpenShift GitOps v1.19, is a new way to use Argo CD. It tries to solve the challenges of the classic deployment models by combining the best of both worlds. The Agent allows you to have a single UI in a central control plane, while the application controller is distributed across the fleet of clusters. Agents on the different clusters will communicate with the central Argo CD instance.

The Agent model introduces a hub and spoke architecture:

Control plane cluster (hub) - The control plane cluster is the central cluster that manages the configuration for multiple spokes.
Workload cluster (spoke) - The workload cluster is the cluster that runs the application workloads deployed by Argo CD.

Each Argo CD Agent on a cluster manages the local Argo CD instance and ensures that applications, AppProjects, and secrets remain synchronized with their source of truth.

The official documentation describes a comparison between the classic deployment models and the Argo CD Agent: GitOps Architecture - Argo CD Agent Comparison

Argo CD Agent Modes

The Argo CD Agent supports two modes of operation: Managed and Autonomous. The mode determines where the authoritative source of truth for the Application .spec field resides.

Managed mode — the control plane/hub defines Argo CD applications and their specifications.
Autonomous mode — each workload cluster/spoke defines its own Argo CD applications and their specifications.

A mixed mode is also possible.

Managed Mode

Using the managed mode means that the control plane is the source of truth and is responsible for the Argo CD application resources and their distribution across the different workload clusters. Any change on the hub cluster will be propagated to the spoke clusters. Any changes made on the spoke/workload cluster will be reverted to match the control plane configuration.

Autonomous Mode

Using this mode, the Argo CD applications are defined on the workload clusters, which serve as their own source of truth. The applications are synchronized back to the control plane for observability. Changes made on the workload cluster are not reverted, but will appear on the control plane. On the other hand, you cannot modify applications directly from the control plane.

Security

The Argo CD Agent uses mTLS certificates to communicate between the hub and the spoke clusters.

The certificate must be created and managed by the user.

Argo CD Agent Installation

The agent consists of two components that are responsible to synchronize the Argo CD applications between the hub and the spoke clusters.

Principal - Deployed on the control plane cluster together with Argo CD. Here the central UI (and API) can be found.
Agent - Deployed on the workload clusters to synchronize the Argo CD applications.

The installation of both is done differently. But before we dive into the installation, let’s have a look at the terminology to understand the different components and their roles.

This is a quote from the official documentation (Argo CD Agent Terminologies)

Principal namespace - Specifies the namespace where you install the Principal component. This namespace is not created by default, you must create it before adding the resources in this namespace. In Argo CD Agent CLI commands, this value is provided using the --principal-namespace flag.
Agent namespace - Specifies the namespace hosting the Agent component. This namespace is not created by default, you must create it before adding the resources in this namespace. In Argo CD Agent CLI commands, this value is provided using the --agent-namespace flag.
Context - A context refers to a named configuration in the oc CLI that allows you to switch between different clusters. You must be logged in to all clusters and assign distinct context names for the hub and spoke clusters. Examples for cluster names include principal-cluster, hub-cluster, managed-agent-cluster, or autonomous-agent-cluster.
Principal context - The context name you provide for the hub (control plane) cluster. For example, if you log in to the hub cluster and rename its context to principal-cluster, you specify it in Argo CD Agent CLI commands as --principal-context principal-cluster.
Agent context - The context name you provide for the spoke (workload) cluster. For example, if you log in to a spoke cluster and rename its context to autonomous-agent-cluster, you specify it in Argo CD Agent CLI commands as --agent-context autonomous-agent-cluster.

A Word about the Setup

To create some kind of real customer scenario, I have created two clusters:

The cluster where we installed the Principal component. This will be the Hub/Management/Principal cluster (We have too many words for this…)
A separate cluster that will be the Agent cluster. This will be the Spoke/Workload cluster.

The first one I have installed on AWS, and the second one is a Bare Metal Single node cluster. Both can reach each other via the Internet.

We are using different contexts for the different clusters for the command line. The argocd-agentctl tool knows the flags --principal-context and --agent-context to switch between the different clusters. Be sure to create the resources on the correct cluster.

Prerequisites

Before we start with the installation of the Principal or Agent component, we need to ensure that the following prerequisites are met:

We have two OpenShift test clusters.
Both clusters can reach each other
OpenShift GitOps Operator is already installed (possible configuration modifications are described in the following sections)

Because of my main focus on OpenShift GitOps, we will try to deploy cluster configurations and not just workload. Therefore, the example Argo CD applications will configure cluster settings. (A banner on the top and bottom of the UI)

Configure OpenShift GitOps Subscription on the Hub Cluster

The OpenShift GitOps Operator installs by default an Argo CD instance. In this test we will disable this, as we do not need that instance. Moreover and even more important, we need to tell the Operator for which namespaces it should feel responsible for. In this case, we will tell the Operator to be responsible for all namespaces on the cluster.

We need to modify the Subscription openshift-gitops-operator and add the following environment variables:

spec:
config:
env:
- name: DISABLE_DEFAULT_ARGOCD_INSTANCE (1)
value: 'true'
- name: ARGOCD_CLUSTER_CONFIG_NAMESPACES (2)
value: '*'

1	Optional: Disable the default Argo CD instance.
2	Tell the Operator for which namespaces it should feel responsible for. In this case, all Namespaces. This is important for a namespace-scoped Argo CD instance, which we will install in the next step.

Activate the Principal Component

To activate the Principal components we first need a cluster (the Hub) where OpenShift GitOps is installed. On this cluster, there might be a running instance of Argo CD already. However, at this time an existing Argo CD cannot be used. Instead, a new Argo CD instance must be created. (This is because the controller must not be activated)

In my case, I will install the Principal component in the namespace argocd-principal.

To create an Argo CD instance we need to create the following configuration:

At this very stage, the principal component must be installed in a separate Argo CD instance, since the controller must not be activated. Therefore we create a new Argo CD instance in a new namespace.

apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
name: hub-argocd
namespace: argocd-principal
spec:
controller:
enabled: false (1)
argoCDAgent:
principal:
enabled: true (2)
auth: "mtls:CN=([^,]+)" (3)
logLevel: "info"
namespace:
allowedNamespaces: (4)
- "*"
tls:
insecureGenerate: false (5)
jwt:
insecureGenerate: false
sourceNamespaces: (6)
- "argocd-agent-bm01"
server:
route:
enabled: true (7)

1	Disable the controller component.
2	Enable the Principal component.
3	Authentication method for the Principal component.
4	Allowed namespaces for the Principal component.
5	Insecure generation of the TLS certificate.
6	Specifies the sourceNamespaces configuration. (Such a list might already exist)
7	Enable the Route for the Principal component.

This will start a Pod called hub-gitops-agent-principal in the namespace argocd-principal. However, this pod will fail at this moment and that is fine.

Pod hub-gitops-agent-principal is failing with:
{"level":"info","msg":"Setting loglevel to info","time":"2026-01-19T13:45:57Z"}
time="2026-01-19T13:45:57Z" level=info msg="Loading gRPC TLS certificate from secret argocd-principal/argocd-agent-principal-tls"
time="2026-01-19T13:45:57Z" level=info msg="Loading root CA certificate from secret argocd-principal/argocd-agent-ca"
time="2026-01-19T13:45:57Z" level=info msg="Loading resource proxy TLS certificate from secrets argocd-principal/argocd-agent-resource-proxy-tls and argocd-principal/argocd-agent-ca"
[FATAL]: Error reading TLS config for resource proxy: error getting proxy certificate: could not read TLS secret argocd-principal/argocd-agent-resource-proxy-tls: secrets "argocd-agent-resource-proxy-tls" not found (1)

1	The secret is not yet available.

The Pod is failing at the moment because the different secrets for authentication, are not yet available. The Secrets are created in a later step, because some settings, such as the principal hostname and resource proxy service names, are available only after the Red Hat OpenShift GitOps Operator enables the principal component.

At this point the Operator created the Route object already:

Route: hub-gitops-agent-principal
Hostname: https://hub-gitops-agent-principal-argocd-principal.apps.ocp.aws.ispworld.at
Service: hub-gitops-agent-principal

Configure the AppProject

If you configured the AppProject with sourceNamespaces, you need to add the following to the AppProject (for example to the default AppProject). This must match exactly the namespaces you have created for the Agent.

spec:
sourceNamespaces:
- "argocd-agent-bm01"

You can also use this patch command:

oc patch appproject default -n argocd-principal --type='merge' \
-p '{"spec": {"sourceNamespaces": ["argocd-agent-bm01"]}}' --context aws

Restart the Argo CD Pods to apply the changes.

Download argocd-agentctl

To create the required secrets, we need to download the argocd-agentctl tool.

This can be found at: https://developers.redhat.com/content-gateway/rest/browse/pub/cgw/openshift-gitops/

Download and install it for your platform.

Create Required Secrets

The following steps will create the required secrets for the Principal component. In this example, we will create our own CA and certificates. This is suitable for development and testing purposes. For production environments, you should use certificates issued by your organization’s PKI or a trusted certificate authority.

Use your company’s CA and certificates for production environments.

Initialize the Certificate Authority (CA)

To create a certificate authority (CA) that signs other certificates, we need to run the following command:

argocd-agentctl pki init \
--principal-namespace argocd-principal \ (1)
--principal-context aws

1	The namespace where the Principal component is running.

This will initialize the CA and store it in the secret argocd-principal/argocd-agent-ca. The certificate looks like:

"Certificate Information:
Common Name: argocd-agent-ca
Subject Alternative Names:
Organization: DO NOT USE IN PRODUCTION
Organization Unit:
Locality:
State:
Country:
Valid From: January 15, 2026
Valid To: January 15, 2036
Issuer: argocd-agent-ca, DO NOT USE IN PRODUCTION
Serial Number: 1 (0x1)"

Generate Service Certificate for the Principal

To generate the server certificate for the Principal’s gRPC service, run the following command:

argocd-agentctl pki issue principal \
--principal-namespace argocd-principal \
--principal-context aws \
--dns "<YOUR PRINCIPAL HOSTNAME>" (1)

1	The hostname of the Principal service. This must match with the hostname of the Principal’s route (spec.host) or, in case a LoadBalancer Service is used, with .status.loadBalancer.ingress.hostname.

Generate the Resource Proxy Certificate

The resource proxy service requires a certificate as well. Since the proxy will run on the same cluster as the Principal, we can use the service name directly. This is generated by the following command:

argocd-agentctl pki issue resource-proxy \
--principal-namespace argocd-principal \
--principal-context aws \
--dns hub-argocd-agent-principal-resource-proxy (1)

1	The service name for the resource-proxy. This must match with the service name of the Resource Proxy service.

Generate the JWT Signing Key

Generate the RSA private key for the JWT signing key by running the following command:

argocd-agentctl jwt create-key \
--principal-namespace argocd-principal \
--principal-context aws

This will generate the RSA private key and store it in the secret argocd-principal/argocd-agent-jwt.

Verify the Principal Component

Now the principal pod should be running successfully.

If the Pod still shows an error, wait a few moments or restart the Pod.

In the logs you should see the following:

{"level":"info","msg":"Setting loglevel to info","time":"2026-01-19T14:09:31Z"}
time="2026-01-19T14:09:31Z" level=info msg="Loading gRPC TLS certificate from secret argocd-principal/argocd-agent-principal-tls"
time="2026-01-19T14:09:31Z" level=info msg="Loading root CA certificate from secret argocd-principal/argocd-agent-ca"
time="2026-01-19T14:09:31Z" level=info msg="Loading resource proxy TLS certificate from secrets argocd-principal/argocd-agent-resource-proxy-tls and argocd-principal/argocd-agent-ca"
time="2026-01-19T14:09:31Z" level=info msg="Loading JWT signing key from secret argocd-principal/argocd-agent-jwt"
time="2026-01-19T14:09:31Z" level=info msg="Starting argocd-agent (server) v99.9.9-unreleased (ns=argocd-principal, allowed_namespaces=[*])" module=server

This concludes the configuration of the Principal component. There are a lot of steps to create the required secrets, but this is only done once. A GitOps-friendly way to achieve this might be done using a Kubernetes Job (if you consider this as GitOps-friendly… which I do).

Activate the Agent Component

After the Principal component is configured, you can activate one or more Agents (spoke or workload clusters) and connect them with the Hub.

The prerequisites are:

The Principal component is configured and running.
You have access to both the Principal and Agent clusters.
The argocd-agentctl CLI tool is installed and accessible from your environment.
The helm CLI is installed and configured. Ensure that the helm CLI version is later than v3.8.0.
OpenShift GitOps Operator is installed and configured on the Agent cluster.

Yes, a separate Helm Chart will be used to install the Agent component on the target cluster. This time it is not a Chart that I created, but one provided by Red Hat. :)

Create Agent Secret on Principal Cluster

We first need to create an agent on the Principal cluster.

argocd-agentctl agent create "argocd-agent-bm01" \ (1)
--principal-context "aws" \ (2)
--principal-namespace "argocd-principal" \ (3)
--resource-proxy-server "hub-argocd-agent-principal-resource-proxy:9090" (4)

1	A (unique) name for the Agent.
2	The context name for the Principal cluster. In my case it is "aws".
3	The namespace where the Principal component is running.
4	The resource proxy server URL. This is the URL of the Principal’s resource proxy service including the port (9090).

This will create the secret cluster-argocd-agent-bm01 with the label argocd.argoproj.io/secret-type: cluster in the Argo CD namespace.

Create the Agent Namespace on the Agent Cluster

Be sure that the target namespace on the agent or workload cluster exists. If not, create it first.

oc create namespace argocd-agent-bm01 --context bm (1)

1	The name of the namespace.

Propagate the Principal CA to the Agent Cluster

To copy the CA certificate from the principal to the agent cluster the following command is used:

argocd-agentctl pki propagate \
--agent-context bm \ (1)
--principal-context aws \ (2)
--principal-namespace argocd-principal \ (3)
--agent-namespace argocd-agent-bm01 (4)

1	The context name for the Agent cluster.
2	The context name for the Principal cluster.
3	The namespace where the Principal component is running.
4	The namespace where the Agent component is running.

This will copy the CA certificate from the Principal cluster to the Agent cluster into the namespace and secret argocd-agent-bm01/argocd-agent-ca.

Only the certificate is copied. The private key is not copied.

Generate a client certificate for the Agent

Now we need to create, based on the imported CA, a client certificate for the Agent. This is done by the following command:

argocd-agentctl pki issue agent "argocd-agent-bm01" \
--principal-context "aws" \
--agent-context "bm" \
--agent-namespace "argocd-agent-bm01" \
--principal-namespace "argocd-principal"

This will create the secret argocd-agent-client-tls on the workload cluster, containing a certificate and a key, signed by the CA certificate imported from the Principal cluster.

Configure OpenShift GitOps Subscription on the Spoke Cluster

We need to modify the Subscription openshift-gitops-operator and add the following environment variables:

spec:
config:
env:
- name: DISABLE_DEFAULT_ARGOCD_INSTANCE (1)
value: 'true'
- name: ARGOCD_CLUSTER_CONFIG_NAMESPACES (2)
value: '*'

1	Disable the default Argo CD instance.
2	Tell the Operator for which namespaces it should feel responsible for. In this case, all Namespaces.

Create Argo CD Instance on the Agent Cluster

To create a minimalistic Argo CD instance on the Agent cluster, we can use the following Argo CD configuration:

apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
name: agent-argocd
namespace: argocd-agent-bm01 (1)
spec:
server:
enabled: false

1	The namespace where the Argo CD instance is running. This is also the name of the Agent we have created earlier on the principal cluster.

This will create a lightweight Argo CD instance in the namespace argocd-agent-bm01.

oc get pod -n argocd-agent-bm01 --context bm
NAME READY STATUS RESTARTS AGE
agent-argocd-application-controller-0 1/1 Running 0 2m49s
agent-argocd-redis-5f6759f6fb-2fdnt 1/1 Running 0 2m49s
agent-argocd-repo-server-7949d97dfd-dsk6b 1/1 Running 0 2m49s

Installing the Agent

To install the agent we will use a Helm Chart provided by Red Hat. This will install the Agent component on the target cluster.

As a reminder, we have two modes of operation for the Agent:

Managed mode — the control plane/hub defines Argo CD applications and their specifications.
Autonomous mode — each workload cluster/spoke defines its own Argo CD applications and their specifications.

Create Required Network Policy

Before we start with the actual installation of the Agent, we need to ensure that the Redis instance on the spoke cluster is accessible for the Agent. We need to create a NetworkPolicy accordingly:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: argocd-agent-bm01-redis-network-policy
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: agent-argocd-redis (1)
ingress:
- ports:
- protocol: TCP
port: 6379
from:
- podSelector:
matchLabels:
app.kubernetes.io/name: argocd-agent-agent
policyTypes:
- Ingress

1	The name of the Redis instance. The label is based on <instance name>-redis.

Apply the NetworkPolicy to the spoke cluster:

oc apply -f network-policy.yaml -n argocd-agent-bm01 --context bm

Add the Helm Chart Repository

Add the Helm repository:

helm repo add openshift-helm-charts https://charts.openshift.io/
helm repo update

Install a managed Agent with the Helm Chart

Install the agent in the managed mode using the Helm Chart. The following parameters are used:

namespaceOverride - The namespace where the Agent is running.
agentMode - The mode of the Agent.
server - The server URL of the Principal component. This is the spec.host setting of the Principal’s route.
argoCdRedisSecretName - The name of the Redis secret.
argoCdRedisPasswordKey - The key of the Redis password.
redisAddress - The address of the Redis instance.

helm install redhat-argocd-agent openshift-helm-charts/redhat-argocd-agent \
--set namespaceOverride=argocd-agent-bm01 \
--set agentMode="managed" \
--set server="serverURL of principal route" \
--set argoCdRedisSecretName="agent-argocd-redis-initial-password" \
--set argoCdRedisPasswordKey="admin.password" \
--set redisAddress="agent-argocd-redis:6379" \
--kube-context "bm"

With this chart a pod on the spoke cluster will be created and will start to synchronize the Argo CD applications between the hub and the spoke cluster.

Verify the Managed Agent

To verify the Agent in managed mode we need to create an Argo CD Application on the hub cluster. We can try the following Application. The Application is taken from the openshift-clusterconfig-gitops repository and simply adds a banner to the top of the OpenShift UI. I typically use this as a quick test if GitOps is working.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: branding
namespace: argocd-agent-bm01 (1)
spec:
destination:
namespace: default
server: 'https://hub-argocd-agent-principal-resource-proxy:9090?agentName=argocd-agent-bm01' (2)
project: default
source:
path: clusters/management-cluster/branding
repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops'
targetRevision: main
syncPolicy:
automated:
prune: true
selfHeal: true

1	The namespace of the agent
2	The server URL of the Principal component. This is the URL plus the port and the agentName. As an alternative you can also use name: argocd-agent-bm01 which is the name of the cluster and might be easier to read.

Apply the Application to the hub cluster:

oc apply -f application.yaml -n argocd-agent-bm01 --context aws

Since the application will try to automatically synchronize the configuration, the status will change to Synced after a few seconds:

status:
resources:
- group: console.openshift.io
kind: ConsoleNotification
name: topbanner
status: Synced
version: v1

and the (top) banner will be visible on the UI:

Figure 1. Banner on the top of the UI

On the hub cluster, the Argo CD Application will be Synced:

oc get applications --context aws -A
NAMESPACE NAME SYNC STATUS HEALTH STATUS
argocd-agent-bm01 branding-banner Synced Healthy

Install an autonomous Agent with the Helm Chart

Let’s cleanup the first installation of the Chart (managed agent) in order to not have any conflicts.

helm uninstall redhat-argocd-agent --kube-context "bm"

Install the agent in the autonomous mode using the Helm Chart. The following parameters are used:

namespaceOverride - The namespace where the Agent is running.
agentMode - The mode of the Agent.
server - The server URL of the Principal component. This is the spec.host setting of the Principal’s route.
argoCdRedisSecretName - The name of the Redis secret.
argoCdRedisPasswordKey - The key of the Redis password.
redisAddress - The address of the Redis instance.

The only difference to the managed mode is the agentMode parameter.

helm install redhat-argocd-agent-autonomous openshift-helm-charts/redhat-argocd-agent \
--set namespaceOverride=argocd-agent-bm01 \
--set agentMode="autonomous" \
--set server="serverURL of principal route" \
--set argoCdRedisSecretName="agent-argocd-redis-initial-password" \
--set argoCdRedisPasswordKey="admin.password" \
--set redisAddress="agent-argocd-redis:6379" \
--kube-context "bm"

With this chart a pod on the spoke cluster will be created and will start to synchronize the Argo CD applications between the hub and the spoke cluster.

Verify the Autonomous Agent

To verify the Agent in autonomous mode we need to create an Argo CD Application on the spoke cluster.

We can try the following Application. It is basically the same as we used for the test for the managed mode, except that this time we will add a banner on the bottom of the UI.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: branding-bottom-banner
namespace: argocd-agent-bm01 (1)
spec:
destination:
namespace: default
server: 'https://kubernetes.default.svc' (2)
project: default
source:
path: clusters/management-cluster/branding-bottom
repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops'
targetRevision: main
syncPolicy:
automated:
prune: true
selfHeal: true

1	The namespace of the agent
2	The server URL of the local cluster, since in autonomous mode the application is managed locally.

Apply the Application to the spoke cluster:

oc apply -f application.yaml -n argocd-agent-bm01 --context bm

Like the managed mode, the status will change to Synced after a few seconds and the (this time) bottom banner will be visible on the UI:

Figure 2. Banner on the bottom of the UI

Moreover, the Argo CD Application will appear on the hub cluster:

oc get applications --context aws -A
NAMESPACE NAME SYNC STATUS HEALTH STATUS
argocd-agent-bm01 branding-bottom-banner Synced Healthy
argocd-agent-bm01 branding-banner Synced Healthy

Troubleshooting

During the installation and configuration of the Argo CD Agent, you might encounter some issues. Here are some issues I encountered during the tests:

Principal Pod Fails to Start

If the Principal pod fails to start with errors about missing secrets, verify that all required secrets have been created:

oc get secrets -n argocd-principal | grep argocd-agent

You should see the following secrets:

argocd-agent-ca - The CA certificate
argocd-agent-principal-tls - The Principal’s TLS certificate
argocd-agent-resource-proxy-tls - The Resource Proxy’s TLS certificate
argocd-agent-jwt - The JWT signing key

If any of these are missing, re-run the corresponding argocd-agentctl command to create them.

Redis Errors in the Principal Pod

When you see errors like the following in the logs of the Principal pod, ensure that the Argo CD instance does not have the controller enabled in that namespace (set spec.controller.enabled: false). Hopefully, this will change in the future.

time="2026-01-20T04:57:10Z" level=error msg="unexpected lack of '_' namespace/name separate: 'app|managed-resources|branding|1.8.3'" connUUID=3c13b30b-9f84-4af6-93d8-e1c03c4c7898 function=redisFxn module=redisProxy

Limitations

While the Argo CD Agent brings significant improvements, there are some limitations to be aware of:

Separate Argo CD Instance Required

Currently, the Principal component cannot be installed alongside an existing Argo CD instance where the application controller is enabled. You must create a separate Argo CD instance with the controller disabled (spec.controller.enabled: false). To me, this is one of the biggest limitations. However, this will be addressed in the future and is tracked in the issue: https://github.com/argoproj-labs/argocd-agent/issues/708

Manual Certificate Management

The mTLS certificates must be created and managed manually by the user. There is no automatic certificate rotation or renewal. For production environments, you should integrate with your organization’s PKI infrastructure and implement a certificate rotation strategy.

Summary

The Argo CD Agent provides a powerful solution for managing multiple clusters with Argo CD. By combining the benefits of centralized management with distributed application controllers, it addresses the scalability and single point of failure challenges of traditional deployment models.

While the initial setup requires several steps, especially around certificate management, the resulting architecture offers a robust foundation for GitOps at scale.

1	Allowed domains
2	Allow subdomains
3	Key size
4	Key type, typically rsa or ec (elliptic curve)

1	The namespace where the OpenBao deployment is running.
2	The self-signed CA Issuer.

YAUB Yet Another Useless Blog on TechBlog about OpenShift/Ansible/Satellite and much more

Creating an auditd rule to monitor Ansible Automation Platform static secrets

Ansible Automation Platform local SECRET_KEY

Configuring Linux auditd

Summary

Using a local Renovate bot to manage Ansible collection dependencies

Understanding renovate

Configuring and running renovate

Secret management and configuration cleanup

Integration with GitHub

Tips and tricks

Renovate does not honor new dependencies in requirements.yaml

Renovate does not open new PR (pull requests) for updated dependencies

Creating a customized RHEL 10 VM image with image-builder

Creating a RHEL 10 VM on macOS with bootc-image-builder

Onboarding to Ansible Automation Platform with Configuration as Code

OKD Single node installation in a disconnected environment

OpenShift Virtualization Networking - The Overview

Who does what: NMState vs. Multus

NMState Operator Installation

NMState Operator Resource Types

Part 1: Building the Bridge (NMState)

Part 2: Defining the NetworkAttachmentDefinition (NAD)

1. The Linux Bridge (CNI-Plugin)

A note on IPAM (IP Address Management)

2. OVN Kubernetes L2 Overlay

3. OVN Kubernetes Secondary Localnet

Part 3: Connecting the VM

GitOps Catalog

The Guide to OpenBao - Secrets Engines KV - Part 8

Introduction

List of Secrets Engines

KV (Key-Value) Secrets Engine

KV Version 1

KV Version 2

KV Version 1 vs Version 2

Enabling KV Engine

Storing Secrets

Reading Secrets

Managing Versions

Custom Metadata

Policies for KV v2

Conclusion

Resources

The Guide to OpenBao - Secrets Engines PKI - Part 9

Quick Navigation

Introduction

PKI Secrets Engine

Setting Up PKI Secrets Engine (Root CA)

Setting Up an Intermediate CA

PKI Policies

Integration with cert-manager

Enable the Kubernetes auth method

Configure the Kubernetes auth method

In-cluster example: reviewer service account and Secret

Configure auth/kubernetes/config

Policy and role for cert-manager

Verify with a Kubernetes login (optional)

Configure the (Cluster)Issuer for cert-manager

Issue a Certificate

Conclusion

Resources

The Guide to OpenBao - Authentication Methods - Part 7

Introduction

Authentication Workflow

Prerequisites for Authentication Methods

Using the CLI

Using the UI

Understanding Policies

Basic Policy Structure

Policy Capabilities

Default Policies

Creating Policies using the CLI

Kubernetes Authentication Method

Step 1: Enable Kubernetes Auth

Step 2: Configure Kubernetes Auth

Step 3: Create a Policy for the Application

Step 4: Create a Kubernetes Auth Role

Ensure the KV v2 secrets engine is mounted at secret/

Step 5: Store the database secret in OpenBao (one-time)

Ensure the KV v2 secrets engine is mounted at `secret/`