Security on TechBlog about OpenShift/Ansible/Satellite and much more

Creating an auditd rule to monitor Ansible Automation Platform static secrets

Tue, 09 Jun 2026 00:00:00 +0000

Ansible Automation Platform (AAP) comes with an elaborate credential management solution. All credentials within AAP are stored encrypted in a PostgreSQL database. But the keys used for encrypting database fields are currently stored on the machine(s) running AAP. In this blog post, we show we created an auditd rule to get at least some kind of monitoring if those keys are accessed.

Ansible Automation Platform local SECRET_KEY

AAP comes with a credential management solution. Credentials are used to

access remote machines
provide configuration data for automation, like API keys
connect AAP to source code repositories and inventories

and much more. You can even integrate external credential management solutions like HashiCorp Vault or OpenBao.

All credentials stored within AAP are saved as encrypted database fields in a PostgreSQL database. There is a main key and AAP creates for every encrypted database field a derived AES-256 encryption key. For more details see the documentation.

But there is one caveat: the main secret key is stored on the local file system. There is currently no supported option to integrate something like a HSM (High Security Module) for storing the main secret securely.

To mitigate this issue we deployed a Linux audit rule that monitors read/write and attribute changes to the main secret.

Configuring Linux auditd

We use the containerized version of AAP 2.6. In the containerized version of AAP those secrets are stored as podman secrets as the user running the AAP containers.

To list AAP podman secrets execute

$ podman secret list
ID NAME DRIVER CREATED UPDATED
02a2eb397dc5cedefde6ae411 hub_resource_server file 2 hours ago 2 hours ago
0dd84dc7b808917c4efb6ea9f controller_channels file 2 hours ago 2 hours ago
85fb73ff57729250c019976ed hub_database_fields file 2 hours ago 2 hours ago
c482ff7c07f908e443d327293 controller_postgres file 2 hours ago 2 hours ago
e6750121a8997cce8185072bf gateway_admin_password file 2 hours ago 2 hours ago
2731b2d03806f88f35809de66 eda_admin_password file 2 hours ago 2 hours ago
31ca44577b4c5b61dd3f0a0b6 eda_secret_key file 2 hours ago 2 hours ago
856473626bad9ff37b1b979d4 eda_resource_server file 2 hours ago 2 hours ago
b20ec0897fc2b6533544c4441 gateway_redis_url file 2 hours ago 2 hours ago
1dfdd812847ed4c1c01b194e2 gateway_db_password file 2 hours ago 2 hours ago
32d04575881dd781be192ef49 eda_db_password file 2 hours ago 2 hours ago
5102f69350717c15177b80821 hub_secret_key file 2 hours ago 2 hours ago
d1faf92922182604cfa746318 controller_resource_server file 2 hours ago 2 hours ago
d29f0d8c2fc709655dd918f2c postgresql_admin_password file 2 hours ago 2 hours ago
1b3ee24261bdff50d2c70e3f0 gateway_secret_key file 2 hours ago 2 hours ago
ae6c9e78e91dea180d22db6ee hub_settings file 2 hours ago 2 hours ago
b348d89bfb7d3b18f33a67d07 controller_secret_key file 2 hours ago 2 hours ago (1)

1	the controller secret key

If we want to know where those secrets are stored we can execute

$ podman secret inspect controller_secret_key
[
{
"ID": "b348d89bfb7d3b18f33a67d07",
"CreatedAt": "2026-06-09T10:57:28.278138715Z",
"UpdatedAt": "2026-06-09T10:57:28.278138715Z",
"Spec": {
"Name": "controller_secret_key",
"Driver": {
"Name": "file",
"Options": {
"path": "/home/admin/.local/share/containers/storage/secrets/filedriver"
}
},
"Labels": {}
}
}
]

Let’s see what’s in the filedriver directory:

$ ls /home/admin/.local/share/containers/storage/secrets/filedriver
secretsdata.json secretsdata.lock(1)

1	we are mainly interested in the secretsdata.json file

So the secretsdata.json file stores all podman secrets that are using the filedriver. For more information see the section Secret Drivers in the podman-secrets-create(1) manpage.

Let’s create a Linux auditd rule to monitor access to this file. As we are using Red Hat Enterprise Linux 10 a more detailed introduction to the Linux audit system is available here.

We created the file /etc/audit/rules.d/aap_secrets.rules with the following content:

-w /home/admin/.local/share/containers/storage/secrets/filedriver/secretsdata.json -p rwa -k aap_secrets (1)

1	-p rwa means monitor read/write/attribute changes, -k tells the audit system log events with an additional key "aap_secrets"

To activate the rule we need to load the audit rule via:

# auditctl -R /etc/audit/rules.d/aap_secrets.rules

You can list loaded audit rules via

# auditctl -l
-w /home/admin/.local/share/containers/storage/secrets/filedriver/secretsdata.json -p rwa -k aap_secrets

As a test we can cat the secrets file and check if auditd logged an event:

# cat /home/admin/.local/share/containers/storage/secrets/filedriver/secretsdata.json
...
# grep aap_secrets /var/log/audit/audit.log
type=SYSCALL msg=audit(1781009319.993:204): arch=c000003e syscall=257 success=yes exit=3 a0=ffffff9c a1=7ffc5276b65c a2=0 a3=0 items=1 ppid=6186 pid=6695 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts2 ses=3 comm="cat" exe="/usr/bin/cat" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="aap_secrets"ARCH=x86_64 SYSCALL=openat AUID="admin" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root" (1)

1	We can see that the cat command (comm=cat) was used by the root user to display the file contents

Summary

In this blog post we demonstrated how to configure the Linux audit system to log events if the AAP database secret key is accessed. The audit subsystem can also monitor system calls, for a detailed introduction see Audit system architecture.

The Guide to OpenBao - Secrets Engines KV - Part 8

Wed, 25 Mar 2026 00:00:00 +0000

Secrets engines are the heart of OpenBao’s functionality. They store, generate, or encrypt data. This article covers the most commonly used secrets engine: KV for static secrets. It will demonstrate how to use the KV secrets engine to store and retrieve secrets. Upcoming articles will cover the PKI secrets engine and the integration with cert-manager.

Introduction

Secrets engines are components which store, generate, or encrypt data. Some engines simply store and read data, others connect to other services and other engines provide encryption as a service.

They are enabled at a path in OpenBao (as everything is done as a path in OpenBao). This means each secrets engine defines its very own path including properties.

The path is case-sensitive. For example, the KV secrets engine enabled at kv/ and KV/ are treated as two distinct instances of KV secrets engine.

The management of the secret engines is done via the CLI or API or, if enabled, the UI. Engines can be enabled, disabled, moved or tuned.

Enable - enables a secrets engine at the given path.
Disable - disables an existing secrets engine. All existing secrets are revoked and the data is deleted.
Move - moves the path for an existing secrets engine.
Tune - tunes global configuration for the secrets engine such as the TTLs.

List of Secrets Engines

To list all enabled secrets engines, we can use the following command:

bao secrets list

Which will output something like:

Path Type Accessor Description
---- ---- -------- -----------
cubbyhole/ cubbyhole cubbyhole_f8b1e784 per-token private secret storage
identity/ identity identity_76940581 identity store
secret/ kv kv_ccb47c73 n/a
sys/ system system_cb0d64b7 system endpoints used for control, policy and debugging

As a reminder: To login to the OpenBao CLI, we can use the following command (assuming that you are using HTTPS and have the CA certificate ready):

export BAO_ADDR='https://127.0.0.1:8200'
export BAO_CACERT="$PWD/openbao-ca.crt"
bao login

KV (Key-Value) Secrets Engine

The KV secrets engine is probably the most commonly used engine. It is a generic key-value store. When you work with KV secrets engine you will note very soon that there are two versions of the engine: KV v1 and KV v2.

KV Version 1

KV v1 is the older version of the KV secrets engine. It does not support versioning and only stores the recently written value for a key. Requests to this engine will be more performant because the storage requests will be fewer and no locking is required. The storage as such will be much smaller, since it does not need to retain any history.

KV Version 2

KV v2 on the other hand supports versioning of secrets. When a version is deleted, the underlying data is not removed, instead you must destroy it intentionally. Storing the history of secrets will increase the required storage space. However, I would recommend using KV v2 in most cases as the benefit of versioning is too great to ignore.

The following example will use KV v2.

KV Version 1 vs Version 2

Feature	Version 1	Version 2
Versioning	No	Yes (up to 10 versions by default)
Check-and-Set	No	Yes (prevents concurrent writes)
Soft delete	No	Yes (recoverable)
Metadata	No	Yes (custom metadata)

Feature

Version 1

Version 2

Versioning

Yes (up to 10 versions by default)

Check-and-Set

Yes (prevents concurrent writes)

Soft delete

Yes (recoverable)

Metadata

Yes (custom metadata)

KV v1 can be upgraded to KV v2 by using the following command as described in the documentation: Upgrading from version 1

Enabling KV Engine

Let’s enable the KV v2 engine at the path secret/ and give it the description "Application secrets".

This path was used in Part 7 of this series already. So you might have to remove or rename the engine first if you have already enabled it.

bao secrets enable \
-path=secret \ (1)
-version=2 \ (2)
-description="Application secrets" \
kv (3)

1	Path where the secrets will be stored
2	Version of the engine in case of KV
3	Name of the engine

Storing Secrets

With the enabled engine, we can now store secrets at the path:

# Put a secret (KV v2)
bao kv put secret/myapp/database \ (1)
username="dbadmin" \ (2)
password="supersecret123" \
host="db.example.com" \
port="5432"

1	Path where the secret will be stored
2	Secret data

Which will output something like:

======= Secret Path =======
secret/data/myapp/database (1)
======= Metadata =======
Key Value
--- -----
created_time 2026-03-13T05:56:52.670296216Z
custom_metadata <nil>
deletion_time n/a
destroyed false
version 1 (2)

1	Path where the secret will be stored
2	Version of the secret

Reading Secrets

To read the secret, we can use one of the following commands:

# Read a secret
bao kv get secret/myapp/database

Which will output something like:

======= Secret Path =======
secret/data/myapp/database (1)
[... more metadata ...]
====== Data ====== (2)
Key Value
--- -----
host db.example.com
password supersecret123
port 5432
username dbadmin

1	Path where the secret will be stored
2	Secret data

As an alternative, we can use the following command to get the secret as JSON:

# Get as JSON
bao kv get -format=json secret/myapp/database

We can also read a specific existing version of the secret by using the following command:

# Read specific version
bao kv get -version=2 secret/myapp/database

Or simply, a specific field:

# Get specific field
bao kv get -field=password secret/myapp/database

If the UI is enabled, the same can be achieved via the browser:

Figure 1. OpenBao Secret Retrieval

Managing Versions

KV v2 keeps a version history. This is a big advantage over KV v1. To try version management, ensure the secret has at least two versions. Create a second version by writing to the same path again (e.g. with updated values):

bao kv put secret/myapp/database username=dbadmin password=newpassword456 port=5432 (1)

1	Updating an existing secret with a new password, which will create a new version in KV v2.

Now you can list versions, soft-delete, undelete, or destroy a specific version:

I will only paste the output of the first command. The rest will be similar … you’ll get the idea.

# List all versions and metadata
bao kv metadata get secret/myapp/database

Which will output something like:

====== Version 1 ====== (1)
Key Value
--- -----
created_time 2026-03-13T05:56:52.670296216Z
deletion_time n/a
destroyed false
====== Version 2 ====== (2)
Key Value
--- -----
created_time 2026-03-13T06:13:18.539285992Z
deletion_time n/a
destroyed false

1	Version 1
2	Version 2

Specific versions can now be (soft-) deleted, permanently destroyed etc.

The last command of the list will completely delete all versions and metadata for a given path aka the secret is gone.

# Delete specific version (soft delete)
bao kv delete -versions=2 secret/myapp/database
# Undelete a version
bao kv undelete -versions=2 secret/myapp/database
# Permanently destroy a version
bao kv destroy -versions=2 secret/myapp/database
# Delete all versions and metadata
bao kv metadata delete secret/myapp/database

When you define a secret you can set the maximum number of versions to keep using the flag -max-versions=5. This is the number of versions that will be kept in the history.

Custom Metadata

A secret’s key metadata can contain multiple custom metadata that can be used to describe the secret. The data will be stored as key-value pairs. Imagine them like labels in Kubernetes.

Let’s extend our secret with some custom metadata:

# Add custom metadata to a secret
bao kv metadata put \
-custom-metadata="owner=team-a" \ (1)
-custom-metadata="environment=production" \ (2)
secret/myapp/database

1	First custom metadata to set an owner label
2	Second custom metadata to set an environment label

Policies for KV v2

OpenBao policies control which authenticated clients may access which paths and with what actions. For the KV v2 secrets engine, any read or write via the API is checked against the token’s policies; without a policy granting the right capabilities, requests return 403. This section shows example policy rules you can attach to roles (e.g. Kubernetes auth roles) or tokens so that applications and users have the minimum access they need.

KV v2 exposes two logical path trees under the mount path (here secret/):

secret/data/<path> — Secret values. Use capabilities such as read, create, update, and delete for reading and writing secret data.
secret/metadata/<path> — Version and metadata. Use list to enumerate keys, and read/delete for version metadata and destroying versions.

The following example grants read-only access to all secrets under myapp/, plus full read/write (including create, update, delete) for the single path myapp/database. Replace myapp with your application or team prefix as needed.

# Note: KV v2 uses /data/ for values and /metadata/ for metadata
# Read secret values under myapp/
path "secret/data/myapp/*" {
capabilities = ["read"]
}
# List secret keys under myapp/ (required for listing, e.g. bao kv list secret/myapp)
path "secret/metadata/myapp/*" {
capabilities = ["list"]
}
# Full access to a specific secret (create, read, update, delete)
path "secret/data/myapp/database" {
capabilities = ["create", "read", "update", "delete"]
}
# Read and delete version metadata for this secret (e.g. destroy version)
path "secret/metadata/myapp/database" {
capabilities = ["read", "list", "delete"]
}

After writing the policy to OpenBao (e.g. bao policy write myapp-kv policy.hcl), attach it to the appropriate auth method role or token so that clients receive a token with this policy. For token renewal, ensure the role or token also has update on auth/token/renew-self if you use renewable tokens.

Part 7 of this series (Authentication Methods) explains the authentication methods and policies in more detail.

Conclusion

This part focused on the KV secrets engine: enabling a mount, storing and reading secrets, version lifecycle (list, soft delete, destroy), custom metadata, and policies that distinguish KV v2’s data/ and metadata/ paths.

Key takeaways:

Prefer KV v2 when you need versioning, check-and-set, soft delete, and richer metadata; use KV v1 only when you explicitly want the simpler, lower-overhead path.
Grant secret/data/… for secret values and secret/metadata/… for listing and version operations—policies must match how the CLI and API resolve KV v2 paths.
Attach policies to tokens or auth roles (for example Kubernetes auth) so applications receive least-privilege access; include auth/token/renew-self if tokens should be renewable.

The PKI secrets engine and integration with cert-manager are covered in the next article in this series.

Resources

KV Secrets Engine v2

The Guide to OpenBao - Secrets Engines PKI - Part 9

Thu, 26 Mar 2026 00:00:00 +0000

Secrets engines are one of the most important concepts in OpenBao. Part 8 covered the KV secrets engine; this article covers the PKI secrets engine and its integration with cert-manager on Kubernetes and OpenShift.

Quick Navigation

Navigate directly to the main sections of this article:

PKI secrets engine
Integration with cert-manager

Introduction

Part 8 of this series explained the basics of secrets engines and the KV secrets engine. This article focuses on the PKI secrets engine and on integrating OpenBao with cert-manager.

As a reminder: to log in to the OpenBao CLI, use the following (assuming HTTPS and a CA certificate available locally):

export BAO_ADDR='https://127.0.0.1:8200'
export BAO_CACERT="$PWD/openbao-ca.crt"
bao login

PKI Secrets Engine

The PKI engine dynamically generates X.509 certificates, acting as a Certificate Authority (CA). You can obtain a certificate without manually generating a private key and CSR on the client: OpenBao provisions the key material and signed certificate according to the role.

The time to live (TTL) of each generated certificate can be kept short to reduce reliance on revocation. It is possible to store the certificates in memory during the startup of an application and discard them when the application shuts down. This way, a certificate is only valid during the application runtime, is kept in memory and is never written to disk.

Setting Up PKI Secrets Engine (Root CA)

Before we can use the PKI secrets engine, we need to enable and configure it. To enable it, we can simply use the following command:
```
bao secrets enable -path=pki pki
```
This will enable the PKI secrets engine at the path pki/. If the path parameter is not provided, the engine will use the name of the engine as path.

The default value of the TTL is 30 days. To increase the maximum lease time for certificates issued under this mount, use -max-lease-ttl=87600h (or another duration).

Note that individual roles can restrict this value to be shorter on a per-certificate basis. This just configures the global maximum for this secrets engine. Source: PKI secrets engine - setup and usage

Next we need to provide a CA certificate and a private key. OpenBao can either generate its own self-signed root certificate, or can use an existing one. Using an existing one is recommended for production environments. This is managed outside of OpenBao and OpenBao is provided with a signed intermediate CA.

We can simply generate a self-signed root certificate by using the following command:

bao write pki/root/generate/internal \
common_name="Example Root CA" \ (1)
issuer_name="root-ca" \ (2)
ttl=8760h \ (3)
key_bits=4096 \ (4)

1 Common name of the root certificate

2 Issuer name of the root certificate

3 Time to live of the root certificate, here 8760 hours (365 days)

Key bits of the root certificate, here 4096 bits (default is 2048 bits)

The response will look something like:

Key Value
--- -----
certificate -----BEGIN CERTIFICATE-----...
expiration 1536807433
issuing_ca -----BEGIN CERTIFICATE-----...
serial_number 7c:f1:fb:2c:6e:4d:99:0e:82:1b:08:0a:81:ed:61:3e:1d:fa:f5:29

The certificate key is stored in OpenBao. The returned certificates are purely informational.

Set URL configuration

The location of the Certificate Revocation List (CRL) and the issuing certificate are stored in the configuration of the PKI secrets engine and must be updated manually.
```
# Configure CA and CRL URLs
bao write pki/config/urls \
issuing_certificates="https://openbao.example.com/v1/pki/ca" \
crl_distribution_points="https://openbao.example.com/v1/pki/crl"
```
Configure a Role

Now we need to define a role for the certificates. Roles define the allowed domains and the maximum time to live for the certificates. The following example creates a role for server certificates. It allows us to issue certificates for the domains example.com and internal.example.com with a maximum time to live of 720 hours (30 days) and a key size of 2048 bits.
```
bao write pki/roles/server-cert \
allowed_domains="example.com,internal.example.com" \ (1)
allow_subdomains=true \ (2)
max_ttl=720h \
key_bits=2048 \ (3)
key_type=rsa \ (4)
require_cn=false \
allow_any_name=false
```
1 Allowed domains

2 Allow subdomains

3 Key size

4 Key type, typically rsa or ec (elliptic curve)

Issuing Certificates

OpenBao is now ready to issue certificates. We will use the issue endpoint to request a new certificate. The following example requests a new certificate for the domain myapp.example.com and the subdomain myapp.internal.example.com with a maximum time to live of 168 hours (7 days) and a key size of 2048 bits.

bao write pki/issue/server-cert \
common_name="myapp.example.com" \
alt_names="myapp.internal.example.com" \
ttl=168h \

This will return the full certificate chain including the root CA certificate and the key. For better readability, the output was shortened to show the certificate only:

Key Value
--- -----
certificate -----BEGIN CERTIFICATE-----
MIIE7zCCAtegAwIBAgIUH0eTVpXCP7iu0Y75dWAvjktbyxcwDQYJKoZIhvcNAQEL
BQAwGjEYMBYGA1UEAxMPRXhhbXBsZSBSb290IENBMB4XDTI2MDMyMjA1MjEyNVoX
DTI2MDMyOTA1MjE1NFowHDEaMBgGA1UEAxMRbXlhcHAuZXhhbXBsZS5jb20wggEi
MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDliY0K3qb8TwNBXOHdfBygs5+Y
Pwh83vtS0Nph6ZsPdI9JTuvI6AbzEz6M283aY7OeJ3m3G2fRwt7ZdgsAcyt+jnL8
Izx/MS/4ztsdZJ/wxs3M42/iFehSofqihgorY1+SGaTdveCFg9H7Egjn1eJFnGB/
iqANzxg5NohE4gf8zpAKDjl4CIPDSmNJX2xQ+3wXJgx1eKeVOeB0jFnMSYskS0hD
qwM+QyYRKodc/akI6Xu7OqWQLMW7qmNS+6TBHsGOopve9EP6ym65Jss6de4X9zIP
BYeWQ0gvG18i22eJrkISRWlO2uw+Fbho0tBhqfRfjsVpxrUrTKwIgxg+F0+5AgMB
AAGjggEpMIIBJTAOBgNVHQ8BAf8EBAMCA6gwHQYDVR0lBBYwFAYIKwYBBQUHAwEG
CCsGAQUFBwMCMB0GA1UdDgQWBBRbFm9oqRl//tc+Z0ert4DKp9KjkDAfBgNVHSME
GDAWgBSL03nIIcEY+IAz/1g2q5NBbufx9DBBBggrBgEFBQcBAQQ1MDMwMQYIKwYB
BQUHMAKGJWh0dHBzOi8vb3BlbmJhby5leGFtcGxlLmNvbS92MS9wa2kvY2EwOAYD
VR0RBDEwL4IRbXlhcHAuZXhhbXBsZS5jb22CGm15YXBwLmludGVybmFsLmV4YW1w
bGUuY29tMDcGA1UdHwQwMC4wLKAqoCiGJmh0dHBzOi8vb3BlbmJhby5leGFtcGxl
LmNvbS92MS9wa2kvY3JsMA0GCSqGSIb3DQEBCwUAA4ICAQBUoLwWgpTaodRKpL3c
zp/yPgHNagZA8SQRLqKttywbHzfPNUSsC56XqfZnOHaq/gobNKrmirUAHPnXRUjO
Jd2+wXBO9aKqAhiy5is2ueNRo+C8IdPQpFNQ+5MW+G4FXJqSkvDUlT47mAoRkKUJ
nsjoV/xzHYKgw/YF7QZtmL8moM9W9Bhg3dB6KEaUvscpNn/9USfPquVLXOuQ/8Bs
6zQGijJUhb5gpRgGjmJhs0Oo/MkABxBUq2jFmvGWP0NklcJO7smtetx/RSrtyFZ0
X7zgMlD1yUy4DaHX13LhszAfercTFNodV/mAfEaJUrW+U9I97MzVg8QJvn9/AsaM
mWjfKHczhd9AzupOlogX8L3cdZxx9hkdD4NcR8tRb4QR5sOqzoBN7NfNpQ699b0F
kxESrHIysqBcx4dYjXLFsUqzRGm2sGZ4j1N8w1OZY/s0kfsG7RXpXJjwtQ2RwPo1
iSjbJ/VLjji2pz5BtVRtHiXnkoVtSD8QBpjCYqyHekXUArJCVkaDPzkB2MWNZXpk
2koi2fYa0U4x4ITqDsKeMpLvGAl2vqkp20lphXXqRwaAdNuEFd69YZ6qXzw/6GdI
TC/Ki7zApSvO0kJvMs9NILcHgDjpndCA/sMpG9iv4wFMpgYet9wc+dzPMXnHcj4o
PGbyb/WJomcxQMDuV1x1S1X2iA==
-----END CERTIFICATE-----
expiration 1774761714
issuing_ca -----BEGIN CERTIFICATE-----
MIIFJTCCAw2gAwIBAgIUJde0eccC1TpbufxVsUBhZhq9hWwwDQYJKoZIhvcNAQEL
BQAwGjEYMBYGA1UEAxMPRXhhbXBsZSBSb290IENBMB4XDTI2MDMyMjA0NDI1OVoX
DTI2MDQyMzA0NDMyOVowGjEYMBYGA1UEAxMPRXhhbXBsZSBSb290IENBMIICIjAN
BgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA1XGanJKZZYySc9+WCXuGp9MLMo/O
MCRAR0u+BQbmxaSBcFLsxfNdkB8Jcunx7RnaGZIJwYXNIy+tNejqwQfiah7pTwJq
SYnXcYqZAETJUFvdYygjs4MCeaebvogmBQcZRzxJNNPzUblYpVOhmSGLvZZ7Vs5F
zY9wsKUEc3c3fMhwbxe/n/KQzvT/VzXSrLOw54LTsG7GNnUIhkuKwNqkhXuVItDy
ps+3z7RG/hOAWN9vktqLybi/E/uTmUqpsMrHSg7EMjqJ2jP8FMyH9EARLRL9gUG3
XFFNhFzjjDKjFrMsTUyVr+9tp4ucCZaakP9ZhjN9R/6FCXXR1K/XTVhTgYkawPHd
+PJaXn2T7NOie+tChiZx5Sii2XZtDm7WMFfwb4xhOBV67pWCSWSjkKl/xIDSrdRj
qLtzVfP/wiSL4sL9fy7Pya2OBtF2d/bdXXQ54JbbRlw3eRP+y+ej63UD+OP602EE
bbzrGB+B7ozwnfCmNNBnF23GgVYtJPWfDvgjXicprvbgjpmQAjeFohA8IIC322w5
sFsYuDL46KEs/KREoHjQNo1pxHKnmITU5GIMmWWoPyidTZuNNc4qWM6cH5RAbJup
IBaJWcWAr4eZ8p7pZd1dMkWEld4tieLdoJv2ENK77rDhwrq8kz66Xw9/xDJ/Nuio
sw5sd3py6Ayz+ekCAwEAAaNjMGEwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQF
MAMBAf8wHQYDVR0OBBYEFIvTecghwRj4gDP/WDark0Fu5/H0MB8GA1UdIwQYMBaA
FIvTecghwRj4gDP/WDark0Fu5/H0MA0GCSqGSIb3DQEBCwUAA4ICAQAJuRiReD46
H85vhc1wzg7JUy6k+q5aUA9iKOupGyYBtkvzniQzhouZGEP47AcWkbPNGqo/3yl+
SCUItkBfPjunBmzcAhhZqyxJJ1q/SgY21DBf3W7WB0yzPx+saCwEi74rf44HbslY
Gp9W9pv0NFDT1+lQGerp978ErGQZh2M8rrGFwhcHmx71/umZ9ju6xyaGRaoHba+Y
8bxhR11j/Zx/UGRQBpjdAe2+VuAHDGpHjd9ieQ45YgoyJYkjBbecXJFttcVnShwu
7rxu8HNV3fMjoa4M6TDaY4z8KDgzHChzQSRB085TJIZzcG+/4zpJredzLe+jEq0P
RAuFEx4FjpRqYQu08HUmepO0Vd8a2ut7WyIBOC5b+pM+55LaGDfDF6rt8H7g6G84
w/RYFP/w/elxp3T1cFgDXsuZ60mfK6SX5ZbH3ZGPN0+xF/VahncUWKSfxSaSjzRI
Eml/Aac22kgtbbtxf8Bn4tgtY30sxqQRVnldHKi02FxS+DNg3SZB1hIpw6myyUo0
XH/ZtyumDLkcCvKzz/KvQGDOWDBmaPfPOPfMgiFXryB+YFMfy75Bhm0+aE+GM8fP
sextO8kawYI62aCHiOClkm2qIBiFtbsNG2HMTL4HKaNDOh9vxesPD5EteVzxTvg4
Oyk8YPB4kGtloH569v2Fj2U24D6KRyqxvQ==
-----END CERTIFICATE-----
not_before 1774156885

That is it. We have just created our first certificate.

Setting Up an Intermediate CA

The test above demonstrated the usage of the PKI secrets engine to issue a certificate. However, the certificate was issued by the root CA. This is not practical or recommended for production environments. Instead, we should use an intermediate CA. The setup is very similar to the root CA, but with a few key differences: Instead of directly signing with the root CA, we will create a CSR (Certificate Signing Request) and sign it with the root CA. This can then be used to issue certificates with the intermediate CA.

First we enable the PKI secrets engine for the intermediate CA and tune it a little bit.
```
# Enable PKI engine for intermediate CA
bao secrets enable -path=pki_int pki
# Configure maximum lease time
bao secrets tune -max-lease-ttl=43800h pki_int
```
We set the maximum TTL of the intermediate CA to 43800 hours (5 years). It should be equal to or less than the root CA.

Generate a CSR (Certificate Signing Request)

# Generate intermediate CSR
bao write -format=json pki_int/intermediate/generate/internal \
common_name="Example Intermediate CA" \
issuer_name="intermediate-ca" \
key_bits=4096 \
| jq -r '.data.csr' > pki_int.csr

This will create the CSR and store it at pki_int.csr, which can be signed by the root CA and then used to issue certificates with the intermediate CA. As an example, the following CSR is generated:

Sign the CSR with the root CA

The CSR must now be signed using the root CA that was generated previously. Use:

bao write -format=json pki/root/sign-intermediate \
csr=@pki_int.csr \ (1)
format=pem_bundle \
ttl=43800h \
| jq -r '.data.certificate' > cert_bundle.pem (2)

1	The CSR file created in the previous step
2	The signed certificate is stored in the file `cert_bundle.pem` This will sign the CSR with the root CA and store the signed certificate at `cert_bundle.pem`.

Configure the set-signed certificate

We need to configure the intermediate CA with the signed certificate, so we can issue new certificates:
```
bao write pki_int/intermediate/set-signed \
certificate=@cert_bundle.pem
```

Creating Certificate Roles

Before we can issue certificates, we need to create roles. The following defines a role for server certificates and a role for client certificates, including allowed domains, maximum TTL, and key type.

# Create a role for issuing server certificates
bao write pki_int/roles/server-cert \
allowed_domains="example.com,internal.example.com" \ (1)
allow_subdomains=true \ (2)
max_ttl=720h \
key_bits=2048 \ (3)
key_type=rsa \
require_cn=false \
allow_any_name=false
# Create a role for client certificates
bao write pki_int/roles/client-cert \
allowed_domains="example.com" \ (1)
allow_subdomains=true \
max_ttl=168h \
key_usage="DigitalSignature,KeyEncipherment" \
ext_key_usage="ClientAuth"

The allowed domains, comma separated list

Allow subdomains, true or false

The key bits, 2048 or 4096

Check out the official documentation for more details of possible options: https://openbao.org/docs/

Issuing Certificates

Finally we can issue certificates. The example below issues a server certificate via the intermediate mount (and the article’s role setup supports a separate client role if you need mutual TLS).

# Issue a server certificate
bao write pki_int/issue/server-cert \
common_name="myapp.example.com" \ (1)
alt_names="myapp.internal.example.com" \ (2)
ttl=168h

1	The common name, the subdomain under example.com is allowed.
2	The alternative names This will print a large amount of detail. Typical fields include: `certificate` `issuing_ca` `private_key` `serial_number`

The output contains the full chain (intermediate and root CA), the leaf certificate, and the private key, which you can use to configure TLS in an application.

+ TIP: It can also be printed out in JSON format which is easier to process later.

+ The following example creates the JSON file with all the components.

bao write -format=json pki_int/issue/server-cert \
common_name="api.example.com" \
ttl=168h > api-cert.json

+ This can then be extracted to the individual components using the jq command.

cat /tmp/api-cert.json | jq -r '.data.certificate'
cat /tmp/api-cert.json | jq -r '.data.private_key'
cat /tmp/api-cert.json | jq -r '.data.ca_chain[]'

PKI Policies

Every OpenBao request is checked against the token’s policies; without an explicit allow, paths are denied. PKI is split into several API paths (issue, sign, ca, crl, …), so you grant only what each client needs.

For example: operators using pki_int/issue/… need the issue path; cert-manager sending a CSR needs the sign path; workloads or validators that fetch trust material need read on the CA (and often the CRL) paths etc.

The example below is a single policy you can attach to a role or token; split into narrower policies in production if different actors should not share the same privileges.

set -e
cat <<'EOF' | bao policy write pki-int-workload -
# Allow issuing certificates with the server-cert role (CLI / API issue endpoint)
path "pki_int/issue/server-cert" {
capabilities = ["create", "update"]
}
# Allow cert-manager (Vault issuer) to sign CSRs via the sign endpoint for the same role
path "pki_int/sign/server-cert" {
capabilities = ["create", "update"]
}
# Allow reading CA certificate
path "pki_int/ca/pem" {
capabilities = ["read"]
}
# Allow reading CRL
path "pki_int/crl/pem" {
capabilities = ["read"]
}
EOF
bao policy read pki-int-workload

Integration with cert-manager

cert-manager requests TLS certificates inside Kubernetes and stores them in Secrets. It includes a Vault issuer that talks to HashiCorp Vault’s HTTP API; OpenBao exposes the same API for PKI and auth, so you can point that issuer at OpenBao instead of Vault.

cert-manager generates a private key and a CSR in the cluster, then calls the PKI sign endpoint for your role (not the issue endpoint you used from the CLI earlier). The path in the issuer must therefore follow mount/sign/role-name, for example pki_int/sign/server-cert, matching the intermediate mount and role from this article.

Before the ClusterIssuer can log in to OpenBao, you need Kubernetes auth enabled, a policy that allows signing on pki_int (see PKI policies), and a role that binds the cert-manager controller’s service account to that policy. Part 7 of this series walks through Kubernetes authentication in detail; the commands below are a minimal summary for cert-manager.

Enable the Kubernetes auth method

Skip this step if kubernetes is already listed under bao auth list.

bao auth enable --description="Kubernetes/OpenShift service account auth" kubernetes

Configure the Kubernetes auth method

OpenBao’s server validates incoming service-account JWTs by calling the Kubernetes TokenReview API. If OpenBao runs inside the cluster (the setup assumed here), configure auth/kubernetes with:

kubernetes_host — API URL reachable from the OpenBao pods (typically https://kubernetes.default.svc:443).
kubernetes_ca_cert — PEM of the cluster CA that signs the API server certificate (same material as in the service-account token Secret).
token_reviewer_jwt — long-lived JWT for a dedicated service account that may create TokenReview objects (via system:auth-delegator).

If OpenBao runs outside the cluster, use your public API URL for kubernetes_host (for example $(oc whoami --show-server)), the same CA and reviewer JWT extracted from the cluster, and ensure the OpenBao process can reach that URL on the network.

In-cluster example: reviewer service account and Secret

Create a service account in the OpenBao namespace (here openbao), bind system:auth-delegator, and request a long-lived token Secret (required from Kubernetes 1.24 onward unless you use another token source).

---
apiVersion: v1
kind: ServiceAccount
metadata:
name: openbao-kube-auth
namespace: openbao (1)
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: openbao-kube-auth-delegator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator (2)
subjects:
- kind: ServiceAccount
name: openbao-kube-auth
namespace: openbao
---
apiVersion: v1
kind: Secret
metadata:
name: openbao-kube-auth-token
namespace: openbao
annotations:
kubernetes.io/service-account.name: openbao-kube-auth
type: kubernetes.io/service-account-token (3)

1	The namespace of the OpenBao service account
2	The name of the ClusterRole that binds the service account to the system:auth-delegator role
3	The type of the Secret, which is a long-lived token Secret

Save the YAML manifest to a file (for example openbao-kube-auth.yaml), apply it, then wait until the Secret contains a token (the control plane fills data.token—often within a few seconds).

oc apply -n openbao -f openbao-kube-auth.yaml
until oc get secret openbao-kube-auth-token -n openbao -o jsonpath='{.data.token}' | grep -q .
do
sleep 2
done

You can manage the reviewer service account, ClusterRoleBinding, and token Secret through GitOps so the configuration stays reproducible.

Configure auth/kubernetes/config

Dump the reviewer JWT and CA to files and configure the Kubernetes auth backend. Use the in-cluster API URL so OpenBao pods reach the Kubernetes API without relying on the external load balancer.

TMP=$(mktemp -d) (1)
trap 'rm -rf "$TMP"' EXIT
oc get secret openbao-kube-auth-token -n openbao -o jsonpath='{.data.token}' | base64 -d > "$TMP/reviewer.jwt" (2)
oc get secret openbao-kube-auth-token -n openbao -o jsonpath='{.data.ca\.crt}' | base64 -d > "$TMP/k8s-ca.crt" (3)
bao write auth/kubernetes/config \ (4)
kubernetes_host="https://kubernetes.default.svc:443" \
kubernetes_ca_cert=@"$TMP/k8s-ca.crt" \
token_reviewer_jwt=@"$TMP/reviewer.jwt"

1	Create a temporary directory to store the reviewer JWT and CA and remove it after the script exits
2	Get the reviewer JWT from the Secret
3	Get the CA from the Secret
4	Write the configuration to the auth/kubernetes/config path using the internal API URL of the Kubernetes cluster.

On plain Kubernetes, replace oc with kubectl and the same Secret paths.

If logins fail after cluster upgrades or with projected service-account tokens, check whether you must set issuer or related options; see OpenBao Kubernetes auth and Part 7.

To inspect the stored Kubernetes auth backend configuration:

bao read auth/kubernetes/config

You should see kubernetes_host and kubernetes_ca_cert; token_reviewer_jwt is often omitted or redacted on read.

Policy and role for cert-manager

Attach a policy (here cert-manager-pki) that matches the PKI paths cert-manager uses—aligned with PKI policies—then map the controller’s service account to that policy.

Adjust bound_service_account_names and bound_service_account_namespaces if your cert-manager installation does not use the cert-manager service account in the cert-manager namespace.

bao policy write cert-manager-pki - <<'EOF'
path "pki_int/sign/server-cert" {
capabilities = ["create", "update"]
}
path "pki_int/ca/pem" {
capabilities = ["read"]
}
path "pki_int/crl/pem" {
capabilities = ["read"]
}
path "auth/token/renew-self" {
capabilities = ["update"]
}
EOF
bao write auth/kubernetes/role/cert-manager \
bound_service_account_names=cert-manager \
bound_service_account_namespaces=cert-manager \
policies=cert-manager-pki \
ttl=1h \
max_ttl=24h

Verify with a Kubernetes login (optional)

After the cert-manager auth role exists, prove that OpenBao accepts the same service-account identity cert-manager will use. Request a short-lived projected token for that service account, then log in to the Kubernetes auth method:

JWT=$(oc create token cert-manager -n cert-manager --duration=15m)
bao write -format=json auth/kubernetes/login role=cert-manager jwt="$JWT" \
| jq -r '.auth.client_token'

A printed client token means TokenReview and role binding succeeded.

Configure the (Cluster)Issuer for cert-manager

To integrate cert-manager with OpenBao we need to configure either a ClusterIssuer (cluster-wide) or an Issuer (namespace-scoped). The setup is very similar.

The ClusterIssuer manifest below must use the same OpenBao role name in spec.vault.auth.kubernetes.role (here cert-manager).

The server URL in the issuer must be reachable from the cert-manager pods (in-cluster Service DNS is typical), for example https://openbao.openbao.svc:8200.

The following is an example of a ClusterIssuer manifest.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: openbao-cluster-issuer
spec:
vault:
path: pki_int/sign/server-cert (1)
server: https://openbao.openbao.svc:8200 (2)
caBundle: LS0tLS1CRUdJTi... (3)
auth:
kubernetes:
role: cert-manager
mountPath: /v1/auth/kubernetes (4)
serviceAccountRef:
name: cert-manager (5)

1	PKI sign path for the OpenBao role (mount/sign/role-name).
2	OpenBao Service URL as seen from cert-manager pods.
3	Required for private / self-signed OpenBao TLS: single-line base64 of the PEM CA (see the command block below). Omit only if the server uses a public CA trusted by default in the cert-manager image.
4	Kubernetes auth mount path in OpenBao (default is `/v1/auth/kubernetes`).
5	Name of the cert-manager controller service account only. Do not add `namespace` under `serviceAccountRef`; it is not part of the cert-manager API.

If OpenBao’s HTTPS certificate is not signed by a CA already trusted inside the cert-manager pods, you must supply trust material; otherwise TLS verification fails with x509: certificate signed by unknown authority.

Use caBundle (base64-encoded PEM of the CA that signed OpenBao’s server certificate) or caBundleSecretRef pointing to a Secret in the cert-manager namespace.

Fetch the certificate OpenBao is using:

oc get secret openbao-server-tls -n openbao -o jsonpath='{.data.ca\.crt}{"\n"}'

This secret was created in a previous article, when we were setting up the OpenBao server.

The returned string must be added to the ClusterIssuer manifest as the value of the caBundle field.

Alternatively, copy the CA PEM into a Secret in the cert-manager namespace and reference it (no giant line in the issuer):

# spec.vault fragment — use instead of caBundle if you prefer
caBundleSecretRef:
name: openbao-ca
key: ca.crt

However, this requires that the Secret exists in the appropriate namespace. For ClusterIssuer, the Secret must be in the cert-manager namespace.

Once the ClusterIssuer is created and is ready, we can issue a certificate.

Issue a Certificate

With the ClusterIssuer in place, create a Certificate in the workload namespace (here myapp, this namespace must exist). cert-manager will create a TLS Secret (myapp-tls below) containing tls.crt and tls.key, and renew the cert before it expires according to renewBefore.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: myapp-tls
namespace: myapp (1)
spec:
secretName: myapp-tls (2)
issuerRef: (3)
name: openbao-cluster-issuer
kind: ClusterIssuer
commonName: myapp.example.com (4)
dnsNames:
- myapp.example.com
duration: 168h (5)
renewBefore: 48h (6)
privateKey:
rotationPolicy: Always (7)

1	The namespace of the workload
2	The name of the TLS Secret
3	The issuer of the certificate
4	The common name of the certificate
5	The duration of the certificate
6	The duration before the certificate is renewed
7	Explicit rotation policy: `Always` matches cert-manager ≥ v1.18 default (new key each renewal). Set `Never` only if you rely on a stable key across renewals.

The values you set in commonName and dnsNames must be allowed by the OpenBao PKI role (allowed_domains, allow_subdomains, allow_bare_domains, and related settings).

If the Certificate never reaches Ready=True, check kubectl describe certificate / kubectl describe certificaterequest (or the same with oc describe on OpenShift) in that namespace, the cert-manager controller logs, and the OpenBao audit log for 403 policy denials or CSR rejection messages from the PKI engine.

When everything is working, a Secret named myapp-tls is created in the myapp namespace containing tls.crt and tls.key.

oc describe secret/myapp-tls -n myapp
[source,bash]
----
Name: myapp-tls
Namespace: myapp
[...output omitted...]
Type: kubernetes.io/tls
Data
====
tls.key: 1679 bytes
ca.crt: 1846 bytes
tls.crt: 3631 bytes

Conclusion

This part showed how to turn OpenBao into a usable CA for cluster workloads and how to automate TLS with cert-manager.

You should now be able to:

Enable the PKI engine, set CA and CRL URLs, define roles, and issue certificates via the issue API (CLI or automation).
Stand up an intermediate CA signed by the root, and keep day-to-day issuance on the intermediate mount (pki_int).
Express least-privilege with policies that separate issue from sign, and allow CA/CRL reads where validators or clients need them.
Wire cert-manager to OpenBao’s Vault-compatible API: Kubernetes auth in-cluster (reviewer JWT and CA), a ClusterIssuer with caBundle (or caBundleSecretRef) for private OpenBao TLS, serviceAccountRef.name only (no namespace field), and a Certificate with an explicit privateKey.rotationPolicy for cert-manager ≥ v1.18.

Resources

PKI Secrets Engine

The Guide to OpenBao - Authentication Methods - Part 7

Mon, 09 Mar 2026 00:00:00 +0000

With OpenBao deployed and running, the next critical step is configuring authentication. Ultimately, you want to limit access to only authorised people. This article covers two common authentication methods: Kubernetes for pods and LDAP for enterprise directories (in a simplified example). There are many more methods, but we cannot cover them all in this article.

Introduction

Authentication in OpenBao verifies the identity of clients before granting access to secrets. OpenBao supports multiple authentication methods, each suited for different use cases. Among others it supports:

Method	Best For	Key Features
Kubernetes	Pods running in K8s/OpenShift	Service account tokens, automatic
OIDC	Human users, SSO	OpenShift OAuth, Keycloak, Azure AD
LDAP	Enterprise directories	Active Directory, OpenLDAP
AppRole	CI/CD, automation	Role ID + Secret ID
Token	Direct access, bootstrap	Simple but less secure

Method

Best For

Key Features

Kubernetes

Pods running in K8s/OpenShift

Service account tokens, automatic

OIDC

Human users, SSO

OpenShift OAuth, Keycloak, Azure AD

LDAP

Enterprise directories

Active Directory, OpenLDAP

AppRole

CI/CD, automation

Role ID + Secret ID

Token

Direct access, bootstrap

Simple but less secure

The full documentation of the authentication methods and the list of available methods (for example Radius) can be found here: OpenBao Authentication Methods.

Authentication Workflow

The authentication workflow in OpenBao from a user perspective is as follows:

A client wants to authenticate to OpenBao and provides credentials.
OpenBao validates the credentials against the authentication method. For example: LDAP
If the authentication method (e.g. LDAP) is successful, it will return the required information about the client.
OpenBao maps this result to policies that are mapped to the authentication method.
OpenBao generates a token that is associated with the policies and returns it to the client.
The client can then use this token for further operations.

Prerequisites for Authentication Methods

Before configuring authentication:

OpenBao is deployed and unsealed (Parts 2-6)
You have admin access to OpenBao (root token)
Access to the required authentication backend:
- For Kubernetes auth: Access to the OpenShift cluster
- For LDAP: Access to the LDAP server
- etc.

Set up your environment:

You can set up your environment by using either the CLI or the UI.

Using the CLI

You can use the CLI to interact with OpenBao. In this example we will forward the port locally to the OpenBao server and set the environment variable BAO_ADDR to the local address. With boa login and the root token you can authenticate against OpenBao with full access.

# Port forward (if needed)
oc port-forward svc/openbao 8200:8200 -n openbao &
# Set environment
export BAO_ADDR='http://127.0.0.1:8200'
# You may need the SSL CA too.
# export BAO_CACERT="$PWD/openbao-ca.crt"
# Login with root token
bao login
# List all authentication methods
bao auth list

The last command returns a list of the available authentication methods. Currently there is only one authentication method enabled: token.

Path Type Accessor Description Version
---- ---- -------- ----------- -------
token/ token auth_token_1020fa7b token based credentials n/a

Using the UI

The UI, if enabled, is accessible via the OpenShift Route (if running on OpenShift and if it was created in Part 3). The password/token is the root token from the initialisation.

Figure 1. OpenBao UI Login Form

The UI has the advantage that it offers more visibility into the configuration and possibilities. For example, at the beginning there is one authentication method token enabled. This is required for the root token authentication.

Figure 2. OpenBao UI Authentication Methods

We can enable the other authentication methods by clicking on the Enable new method link to see what options are available.

Figure 3. OpenBao UI Configuring New Authentication Method

Understanding Policies

Before you can actually use an authentication method, it is important to understand what policies do. A policy is a way to declaratively define what (which paths) authenticated users can access or not. All paths are denied by default. A policy is mapped to an authentication method. For example, when we look at LDAP, we could configure something like this:

Every member of group "dev" is mapped to a policy named "dev-policy". The policy then allows the user to read the secrets under the path "secret/data/dev/*".

Basic Policy Structure

A policy is a set of path rules. Each rule says: “for this path (or path prefix), the bearer of this policy may use these capabilities.” Paths are tied to OpenBao’s internal API: for example, secrets in the KV v2 engine live under secret/data/<mount-path>/<key>, so a path like secret/data/myapp/* means “any key under the myapp prefix in the KV v2 store mounted at secret/.”

The example below does two things that are typical for an application policy:

Secrets access: It allows read and list on secret/data/myapp/*.
- read lets the client fetch the value of a secret at a path (e.g. secret/data/myapp/database).
- list is required to list keys under a path (e.g. to discover myapp/database, myapp/api-key); without it, the client would need to know every path in advance.
Token renewal: It allows update on auth/token/renew-self. Tokens often have a limited lifetime; this path lets the holder extend their own token without needing the root token or extra permissions. Including it avoids applications losing access when the token expires.

Policies can be written in JSON or HCL.

# Example policy: myapp-read-policy.hcl
# Allow reading secrets from specific path (KV v2 engine at mount "secret/")
path "secret/data/myapp/*" {
capabilities = ["read", "list"]
}
# Allow renewing own token so the app can extend its token before expiry
path "auth/token/renew-self" {
capabilities = ["update"]
}

The path secret/data/myapp/* assumes you have a KV v2 secrets engine mounted at secret/ and will store application secrets under keys like myapp/database, myapp/api-key, etc. Adjust the mount path and prefix to match your setup.

The use of globs (*) may result in surprising or unexpected behaviour. Use them with caution.

Policy Capabilities

Each path defined in a policy must have at least one capability. This provides control over the allowed or denied operations a user may perform. There are several capabilities available:

create - Create new data at the given path
read - Read data
update - Update existing data
patch - Patch existing data (Partial update)
delete - Delete data
list - List keys at a path
scan - Scan or browse the path for keys
sudo - Allows access to paths that are root-protected.
deny - Explicitly deny (overrides other grants)

Default Policies

When OpenBao is deployed the first time, there are no additional authentication methods enabled, except for token. This is required for the root access itself. To allow the administrator to access using the root token, there is one default policy called root. This policy is somewhat of a catch-all policy; it allows the administrator to access everything.

In addition to the root policy, there is one other default policy called default. This policy is used for all authenticated users. It allows them, for example, to look up their own properties and to renew or revoke their own token.

Let’s look at the default policies in the CLI:

A quick reminder of how to log in to OpenBao using the CLI when running it against Kubernetes: first open a port-forward, set the variable BAO_ADDR to the local address; if you use HTTPS you may need to set the SSL CA certificate, then log in with the root token.

You need to have the CA certificate in the current directory, which you might fetch with:

oc get secret openbao-ca-secret -n openbao -o jsonpath='{.data.ca\.crt}' | base64 -d > openbao-ca.crt

Port forward, set the environment variables and login with the root token:

# Port forward (if needed)
oc port-forward svc/openbao 8200:8200 -n openbao &
# Set environment
export BAO_ADDR='https://127.0.0.1:8200'
# You may need the SSL CA too.
export BAO_CACERT="$PWD/openbao-ca.crt"
# Login with root token
bao login

Now to list and fetch the current policies you can use the following commands:

# List policies
bao policy list
# Returns something like this:
#default
#root
# Read a policy
bao policy read default

The second command returns the policy content:

# Allow tokens to look up their own properties
path "auth/token/lookup-self" {
capabilities = ["read"]
}
# Allow tokens to renew themselves
path "auth/token/renew-self" {
capabilities = ["update"]
}
# Allow tokens to revoke themselves
path "auth/token/revoke-self" {
capabilities = ["update"]
}
# Allow a token to look up its own capabilities on a path
path "sys/capabilities-self" {
[...]

Again you will see the same policies in the UI. You can also manage them or create new ones using the UI.

Figure 4. OpenBao UI Listing Policies

Creating Policies using the CLI

Creating a policy, for example using the CLI, is straightforward. You can create a policy from a file or inline. The syntax is either JSON or HCL. The example below creates a policy called "myapp-read" from a file called "myapp-read-policy.hcl". The policy allows reading and listing secrets under the path "secret/data/myapp/*".

# Create a policy from file
bao policy write myapp-read myapp-read-policy.hcl
# Or inline
bao policy write myapp-read - <<EOF
path "secret/data/myapp/*" {
capabilities = ["read", "list"]
}
EOF
# List policies
bao policy list
# Read a policy
bao policy read myapp-read

Kubernetes Authentication Method

As we work a lot with OpenShift/Kubernetes, this method of authentication will be the first to try. The Kubernetes auth method can be used to authenticate against OpenBao using Kubernetes Service Account tokens. It is essential for pods to access secrets.

Let us try a simple example.

Step 1: Enable Kubernetes Auth

Assuming you are still logged into OpenBao, you can enable the kubernetes auth method with the following command:

bao auth enable --description="Authentication method for Kubernetes/OpenShift cluster" kubernetes

This simply enabled the authentication method and set a description. However, we need to configure it further.

Step 2: Configure Kubernetes Auth

To configure the kubernetes auth method, you need to get the Kubernetes API server address. You can get it with the following command (assuming you are logged into the OpenShift cluster):

The command oc can be replaced by kubectl if you are not using an OpenShift cluster.

# Get the Kubernetes API server address
KUBERNETES_HOST=$(oc whoami --show-server) (1)
# Configure the auth method
bao write auth/kubernetes/config \ (2)
kubernetes_host="$KUBERNETES_HOST"

1	Get the Kubernetes API server address. This will use something like https://api.cluster.example.com:6443
2	Configure the auth method with the Kubernetes API server address.

When running inside Kubernetes, OpenBao automatically uses the pod’s service account to communicate with the Kubernetes API.

Step 3: Create a Policy for the Application

Now we need to associate a policy to the authentication method. We could also assign the default policy but we want to provide access to the path secret/data/expense/database and secret/metadata/expense/config for the application.

# Create policy for an example application
bao policy write expense-app - <<EOF
# Read database credentials
path "secret/data/expense/database" {
capabilities = ["read"]
}
# Read application config
path "secret/data/expense/config" {
capabilities = ["read", "list"]
}
# Allow token renewal
path "auth/token/renew-self" {
capabilities = ["update"]
}
EOF

Step 4: Create a Kubernetes Auth Role

As the next step we map the policy to the authentication method. We will further limit the access to the namespace expense and the service account expense-app.

# Create a role that maps Kubernetes service accounts to OpenBao policies
bao write auth/kubernetes/role/expense-app \
bound_service_account_names=expense-app \ (1)
bound_service_account_namespaces=expense \ (2)
policies=expense-app \ (3)
ttl=1h \ (4)
max_ttl=24h (5)

1	`bound_service_account_names`: K8s service account name(s)
2	`bound_service_account_namespaces`: K8s namespace(s)
3	`policies`: OpenBao policies to attach
4	`ttl`: Token time-to-live
5	`max_ttl`: Maximum token lifetime

All of the configuration above, created using the CLI, can also be done using the UI of OpenBao.

Ensure the KV v2 secrets engine is mounted at `secret/`

We will discuss secret engines in more detail in the next part of the guide.

The policy and demo commands in this guide use the path secret/data/expense/database, which assumes a KV v2 secrets engine mounted at secret/. To check and enable it:

Check existing mounts: List enabled secrets engines and look for secret/ with type kv (version 2):
```
# List all secrets engine mounts (requires a token with read on sys/mounts)
bao secrets list
```
Look for an entry like secret/ with type kv or kv-v2. If you see secret/ and it is KV v2, you can skip to Step 5.
Enable KV v2 at secret/ if missing: If secret/ is not listed, or you want to use a fresh mount, enable the KV v2 engine at the path secret:
```
# Enable KV v2 secrets engine at path "secret" (requires root or policy with sys/mounts capability)
bao secrets enable -path=secret -version=2 kv
```
If secret/ already exists as KV v1, you cannot change it in place to v2; use a different path (e.g. secretv2/) and update the policy and demo paths accordingly (secretv2/data/expense/database, etc.).

Step 5: Store the database secret in OpenBao (one-time)

Before the application can read it, store the database password (and any other keys) at the path the policy allows. Run this once from a machine that has OpenBao access (e.g. bao CLI and a token).

You must use a token that has create and update capability on the KV path. The expense-app policy is read-only (for the application). Use the root token from OpenBao initialisation, or log in with bao login and use a token that has a policy granting create and update on secret/data/expense/*. If you use a read-only token (e.g. one that only has the expense-app policy), you will get a 403 "preflight capability check returned 403".

# Store the database secret at secret/data/expense/database (KV v2 path)
bao kv put secret/expense/database password="my-super-secret-db-password" username="expense_db_user"

This creates (or overwrites) the secret at secret/data/expense/database. The policy grants the expense-app role read on this path only; the application cannot create or update it.

Step 6: Create the expense namespace and demo application

Apply the following manifests to create the namespace, service account, optional long-lived token (Kubernetes 1.24+), and a demo deployment that authenticates to OpenBao and fetches the database secret. Adjust the OpenBao URL if your OpenBao is in another namespace or uses HTTPS (e.g. https://openbao.openbao.svc:8200).

Remember: We have created a policy that allows the service account with the name expense-app in the namespace expense to read the secret at secret/data/expense/database. If you want to use different paths or namespaces, you need to adjust the policy accordingly.

# full-demo-expense-app.yaml
# 1. Namespace
apiVersion: v1
kind: Namespace (1)
metadata:
name: expense
---
# 2. Service account used by the demo app to authenticate to OpenBao
apiVersion: v1
kind: ServiceAccount (2)
metadata:
name: expense-app
namespace: expense
---
# 3. Long-lived token for Kubernetes 1.24+ (optional; allows pods to use the SA token)
apiVersion: v1
kind: Secret (3)
metadata:
name: expense-app-token
namespace: expense
annotations:
kubernetes.io/service-account.name: expense-app
type: kubernetes.io/service-account-token
---
# 4. Demo deployment: authenticates to OpenBao and fetches secret/data/expense/database
apiVersion: apps/v1
kind: Deployment
metadata:
name: expense-demo
namespace: expense
labels:
app: expense-demo
spec:
replicas: 1
selector:
matchLabels:
app: expense-demo
template:
metadata:
labels:
app: expense-demo
spec:
serviceAccountName: expense-app
containers:
- name: demo
image: registry.access.redhat.com/ubi9/ubi-minimal:latest
command:
- /bin/sh
- -c
- |
echo "=== Installing curl ==="
microdnf install -y curl 2>/dev/null || true
echo "=== Fetching database secret from OpenBao ==="
JWT=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
OPENBAO_URL="${OPENBAO_URL:-http://openbao.openbao:8200}"
RESP=$(curl -s -k -X POST -H "Content-Type: application/json" \ (4)
-d "{\"jwt\": \"$JWT\", \"role\": \"expense-app\"}" \
"$OPENBAO_URL/v1/auth/kubernetes/login")
TOKEN=$(echo "$RESP" | sed -n 's/.*"client_token":"\([^"]*\)".*/\1/p')
if [ -z "$TOKEN" ]; then
echo "ERROR: Failed to get OpenBao token. Check OpenBao URL ($OPENBAO_URL), role expense-app, and network."
echo "Response: $RESP"
exit 1
fi
echo "OpenBao token obtained. Reading secret/data/expense/database ..."
SECRET=$(curl -s -k -H "X-Vault-Token: $TOKEN" "$OPENBAO_URL/v1/secret/data/expense/database") (5)
echo "$SECRET"
PASSWORD=$(echo "$SECRET" | sed -n 's/.*"password":"\([^"]*\)".*/\1/p')
echo "Database password (data.data.password): ${PASSWORD:-<not set>}"
echo "=== Demo complete. Pod stays running; exec in to run more curl commands. ==="
exec sleep infinity
env:
# Override if OpenBao is in another namespace or uses HTTPS
- name: OPENBAO_URL
value: "https://openbao.openbao:8200"
resources:
requests:
memory: "64Mi"
cpu: "10m"

1	Create the namespace
2	Create the service account
3	Create the secret (long-lived token for Kubernetes 1.24+)
4	Authenticate to OpenBao and get a client token (using the Kubernetes JWT)
5	Fetch the database secret

The examples above use curl -k to bypass certificate verification. For production, you should adjust this and mount the CA certificate in the pod (see the note earlier in this section on using the openbao-ca Secret).

Apply the manifests and wait for the pod to be ready:

oc apply -f full-demo-expense-app.yaml
# Wait for the demo pod to be running
oc -n expense rollout status deployment/expense-demo
# View the log: you should see the OpenBao token response and the fetched secret (including the password)
oc -n expense logs -l app=expense-demo -f

The log output shows the application authenticating with the Kubernetes JWT, receiving an OpenBao token, and reading the secret at secret/data/expense/database. The field data.data.password is the database password you stored in Step 5.

Step 7: Test Kubernetes authentication manually

From any pod that uses the same service account (e.g. the demo pod above), you can run the same steps by hand:

# From within a pod using the service account (e.g. oc exec -it deployment/expense-demo -n expense -- /bin/sh)
JWT=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
# 1. Authenticate to OpenBao and get a client token (sed works on UBI minimal; use jq -r .auth.client_token if jq is available)
LOGIN=$(curl -s -k --request POST \
--data "{\"jwt\": \"$JWT\", \"role\": \"expense-app\"}" \
https://openbao.openbao:8200/v1/auth/kubernetes/login)
TOKEN=$(echo "$LOGIN" | sed -n 's/.*"client_token":"\([^"]*\)".*/\1/p')
# 2. Fetch the database secret (same path the policy allows)
curl -s -k -H "X-Vault-Token: $TOKEN" https://openbao.openbao:8200/v1/secret/data/expense/database

Summary: Kubernetes auth method

A lot of steps were involved to configure the Kubernetes auth method. Let us summarize them:

In OpenBao: Enabled the Kubernetes auth method, configured it with the cluster API server address, created the expense-app policy (read on secret/data/expense/database and secret/data/expense/config, update on auth/token/renew-self), and created the expense-app role that binds the Kubernetes service account expense-app in namespace expense to that policy.
Secrets engine: Ensured the KV v2 engine is mounted at secret/ and stored the database secret at secret/expense/database (one-time, using the root token).
In the cluster: Created the expense namespace, the expense-app service account, an optional long-lived token Secret (for Kubernetes 1.24+), and the expense-demo deployment that uses that service account to authenticate to OpenBao and fetch the secret. For HTTPS, created the openbao-ca Secret in expense with the OpenBao server CA.
Verification: Applied the manifests, confirmed the demo pod starts and logs show a successful login and the fetched secret (including the database password). Optionally ran the same login and read steps manually via oc exec into the pod.

LDAP Authentication Method

LDAP auth integrates with enterprise directories like Active Directory.

This is a very simplified example. As a prerequisite you need to have an LDAP server and a service account with the necessary permissions to read the users and groups. This is not part of this guide.

Step 1: Enable LDAP Auth

To enable LDAP authentication, you need to use the following command:

bao auth enable ldap

Step 2: Configure LDAP (Active Directory Example)

In this step you need to configure the LDAP server address, the user and group attributes, the bind DN and the bind password. The example below configures the LDAP server address to ldaps://ad.example.com:636, the user attribute to sAMAccountName, the user DN to OU=Users,DC=example,DC=com, the group DN to OU=Groups,DC=example,DC=com, the group attribute to cn, the group filter to (&(objectClass=group)(member:1.2.840.113556.1.4.1941:={{.UserDN}})), the bind DN to CN=openbao-svc,OU=ServiceAccounts,DC=example,DC=com and the bind password to service-account-password.

bao write auth/ldap/config \
url="ldaps://ad.example.com:636" \
userattr="sAMAccountName" \
userdn="OU=Users,DC=example,DC=com" \
groupdn="OU=Groups,DC=example,DC=com" \
groupattr="cn" \
groupfilter="(&(objectClass=group)(member:1.2.840.113556.1.4.1941:={{.UserDN}}))" \
binddn="CN=openbao-svc,OU=ServiceAccounts,DC=example,DC=com" \
bindpass="service-account-password" \
certificate=@/path/to/ldap-ca.pem \
insecure_tls=false \
starttls=false

Step 3: Create the policies used by the group mapping

Before mapping LDAP groups to policy names, create those policies. Below are example definitions: admin-policy grants read/list/update on secrets (e.g. for operators), and user-policy grants read-only access to a limited path.

admin-policy — for members of openbao-admins (e.g. operators who may read and update secrets)

user-policy — for members of openbao-users (e.g. developers with read-only access to a subset of secrets)

Create them with the CLI (run before the group mapping in Step 4):

# Create admin-policy (e.g. from file admin-policy.hcl)
bao policy write admin-policy - <<EOF
path "secret/data/*" {
capabilities = ["read", "list", "update", "create", "delete"]
}
path "secret/metadata/*" {
capabilities = ["read", "list"]
}
path "auth/token/renew-self" {
capabilities = ["update"]
}
EOF
# Create user-policy
bao policy write user-policy - <<EOF
path "secret/data/myapp/*" {
capabilities = ["read", "list"]
}
path "auth/token/renew-self" {
capabilities = ["update"]
}
EOF

Adjust paths (e.g. secret/data/myapp/) to match your KV v2 mount and the paths you want each role to access. The default policy is built-in and grants token lookup/renew/revoke-self. Attaching it in addition to admin-policy or user-policy is a typical pattern.

Step 4: Map LDAP groups to policies

Now map the LDAP groups to the policy names. The example below maps the LDAP group openbao-admins to the policies admin-policy and default and the LDAP group openbao-users to the policies user-policy and default.

# Map an LDAP group to policies
bao write auth/ldap/groups/openbao-admins \
policies="admin-policy,default"
bao write auth/ldap/groups/openbao-users \
policies="user-policy,default"

Step 5: Test LDAP authentication

Assuming you have a user jdoe in the LDAP group openbao-users, you can test the LDAP authentication with the following commands:

# Login with LDAP credentials
bao login -method=ldap username=jdoe
# Or via API
curl --request POST \
--data '{"password": "user-password"}' \
$BAO_ADDR/v1/auth/ldap/login/jdoe

What jdoe can and cannot do (user-policy)

jdoe holds user-policy (and default), so their token has only the capabilities defined there.

What jdoe can do:

Get a secret under secret/data/myapp/*: after logging in, jdoe can read and list secrets in that path. For example:

# After bao login -method=ldap username=jdoe (and entering the password)
bao kv get secret/myapp/database
# Or via API (use the client_token from the login response)
curl -s -H "X-Vault-Token: $TOKEN" $BAO_ADDR/v1/secret/data/myapp/database

List keys under secret/data/myapp/ to discover available secrets.
Renew or revoke their own token (from the default policy) and look up their own token properties.

What jdoe cannot do:

Read secrets outside myapp — e.g. secret/data/expense/database or secret/data/other-app/* returns permission denied.
Create, update, or delete any secret — user-policy grants only read and list on secret/data/myapp/*, so write operations are denied.
Access admin or system paths — no access to sys/, other auth methods, or policies. Only the paths explicitly granted in *user-policy and default are allowed.

Common Practices for Authentication

Use the principle of least privilege - Do not grant more permissions than necessary.

# Bad: Too broad
path "secret/*" {
capabilities = ["read", "list"]
}
# Good: Specific paths
path "secret/data/myapp/database" {
capabilities = ["read"]
}

Set Appropriate Token TTLs - Set the appropriate TTLs for different use cases.

# Short-lived for automated systems
bao write auth/kubernetes/role/batch-job \
ttl=5m \
max_ttl=15m
# Longer for interactive users
bao write auth/oidc/role/human-user \
ttl=8h \
max_ttl=24h

Enable Audit Logging - Enable audit logging to track authentication attempts and other activities.

# Enable file audit
bao audit enable file file_path=/var/log/openbao/audit.log
# All authentication attempts are logged

Regularly Rotate Credentials - Rotate the credentials periodically to reduce the risk of compromise.

# Rotate AppRole Secret IDs periodically
bao write -f auth/approle/role/cicd-role/secret-id
# Revoke old tokens
bao token revoke <old-token>

Common Authentication Patterns

Pattern 1: Application Pods

Pod → Service Account Token → Kubernetes Auth → Policy → Secret

Pattern 2: Human Users

User → LDAP/OIDC Login → Group Mapping → Policy → Secret

Pattern 3: CI/CD Pipeline

Pipeline → AppRole (Role ID + Secret ID) → Policy → Secret

What is Coming Next?

In Part 8, we will explore secrets engines, like:

KV (Key-Value) for static secrets

In upcoming parts, I will try to cover common practices for OpenBao operation and maintenance. If there is time and an opportunity to test other authentication methods, I will do so.

Conclusion

Proper authentication configuration is crucial for OpenBao security. Key takeaways:

Use Kubernetes auth for pods
Use LDAP/OIDC for human users (SSO)
Use AppRole for automation
Create specific policies with least privilege
Set appropriate TTLs for different use cases
Enable audit logging for compliance

Resources

The Guide to OpenBao - Initialisation, Unsealing, and Auto-Unseal - Part 6

Thu, 05 Mar 2026 00:00:00 +0000

After deploying OpenBao via GitOps (Part 5), OpenBao must be initialised and then unsealed before it becomes functional. You usually do not want to do this unsealing manually, since this is not scalable especially in bigger, productive environments. This article explains how to handle initialisation and unsealing, and possible options to configure an auto-unseal process so that OpenBao unseals itself on every restart without manual key entry.

What is happening?

OpenBao starts in a sealed state. You must:

Initialise (once): run bao operator init to generate unseal keys (or recovery keys when using auto-unseal).
Unseal (after each restart): provide a threshold of unseal keys so each node can decrypt the root key. In the previous articles, we used a threshold of 3 (out of 5). This means that you need to run bao operator unseal three times with different keys on every node after every start before the service becomes available.

Unseal workflow (assuming a threshold of 3 and 5 nodes and initialisation already happened):

---
title: "Unseal Workflow"
config:
theme: 'dark'
---
flowchart LR
A["openbao-0 starts → 3× unseal using different keys"]
A --> B["openbao-1 starts → 3× unseal using different keys"]
B --> C["openbao-2 starts → 3× unseal using different keys"]
C --> D[OpenBao is ready]

This is a chicken-egg problem: you need to start somewhere, but you cannot start without the keys.

Manual unsealing does not scale very well and for production, auto-unseal is recommended.

This article covers:

Practical approaches to initialisation and unsealing (manual, Jobs, init containers).
Potential auto-unseal options: with AWS KMS, static key, and transit seal.

Handling Initialisation and Unsealing

The challenge with GitOps is that initialisation and unsealing are one-off or stateful operations. Below are several approaches. The problem is always the same: where do you start? You must have the unseal keys available to start the process. Do you keep them locally, do you keep them in a Kubernetes Secret, do you use an external Key Management System (KMS)? The most robust for production in my opinion is auto-unseal using a KMS (see [_approach_4_auto_unseal_with_kms_recommended_for_production] and the following sections). It uses an external Key Management System (KMS) to store the keys. This way, the keys are not stored in the cluster and are not exposed to the risk of being compromised.

Let’s have a look at the different options.

Approach 1: Manual Initialisation (simplest)

While manual initialisation and unsealing is the simplest approach, it is not recommended for production. Still, I would like to mention it here for completeness and at least provide a simple for loop to unseal the OpenBao service (on Kubernetes/OpenShift).

Manual unsealing is still a valid option when you have a small cluster and assume that it is not required often.

After Argo CD deploys OpenBao, the first pod, openbao-0, will not become ready until it is initialised and unsealed. Once done, the next pod, openbao-1, will try to start and wait until it is unsealed, and then the third pod, and so on.

The following, simplified script will go through the pods and unseal them one by one. It initialises the first pod and stores the unseal keys in the file openbao-init.json locally on your machine. From there it takes three different keys and unseals the OpenBao service. Since it takes a while until other pods are pulling the image and starting, we will use a sleep timer of 30 seconds between each pod. This may or may not be enough depending on your network and cluster speed. As said, this is a very simplified script:

The CLI tool oc can be replaced by kubectl in case you are using a different cluster. In addition, you will need the jq command line tool to parse the JSON file.

If your OpenBao was initialised previously, you can remove the first three commands and start with the unseal commands.

# Initialise the first pod, assuming the Pod openbao-0 is already running.
# You can skip this if OpenBao was initialised previously.
echo "Initialising OpenBao on openbao-0 (keys will be saved to openbao-init.json)..."
oc exec -it openbao-0 -n openbao -- bao operator init \
-key-shares=5 -key-threshold=3 -format=json > openbao-init.json (1)
echo "Init complete. Unseal keys and root token are in openbao-init.json."
# Unseal each pod (three times each with different keys)
echo "Unsealing pods openbao-0, openbao-1, openbao-2 (3 key steps each)..."
for i in 0 1 2; do
sleep_timer=30
SLEEPER_TMP=1
echo "Unsealing openbao-$i..."
for j in 1 2 3; do (2)
KEY=$(cat openbao-init.json | jq -r ".unseal_keys_b64[$((j-1))]")
oc exec -it openbao-$i -n openbao -- bao operator unseal $KEY
done
echo "openbao-$i unsealed. Waiting ${sleep_timer}s for next pod to be ready..."
while [[ $SLEEPER_TMP -le "$sleep_timer" ]]; do (3)
if (( SLEEPER_TMP % 10 == 0 )); then
echo -n "$SLEEPER_TMP"
else
echo -n "."
fi
sleep 1
SLEEPER_TMP=$((SLEEPER_TMP + 1))
done
echo ""
done
echo "All pods unsealed. Store init data securely (e.g. in a password manager)."
# Store init data securely (e.g. in a password manager)

1	Initialise the first pod, assuming the Pod openbao-0 is already running. This will store the keys (including the root token) in the file openbao-init.json.
2	Unseal each pod (three times each with different keys). The keys are taken from the openbao-init.json file.
3	Wait 30 seconds between each pod to give the other pods time to start and pull the image … super-fancy sleep timer.

The unseal keys as well as the root token are stored in the openbao-init.json file. This file is stored locally on your machine. Make sure to store it securely and do not share it with anyone.

Approach 2: Kubernetes Job for initialisation

A Job can run after OpenBao is deployed, initialise it if needed, and store the init output (unseal keys and root token) in a Kubernetes Secret. You can then unseal manually or with a second Job that reads from that Secret. This section assumes the init output is stored in a Kubernetes Secret and explains how to create that Secret from the Job and how to use it.

How the Secret is created

The init Job runs bao operator init and then creates the Secret using oc (or kubectl). The Secret name and key used here are openbao-init-data and init.json (the file contains the JSON output of bao operator init, including unseal_keys_b64 and root_token). One advantage of this approach is that you can integrate it into your GitOps pipeline. However, we can argue whether it is a good idea to store the unseal keys and the root token as a Secret in the same cluster where OpenBao is running.

Example structure of init.json:

{
"unseal_keys_b64": [
"key1",
"key2",
"key3",
"key4",
"key5"
],
"unseal_keys_hex": [
"hex1",
"hex2",
"hex3",
"hex4",
"hex5"
],
"unseal_shares": 5,
"unseal_threshold": 3,
"recovery_keys_b64": null,
"recovery_keys_hex": null,
"recovery_keys_shares": 0,
"recovery_keys_threshold": 0,
"root_token": "root.token"
}

You need at least unseal_threshold keys (e.g. 3) from unseal_keys_b64 to unseal each node. Store this file securely; anyone with access can unseal OpenBao and the root_token has full admin access.

RBAC Configuration

The Job’s service account (openbao-init) must be allowed to create (and optionally get/update) Secrets in the openbao namespace. Create a Role and RoleBinding:

apiVersion: v1
kind: ServiceAccount
metadata:
name: openbao-init (1)
namespace: openbao
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: openbao-init-secret-writer
namespace: openbao
rules: (2)
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create", "get", "update", "patch"]
- apiGroups: [""] (3)
resources: ["pods"]
verbs: ["get", "list"]
- apiGroups: [""] (4)
resources: ["pods/exec"]
verbs: ["create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding (5)
metadata:
name: openbao-init-secret-writer
namespace: openbao
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: openbao-init-secret-writer
subjects:
- kind: ServiceAccount
name: openbao-init
namespace: openbao

1	The ServiceAccount is used by the Job(s) to create the Secret and unseal the OpenBao pods.
2	The Role is used by the Job to create the Secret. The rules might be extended to include the permission to execute into the pods to unseal the OpenBao service (see below).
3	Permissions to get and list the Pods, required for the unseal process later
4	Permissions to execute into the Pods, required for the unseal process later
5	The RoleBinding is used by the Job to create the Secret.

The Job must also be able to reach the OpenBao API (e.g. openbao.openbao.svc:8200). This is usually possible when running in the same namespace as the OpenBao service.

The example below shows a working Job manifest that will perform the initialisation and the creation of the Secret in the openbao namespace. The manifest is already prepared for Argo CD with useful annotations. It runs an init container to initialise the OpenBao service and a main container to create the Secret. The init container writes (using the OpenBao CLI) the init.json file to a shared volume and the main container (using the kubectl command) creates the Secret from that file. In addition, the Secret containing the CA certificate is mounted as a volume and the BAO_CACERT environment variable is set to the path of the CA certificate.

Init Job that creates the Secret

apiVersion: batch/v1
kind: Job
metadata:
name: openbao-init
namespace: openbao
annotations: (1)
argocd.argoproj.io/sync-wave: "30"
argocd.argoproj.io/hook: PostSync
argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
template:
spec:
serviceAccountName: openbao-init (2)
initContainers: (3)
- name: openbao-init
image: ghcr.io/openbao/openbao:latest
command: (4)
- /bin/sh
- -c
- |
if [ -f /etc/openbao-ca/ca.crt ]; then
export BAO_CACERT=/etc/openbao-ca/ca.crt
export BAO_ADDR=https://openbao:8200
echo "Using TLS CA from /etc/openbao-ca/ca.crt"
else
export BAO_ADDR=http://openbao:8200
echo "Using plain HTTP"
fi
until bao status 2>&1 | grep -q "Initialized"; do
echo "Waiting for OpenBao..."
sleep 5
done
if bao status | grep -q "Initialized.*false"; then
echo "Initialising OpenBao..."
bao operator init -key-shares=5 -key-threshold=3 \
-format=json > /shared/init.json
echo "Init complete; init.json written to shared volume."
else
echo "OpenBao already initialised (skipping init)."
fi
volumeMounts:
- name: openbao-ca
mountPath: /etc/openbao-ca
readOnly: true
- name: shared
mountPath: /shared
containers: (5)
- name: create-secret
image: bitnami/kubectl:latest
command:
- /bin/sh
- -c
- |
if [ -f /shared/init.json ]; then
echo "Creating Secret openbao-init-data from init.json..."
kubectl create secret generic openbao-init-data \
--from-file=init.json=/shared/init.json \
-n openbao --dry-run=client -o yaml | kubectl apply -f -
echo "Initialisation complete!"
else
echo "No init.json (OpenBao already initialised or init skipped)."
fi
volumeMounts:
- name: shared
mountPath: /shared
readOnly: true
volumes:
- name: openbao-ca
secret:
secretName: openbao-ca-secret
optional: true
- name: shared (6)
emptyDir: {}
restartPolicy: Never
backoffLimit: 3

1	The annotations are used by Argo CD to trigger the Job after the OpenBao deployment.
2	The ServiceAccount is used by the Job to initialise the OpenBao service.
3	The init container runs the OpenBao CLI (wait for API, then operator init) and writes init.json to a shared volume.
4	The command is used to initialise the OpenBao service.
5	The main container runs kubectl to create the Secret from that file. This avoids needing a single image that bundles both `bao` and `kubectl`.
6	The shared volume is used to store the init.json file.

As a result the secret openbao-init-data is created in the openbao namespace. It contains the unseal keys and the root token.

Using the Secret in a Job (unseal)

A second Job can use the Secret openbao-init-data that was created by the init Job to unseal each OpenBao pod. A lot is happening here:

The init container extracts the unseal keys from the Secret using jq and writes them to a shared volume.
The main container is reading the unseal keys from the shared volume and unseals the first pod
Then the process is waiting for 30 seconds, to give the next pod time to pull the image and start
The process is repeated for the next pod and so on until all pods are unsealed.

First of all, the unseal Job needs permission to exec into the OpenBao pods. Add a Role that grants the Job’s service account create on pods/exec in the openbao namespace. To make it easier, we can extend the Role openbao-init-secret-writer that was created in the init Job.

These rules were already applied with the configuration for the init Job, but we will repeat them here for completeness.

rules: (1)
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"]

1	The role `openbao-init-secret-writer` must be extended to include the permission to execute into the pods to unseal the OpenBao service.

apiVersion: batch/v1
kind: Job
metadata:
name: openbao-unseal
namespace: openbao
annotations:
argocd.argoproj.io/sync-wave: "35"
argocd.argoproj.io/hook: PostSync
argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
template:
spec:
serviceAccountName: openbao-init (1)
initContainers:
- name: extract-keys
image: quay.io/codefreshplugins/curl-jq
command:
- /bin/sh
- -c
- |
if [ ! -f /secrets/init.json ]; then
echo "Secret not found; run init Job first."
exit 1
fi
for i in 0 1 2; do
jq -r ".unseal_keys_b64[$i]" /secrets/init.json | tr -d '\n' > "/shared/key$i" (2)
done
echo "Unseal keys extracted to shared volume."
volumeMounts:
- name: init-secret
mountPath: /secrets
readOnly: true
- name: shared
mountPath: /shared
containers: (3)
- name: unseal
image: bitnami/kubectl:latest
command: (4)
- /bin/sh
- -c
- |
if [ -f /etc/openbao-ca/ca.crt ]; then
export BAO_CACERT=/etc/openbao-ca/ca.crt
export BAO_ADDR=https://openbao:8200
echo "Using TLS CA from /etc/openbao-ca/ca.crt"
else
export BAO_ADDR=http://openbao:8200
echo "Using plain HTTP"
fi
sleep_timer=30
if [ ! -f /shared/key0 ]; then
echo "Keys not found; extract-keys init container may have failed."
exit 1
fi
echo "Unsealing pods openbao-0, openbao-1, openbao-2 (3 key steps each), ${sleep_timer}s delay between pods..."
for pod in openbao-0 openbao-1 openbao-2; do
echo "Unsealing $pod..."
for i in 0 1 2; do
key=$(cat "/shared/key$i" | tr -d '\n')
kubectl exec -n openbao "$pod" -- sh -c 'bao operator unseal "$1"' _ "$key" || true
done
echo "$pod unsealed."
if [ "$pod" != "openbao-2" ]; then
echo "Waiting ${sleep_timer}s for next pod to be ready..."
SLEEPER_TMP=1
while [ "$SLEEPER_TMP" -le "$sleep_timer" ]; do
if [ $(( SLEEPER_TMP % 10 )) -eq 0 ]; then
echo -n "$SLEEPER_TMP"
else
echo -n "."
fi
sleep 1
SLEEPER_TMP=$(( SLEEPER_TMP + 1 ))
done
echo ""
fi
done
echo "Unseal complete."
volumeMounts:
- name: shared
mountPath: /shared
readOnly: true
- name: openbao-ca
mountPath: /etc/openbao-ca
readOnly: true
volumes: (5)
- name: init-secret
secret:
secretName: openbao-init-data
- name: shared
emptyDir: {}
- name: openbao-ca
secret:
secretName: openbao-ca-secret
optional: true
restartPolicy: Never
backoffLimit: 2

1	The ServiceAccount is used by the Job to unseal the OpenBao pods. Here we are using the same ServiceAccount as the one used to initialise the OpenBao service.
2	The init container extracts unseal keys from init.json using jq and writes them to a shared volume.
3	The main container runs kubectl to exec into each OpenBao pod and run `bao operator unseal` (the OpenBao image is used inside the target pods).
4	The command is used to unseal the OpenBao service.
5	The volumes that are mounted: init-secret (contains the unseal keys), shared (contains the extracted unseal keys, shared between containers), openbao-ca (contains the CA certificate).

Both Jobs are using the same ServiceAccount and therefore the same RBAC rules. It is also possible to use a different ServiceAccount for the unseal Job.

Store init data securely. Keeping unseal keys in a cluster Secret is convenient but less secure than auto-unseal or an external secrets manager. Restrict access to the openbao-init-data Secret (e.g. RBAC and network policies) and consider rotating or moving keys to a more secure store after bootstrap.

Approach 3: Auto-unseal with Key Management Service KMS (recommended for production)

Configure a KMS (e.g. AWS KMS, Azure Key Vault, GCP Cloud KMS, Transit etc.) so that OpenBao unseals itself on every restart. You initialise once and thereafter no manual unseal step is needed.

The rest of this article will mention the different options. However, I can only demonstrate a real example using the AWS KMS option.

Auto-unseal options overview

Without auto-unseal, you must run bao operator unseal with a threshold of keys after each restart. With auto-unseal, OpenBao uses a seal backend to protect the root key and unseals itself on startup.

Supported options:

Option

Use case

Notes

1. AWS KMS

AWS (EKS, OpenShift on AWS)

IRSA or IAM user; no keys in cluster with IRSA.

2. Azure Key Vault

Azure (AKS, OpenShift on Azure)

Service principal or managed identity.

3. Google Cloud KMS

GCP (GKE)

Service account key or Workload Identity.

5. Transit

Existing Vault/OpenBao as root of trust; or a dedicated seal-only OpenBao that exists only to unseal production

External transit engine; token in a Secret.

Why use OpenBao when the seal is in KMS? Why not keep all secrets in KMS and skip OpenBao?

When using a cloud KMS (or Key Vault) for auto-unseal, a natural question is: why not store and retrieve all secrets from the KMS and run without OpenBao at all? OpenBao remains useful for operators and applications for several reasons:

KMS is for keys, not a full secrets platform. AWS KMS, Azure Key Vault keys, and GCP Cloud KMS are built to encrypt and decrypt small blobs (e.g. data encryption keys, or OpenBao’s seal blob). They are not a general-purpose secrets store with versioning, path-based access, dynamic credentials, and per-request audit. OpenBao provides that layer: applications request secrets by path, get short-lived tokens, and you get a single place to define who can read what.
Dynamic secrets. OpenBao can generate short-lived credentials on demand (e.g. database users, cloud IAM roles, PKI certificates). The KMS does not create or rotate such credentials; it only holds keys. If you need “give this pod a DB password that expires in 1 hour,” that is OpenBao’s domain, not the KMS.
Fine-grained, identity-aware access. OpenBao supports auth methods (Kubernetes, OIDC, AppRole, LDAP) and path-based policies. You can say “this service account may read only secret/data/myapp/*.” The KMS has IAM or Key Vault RBAC, but not the same model of “one API, many identities, many paths” that applications and operators use day to day.
Audit and compliance. OpenBao logs every secret read and write with identity and path. That gives a clear audit trail of who accessed which secret and when. KMS audit logs key usage, which is different from “which application read which secret at what time” in a unified way.
Abstraction and portability. Applications talk to OpenBao’s API. You can move the seal or the storage backend (or even the cloud) without changing how applications request secrets. If you put everything directly in a cloud KMS, you tie every app to that vendor’s API and limits.
Encryption as a service (Transit). OpenBao’s Transit secrets engine lets applications encrypt data with a key without ever seeing the key—useful for application-level encryption and key rotation. KMS can do something similar, but OpenBao integrates that with the same auth and audit as the rest of your secrets.

In short: the KMS is the root of trust for the seal (so OpenBao can unseal itself). OpenBao is the secrets management platform that applications and operators use for storing, retrieving, generating, and auditing secrets. Both layers complement each other.

General flow (same for all options):

Set up the seal backend (KMS key, Key Vault, static key, or transit server).
Add the appropriate seal "…" stanza to OpenBao configuration (e.g. in Helm server.ha.raft.config).
Deploy OpenBao; pods start and remain sealed until initialised.
Run one-time initialisation: bao operator init -recovery-shares=5 -recovery-threshold=3.
From then on, restarts are automatically unsealed.

Option: AWS KMS

Best for: EKS, OpenShift on AWS (ROSA), or any Kubernetes on AWS.

Create a KMS key

Create a symmetric KMS key used only for the OpenBao seal (do not use for application data).

AWS WebUI:
- search for: KMS → Create key → Symmetric, Encrypt/Decrypt
- Optionally set an alias (e.g. openbao-unseal).
- OPTIONAL: Define key administrative permissions
- OPTIONAL: Define key usage permissions
- Edit key policy

AWS CLI: (requires the AWS CLI to be installed and configured)

aws kms create-key \
--description "OpenBao auto-unseal key" \
--key-usage ENCRYPT_DECRYPT
aws kms create-alias \
--alias-name openbao-unseal \
--target-key-id <key-id-from-above>

Note the Key ID or Alias and the region (e.g. us-west-1) from the output of the command.

Key policy for OpenBao

The OpenBao process needs permission to use the key. Create an IAM policy. In the UI you can configure this directly when creating the key. When using the CLI, create a policy.json file with the following content:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": "arn:aws:kms:<REGION>:<ACCOUNT_ID>:key/<KEY_ID>" (1)
}
]
}

1	Replace `<REGION>`, `<ACCOUNT_ID>`, and `<KEY_ID>` with your values (or use the key ARN).

Create an IAM user for the seal

When using static credentials (e.g. on OpenShift or when IRSA is not available), create a dedicated IAM user that has permission only to use the KMS key above. Use that user’s access keys in the Kubernetes Secret.

AWS WebUI:
- search for: IAM → Users → Create user
- User name: e.g. openbao-seal
- DO NOT select "Provide user access to the AWS Management Console" (programmatic access only)
- Create user
- Open the created user → Permissions → Add permissions → Create inline policy (or Attach policies → Create policy)
- In the policy editor, choose JSON and paste the policy from Key policy for OpenBao, replacing <REGION>, <ACCOUNT_ID>, and <KEY_ID> with your KMS key ARN or alias
- Name the policy (e.g. OpenBaoSealKMS) and attach it to the user
- Security credentials → Create access key → Application running outside AWS (or "Third-party service") → Create access key
- Save the Access key ID and Secret access key; you will put them in the Kubernetes Secret openbao-aws-credentials.

AWS CLI: (requires the AWS CLI to be installed and configured)

Create a policy file (e.g. openbao-seal-policy.json) with the same JSON as in Key policy for OpenBao, then create the user, attach the policy, and create access keys:

# Replace REGION, ACCOUNT_ID, and KEY_ID in the policy (or use alias: arn:aws:kms:REGION:ACCOUNT_ID:alias/openbao-unseal)
aws iam create-policy \
--policy-name OpenBaoSealKMS \
--policy-document file://openbao-seal-policy.json
aws iam create-user --user-name openbao-seal
aws iam attach-user-policy \
--user-name openbao-seal \
--policy-arn arn:aws:iam::ACCOUNT_ID:policy/OpenBaoSealKMS
aws iam create-access-key --user-name openbao-seal

The last command returns AccessKeyId and SecretAccessKey; store them in the Secret openbao-aws-credentials with key AWS_REGION set to your region (e.g. us-west-1).

Create the Secret

Create the Secret openbao-aws-credentials in the openbao namespace. It contains the access key, secret access key, and region.

apiVersion: v1
kind: Secret
metadata:
name: openbao-aws-credentials
namespace: openbao
type: Opaque
stringData: (1)
AWS_ACCESS_KEY_ID: <your-access-key>
AWS_SECRET_ACCESS_KEY: <your-secret-key>
AWS_REGION: <your-region>

1	Replace `<your-access-key>`, `<your-secret-key>`, and `<your-region>` with the values from the previous steps.

Helm / Argo CD configuration (AWS)

Add the seal "awskms" block inside the same HCL config that contains listener, storage "raft". In Helm values this is typically under openbao.server.ha.raft.config.

Be sure to update the region and key ID.

The values file we used for Argo CD can be found at: OpenBao Argo CD values file

openbao:
server:
extraSecretEnvironmentVars: (1)
- envName: AWS_ACCESS_KEY_ID
secretName: openbao-aws-credentials
secretKey: AWS_ACCESS_KEY_ID
- envName: AWS_SECRET_ACCESS_KEY
secretName: openbao-aws-credentials
secretKey: AWS_SECRET_ACCESS_KEY
- envName: AWS_REGION
secretName: openbao-aws-credentials
secretKey: AWS_REGION
ha:
raft:
config: |
ui = true
seal "awskms" { (2)
region = "us-east-1"
kms_key_id = "alias/openbao-unseal"
}
listener "tcp" {
tls_disable = 0
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/openbao/tls/openbao-server-tls/tls.crt"
tls_key_file = "/openbao/tls/openbao-server-tls/tls.key"
tls_min_version = "tls12"
}
storage "raft" {
path = "/openbao/data"
}
service_registration "kubernetes" {}

1	If you use static credentials (e.g. for OpenShift), inject them via a Secret.
2	This must be added. Be sure to update your region and key ID.

The seal "awskms" block alone is not enough. OpenBao must have AWS credentials available at runtime. If you see an error such as NoCredentialProviders: no valid providers in chain or error fetching AWS KMS wrapping key information, the pod has no credentials. You must either inject static credentials from a Secret (see below) or use IRSA so the pod can assume an IAM role.

Approach 4: Init container with external secret store

Another option is to use an init container that retrieves secrets from an external source. Two use cases can be considered here:

Shamir unseal: The init container fetches the unseal keys from an external secret manager and then runs bao operator unseal (or writes keys to a file used by a sidecar/unseal script). This is the classic “external secret store” idea (e.g. using AWS Secrets Manager, HashiCorp Vault, another Kubernetes cluster).
Transit seal token: The init container fetches the token that production OpenBao uses to call the dedicated unseal service (see Dedicated unseal service (seal-only OpenBao)). That token is written to a file or shared volume so the main OpenBao process can use it in seal "transit". The token never lives in a Kubernetes Secret in Git or in plain YAML; it is fetched at pod start from the external store.

Below is a generic placeholder for the init container.

Option: init Container example

The following options are generic and more theoretical. Unlike the other options, I could not test them in a real environment yet. This may be covered in a future article.

Generic placeholder (implement according to your secret store):

# Additional values for the Helm chart
server:
extraInitContainers:
- name: fetch-secrets
image: registry.access.redhat.com/ubi9/ubi-minimal:latest
command:
- /bin/sh
- -c
- |
# Fetch unseal keys OR transit token from external secret manager
echo "Waiting for OpenBao to be ready..."
sleep 30
# Your fetch logic here (e.g. aws secretsmanager get-secret-value, vault kv get, curl to API) (1)
env:
- name: EXTERNAL_SECRET_ENDPOINT (2)
value: "https://your-external-secret-store.example.com"

1	Replace with your fetch logic.
2	Replace with your external secret store endpoint.

Option: Transit seal

When you already have a Vault or OpenBao cluster and want it to act as the root of trust (e.g. central Vault encrypts the seal key), you can use this option.

To prepare everything, you need to do the following:

Enable the Transit secrets engine on the external Vault/OpenBao and create a key (e.g. openbao-unseal).
Grant the token used by this OpenBao permission to encrypt/decrypt with that key.
Store the token in a Kubernetes Secret and inject via extraSecretEnvironmentVars

Do not put the token in the config file.

The following HCL configuration is used to configure the OpenBao cluster to use the transit seal.

seal "transit" {
address = "https://external-vault.example.com:8200"
token = "s.xxx"
disable_renewal = "false"
key_name = "openbao-unseal"
mount_path = "transit/"
}

See the OpenBao transit seal documentation for further details.

Dedicated unseal service (seal-only OpenBao)

A common and recommended pattern is to run a separate, standalone OpenBao instance whose only role is to provide the seal for your production OpenBao. In other words: the external OpenBao holds the unseal keys—via its Transit secrets engine—and the production OpenBao cluster uses the transit seal to auto-unseal by calling that external instance. No application secrets or other workloads run on the dedicated instance; it exists solely to unseal the production OpenBao.

Why use a dedicated unseal service?

Separation of concerns: The root of trust for unsealing lives outside the production cluster. If the production OpenBao is compromised or rebuilt, the seal keys remain in the dedicated instance.
Simpler operations: You unseal and manage only one small, locked-down OpenBao (the seal service). The production OpenBao unseals itself automatically via transit.
No cloud KMS required: On-premise or air-gapped environments can use this pattern instead of AWS KMS, Azure Key Vault, or GCP Cloud KMS.
Clear trust boundary: The dedicated instance can be hardened, network-isolated, and backed up independently.

Architecture (high level):

Dedicated unseal service: A standalone OpenBao (single node is often enough) with the Transit secrets engine enabled and a dedicated transit key (e.g. production-openbao-unseal). This instance is initialised and unsealed once; you keep its unseal keys and root token very secure. It does not store application secrets.
Production OpenBao: Your HA OpenBao cluster (e.g. in Kubernetes) is configured with seal "transit" pointing at the dedicated service URL and a token that has permission only to use that transit key. On startup, each production node asks the dedicated OpenBao to decrypt its seal blob and thus unseals automatically.

Migration from Shamir to auto-unseal

You might wonder if you can migrate from Shamir to auto-unseal. The answer is yes, you can.

If OpenBao was previously initialised with Shamir unseal keys and you want to switch to any auto-unseal backend, you can do the following:

Plan a short maintenance window. Raft HA will tolerate one node at a time.
Add the appropriate seal "…" block to the config and deploy (e.g. via Argo CD).

Do NOT re-run init.
Unseal the leader with the existing Shamir unseal keys.
Run seal migration on the leader:
```
bao operator unseal -migrate
```
Restart the leader. It should auto-unseal via the new seal.
Update and restart standby nodes. They should auto-unseal as well.

Resources

The Guide to OpenBao - GitOps Deployment with Argo CD - Part 5

Fri, 20 Feb 2026 00:00:00 +0000

Following the GitOps mantra "If it is not in Git, it does not exist", this article demonstrates how to deploy and manage OpenBao using Argo CD. This approach provides version control, audit trails, and declarative management for your secret management infrastructure.

Introduction

Deploying OpenBao via GitOps offers significant advantages:

Version Control: All configuration changes are tracked in Git
Audit Trail: Who changed what, when, and why
Declarative: Desired state is defined, not imperative commands
Reproducible: Same deployment process across environments
Self-healing: Argo CD ensures actual state matches desired state

However, there are challenges specific to secret management:

Initial unsealing requires manual intervention or automationxw
Root tokens and unseal keys must be handled carefully
Chicken-and-egg problem: How to store OpenBao secrets before OpenBao exists?

This article will focus on the deployment of the applicaiotns for the OpenBao deployment. The problem with the automatic unsealing will be addressed in the next article.

Prerequisites

Before you begin, ensure you have:

OpenShift GitOps (Argo CD) installed and configured
A Git repository for your cluster configuration
Understanding of the App-of-Apps pattern (see Configure App-of-Apps)
The official OpenBao Helm chart from Part 3

The (Wrapper) Helm Chart

The official OpenBao Helm Chart works well for deploying OpenBao itself. However, additional settings such as certificates are not covered there. Therefore, I created a (wrapper) Helm Chart that includes the official OpenBao Helm Chart, a Cert-Manager Helm Chart that I created a while ago, and allows you to add additional objects in the templates folder.

I will follow the same setup as discussed in Part 4. This means:

Create Issuer for Cert-Manager
Create CA Certificate for OpenBao
Create Certificate for OpenBao Server
Create Certificate for OpenBao Agent Injector
Install OpenBao Helm Chart

However, there is one significant issue with this approach: we need the certificate details (especially the CA certificate) before we can install the OpenBao Helm Chart, since the certificate must be referenced in the values file. This is a chicken-and-egg problem—particularly if you use self-signed CA certificates.

Since you typically create the CA certificate first and only once (or have it already), a separate Application for the CA certificate is a good approach. Once this is synchronised, the OpenBao Application can be updated and deployed.

This means we will need two Helm Charts: one for the CA certificate and one for OpenBao itself.

Helm Chart for the CA Certificate

Create a new Helm Chart for the CA certificate. This will use the Cert-Manager Helm Chart as a dependency and will create the:

Issuer for Cert-Manager
Request the CA certificate from the Issuer
Create the openbao namespace

The example I am using can be found at GitOps openbao-ca-certificate.

Please check the GitOps Repository Structure and subsequent articles for more information on how to structure your Git repository.

In the Chart.yaml file you can see two dependencies:

dependencies:
- name: tpl
version: ~1.0.0
repository: https://charts.stderr.at/
- name: cert-manager
version: ~2.0.3 (1)
repository: https://charts.stderr.at/
condition: cert-manager.enabled

1	The Cert-Manager Helm Chart version must be at least 2.0.3 to support the namespace parameter for the Issuer.

The tpl dependency is a library that contains templates for the namespace and other shared components (I keep this library in my Helm Charts repository). The cert-manager dependency is the Cert-Manager Helm Chart.

The values.yaml file below contains the configuration for the Cert-Manager:

All settings that are passed to a subchart (cert-manager) must be prefixed with the name of the subchart. In this case, cert-manager..

namespace:
create: true (1)
name: "openbao"
description: "OpenBao Namespace"
displayName: "OpenBao Namespace"
additionalLabels:
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/audit-version: latest
pod-security.kubernetes.io/warn: restricted
pod-security.kubernetes.io/warn-version: latest
cert-manager: (2)
enabled: true (3)
issuer:
# Name of issuer
- name: openbao-selfsigned (4)
# -- Syncwave to create this issuer
syncwave: 5 (5)
# -- Type can be either ClusterIssuer or Issuer
type: Issuer
# -- Enable this issuer.
# @default -- false
enabled: true
# -- Namespace for Issuer (ignored for ClusterIssuer). Defaults to the default namespace when not set.
namespace: openbao (6)
# -- Create a selfSigned issuer. The SelfSigned issuer doesn't represent a certificate authority as such, but instead denotes that certificates will "sign themselves" using a given private key.
selfSigned: true (7)
certificates: (8)
enabled: true
# List of certificates
certificate:
- name: openbao-ca
enabled: true
namespace: openbao
syncwave: "10"
secretName: openbao-ca-secret
duration: 87660h
dnsNames:
- openbao-ca
privateKey:
algorithm: ECDSA
size: 256
rotationPolicy: Always
isCA: true
# Reference to the issuer that shall be used.
issuerRef:
name: openbao-selfsigned
kind: Issuer

1	Create the openbao namespace with various settings.
2	Enable the Cert-Manager subchart. All settings below will be passed to the Cert-Manager subchart.
3	Enables Cert-Manager.
4	Name of the issuer.
5	Syncwave for this issuer; it must be lower than the syncwave of the certificate.
6	Namespace for the issuer; supported since version 2.0.3 of the Cert-Manager Helm Chart.
7	Creates a self-signed issuer.
8	The certificate section and all the settings. Verify the Part 4 for more information.

This chart creates the openbao namespace, which is required for the OpenBao deployment.

Helm Chart for the OpenBao Deployment

Create a new Helm Chart for the OpenBao deployment. This will use the official OpenBao Helm chart and the Cert-Manager Helm Chart as dependencies and will create the:

OpenBao deployment (including Route)
Issuer for the OpenBao server using the CA certificate
Required OpenBao certificates based on the CA certificate

I have added the full values.yaml file below. Please check the GitOps openbao/values.yaml to fetch the full chart.

Besides the two certificates added here in the values.yaml file, I made two further changes: fullnameOverride and nameOverride are both set to openbao. This is good practice for setting the name of the deployment when using Argo CD.

This values file contains the CA certificate in plain text. Whether that is acceptable is debatable—it is a public certificate, and here it is self-signed and used only in my demo environment.

# Full values.yaml file for the OpenBao deployment
########################################################
# Cert-Manager
########################################################
cert-manager:
enabled: true
issuer:
# Name of issuer
- name: openbao-ca-issuer (1)
# -- Syncwave to create this issuer
syncwave: '-1'
# -- Type can be either ClusterIssuer or Issuer
type: Issuer
# -- Enable this issuer.
# @default -- false
enabled: true
# -- Namespace for Issuer (ignored for ClusterIssuer). Defaults to the default namespace when not set.
namespace: openbao
ca:
secretName: openbao-ca-secret
certificates:
enabled: true
# List of certificates
certificate:
########################################################
# OpenBao Server TLS Certificate
########################################################
- name: openbao-server-tls (2)
enabled: true
namespace: openbao
syncwave: "0"
secretName: openbao-server-tls
duration: 8760h
renewBefore: 720h
dnsNames:
- openbao.openbao.svc
- openbao.apps.ocp.aws.ispworld.at # Route host (adjust to your domain)
- openbao
- openbao.openbao
- openbao.openbao.svc
- openbao.openbao.svc.cluster.local
- openbao-internal
- openbao-internal.openbao
- openbao-internal.openbao.svc
- openbao-internal.openbao.svc.cluster.local
- openbao-0.openbao-internal
- openbao-0.openbao-internal.openbao
- openbao-0.openbao-internal.openbao.svc
- openbao-0.openbao-internal.openbao.svc.cluster.local
- openbao-1.openbao-internal
- openbao-1.openbao-internal.openbao
- openbao-1.openbao-internal.openbao.svc
- openbao-2.openbao-internal
- openbao-2.openbao-internal.openbao
- openbao-2.openbao-internal.openbao.svc
ipAddresses:
- 127.0.0.1
- "::1"
# Reference to the issuer that shall be used.
issuerRef:
name: openbao-ca-issuer
kind: Issuer
########################################################
# Injector TLS Certificate
########################################################
- name: injector-certificate (3)
enabled: true
namespace: openbao
syncwave: "0"
secretName: injector-tls
duration: 24h
renewBefore: 144m
dnsNames:
- openbao-agent-injector-svc
- openbao-agent-injector-svc.openbao
- openbao-agent-injector-svc.openbao.svc
- openbao-agent-injector-svc.openbao.svc.cluster.local
# Reference to the issuer that shall be used.
issuerRef:
name: openbao-ca-issuer
kind: Issuer
########################################################
# OpenBao Deployment
########################################################
openbao: (4)
# Override the full name of the deployment via Argo CD
fullnameOverride: openbao
nameOverride: openbao
global:
# Enable OpenShift-specific settings
openshift: true
# -- The namespace to deploy to. Defaults to the `helm` installation namespace.
namespace: openbao
# Required when TLS is enabled: tells the chart to use HTTPS for readiness/liveness
# probes and for in-pod API_ADDR (127.0.0.1:8200). Otherwise you get "client sent
# an HTTP request to an HTTPS server" from the probes.
tlsDisable: false
server:
extraEnvironmentVars:
BAO_CACERT: /openbao/tls/openbao-server-tls/ca.crt
# High Availability configuration
ha:
enabled: true
replicas: 3
# Raft storage configuration
raft:
enabled: true
setNodeId: true
config: |
ui = true
listener "tcp" {
tls_disable = 0
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/openbao/tls/openbao-server-tls/tls.crt"
tls_key_file = "/openbao/tls/openbao-server-tls/tls.key"
tls_min_version = "tls12"
telemetry {
unauthenticated_metrics_access = "true"
}
}
storage "raft" {
path = "/openbao/data"
retry_join {
leader_api_addr = "https://openbao-0.openbao-internal:8200"
leader_tls_servername = "openbao-0.openbao-internal"
leader_ca_cert_file = "/openbao/tls/openbao-server-tls/ca.crt"
}
retry_join {
leader_api_addr = "https://openbao-1.openbao-internal:8200"
leader_tls_servername = "openbao-1.openbao-internal"
leader_ca_cert_file = "/openbao/tls/openbao-server-tls/ca.crt"
}
retry_join {
leader_api_addr = "https://openbao-2.openbao-internal:8200"
leader_tls_servername = "openbao-2.openbao-internal"
leader_ca_cert_file = "/openbao/tls/openbao-server-tls/ca.crt"
}
}
service_registration "kubernetes" {}
telemetry {
prometheus_retention_time = "30s"
disable_hostname = true
}
route:
enabled: true
host: openbao.apps.ocp.aws.ispworld.at
tls:
# Route terminates client TLS; backend can use reencrypt or passthrough
termination: reencrypt
insecureEdgeTerminationPolicy: Redirect
destinationCACertificate: |
-----BEGIN CERTIFICATE-----
MIIBWzCCAQCgAwIBAgIQNdbg4KIu9oi6dDClE8drmjAKBggqhkjOPQQDAjAAMB4X
DTI2MDIxODA5NDIxNFoXDTM2MDIxODIxNDIxNFowADBZMBMGByqGSM49AgEGCCqG
SM49AwEHA0IABIeYw35/kEHvyctLtOA5xMlyQNxUtXtfBbZMUfPh6AN5MFjIGuNS
cn07a3EpSpfY6/3DaPpu+4wYNFlc+/qDNYajXDBaMA4GA1UdDwEB/wQEAwICpDAP
BgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQgyRk5Vv9K0VULcCgga5mRg4O9kTAY
BgNVHREBAf8EDjAMggpvcGVuYmFvLWNhMAoGCCqGSM49BAMCA0kAMEYCIQCwQ7lZ
Q0jzUjJFzpTGkQjU2+OB159LIQMSbSQ7dz8nVQIhAIDa7f87tjQxDxbJio+/vJx2
awFaWnueGOOQpvwCcV/+
-----END CERTIFICATE-----
extraVolumes:
- type: secret
name: openbao-server-tls
path: /openbao/tls
readOnly: true
extraVolumeMounts:
- name: openbao-server-tls
mountPath: /openbao/tls
readOnly: true
# Resource requests and limits
resources:
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 1Gi
cpu: 1000m
# Persistent volume for data
dataStorage:
enabled: true
size: 10Gi
# storageClass: "gp3-csi"
# Injector configuration
injector:
enabled: true
replicas: 2 # HA for the injector too
certs:
secretName: injector-tls
# For a private CA: set caBundle to the CA cert (PEM) so the Kubernetes API server trusts the injector webhook. E.g. oc get secret openbao-ca-secret -n openbao -o jsonpath='{.data.ca\.crt}' | base64 -d
caBundle: "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJXekNDQVFDZ0F3SUJBZ0lRTmRiZzRLSXU5b2k2ZERDbEU4ZHJtakFLQmdncWhrak9QUVFEQWpBQU1CNFgKRFRJMk1ESXhPREE1TkRJeE5Gb1hEVE0yTURJeE9ESXhOREl4TkZvd0FEQlpNQk1HQnlxR1NNNDlBZ0VHQ0NxRwpTTTQ5QXdFSEEwSUFCSWVZdzM1L2tFSHZ5Y3RMdE9BNXhNbHlRTnhVdFh0ZkJiWk1VZlBoNkFONU1GaklHdU5TCmNuMDdhM0VwU3BmWTYvM0RhUHB1KzR3WU5GbGMrL3FETllhalhEQmFNQTRHQTFVZER3RUIvd1FFQXdJQ3BEQVAKQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlFneVJrNVZ2OUswVlVMY0NnZ2E1bVJnNE85a1RBWQpCZ05WSFJFQkFmOEVEakFNZ2dwdmNHVnVZbUZ2TFdOaE1Bb0dDQ3FHU000OUJBTUNBMGtBTUVZQ0lRQ3dRN2xaClEwanpVakpGenBUR2tRalUyK09CMTU5TElRTVNiU1E3ZHo4blZRSWhBSURhN2Y4N3RqUXhEeGJKaW8rL3ZKeDIKYXdGYVdudWVHT09RcHZ3Q2NWLysKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo="
certName: tls.crt
keyName: tls.key
# UI configuration
ui:
enabled: true

1	Issuer for OpenBao. The syncwave is set to '-1' to create the issuer before the certificate request and the OpenBao deployment.
2	Certificate for the OpenBao server, with all DNS names and IP addresses used to reach it. The syncwave is set to '0' so the certificate is created after the issuer.
3	Certificate for the OpenBao Agent Injector, with all DNS names used to reach it. The syncwave is set to '0' so the certificate is created after the issuer.
4	OpenBao deployment configuration; the two overrides fullnameOverride and nameOverride are both set to openbao. See Part 4 for more information.

Creating Argo CD Applications

Whilst the Helm Charts are ready, we need to create the Argo CD Applications for the CA certificate and the OpenBao deployment. If you read our blog carefully, you will notice that these Application resources are created automatically when I add something to the folder clusters/management-cluster :) This is done by leveraging ApplicationSet resources and is fully described in the GitOps Repository Structure and subsequent articles.

For the sake of simplicity, I will show the created Application resources below. The setup is the same for both; they simply target a different path in the Git repository.

# Full Application resource for the OpenBao CA Certificate
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: in-cluster-openbao-ca-certificate
namespace: openshift-gitops
spec:
destination:
name: in-cluster (1)
namespace: default (2)
info:
- name: Description
value: ApplicationSet that Deploys on Management Cluster Configuration (using Git Generator)
project: in-cluster (3)
source:
path: clusters/management-cluster/openbao-ca-certificate (4)
repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops' (5)
targetRevision: main
syncPolicy:
retry:
backoff:
duration: 5s
factor: 2
maxDuration: 3m
limit: 5
---
# Full Application resource for the OpenBao deployment
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: in-cluster-openbao
namespace: openshift-gitops
spec:
destination:
name: in-cluster (1)
namespace: openbao (2)
info:
- name: Description
value: ApplicationSet that Deploys on Management Cluster Configuration (using Git Generator)
project: in-cluster (3)
source:
path: clusters/management-cluster/openbao (6)
repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops' (5)
targetRevision: main
syncPolicy:
retry:
backoff:
duration: 5s
factor: 2
maxDuration: 3m
limit: 5

1	Target cluster; here, the local cluster
2	Namespace of the target cluster; here, the OpenBao deployment will be installed.
3	Argo CD Project (must exist)
4	Path to the Git repository for the CA certificate
5	URL of the Git repository
6	Path to the OpenBao deployment

This will create the following Applications: - in-cluster-openbao-ca-certificate - in-cluster-openbao

The ApplicationSet adds the prefix in-cluster- to each Application name so that they remain unique in Argo CD.

Figure 1. Argo CD: OpenBao Argo CD Applications

The first Application to synchronise is in-cluster-openbao-ca-certificate.

It creates the following:

Namespace
CA Issuer
CA Certificate

Then synchronise in-cluster-openbao. It creates:

OpenBao deployment
OpenBao Agent Injector
OpenBao Server TLS Certificate
OpenBao Agent Injector TLS Certificate

OpenBao is running—what next?

Deploying OpenBao via GitOps gives you version control and declarative management for your secret management infrastructure.

Whilst OpenBao is running and managed by Argo CD, the next step is to configure it: you will need to handle initialisation (for a new cluster) and unsealing (see Part 3 and Part 4). That manual approach does not scale. In the next article, I will discuss ways to automate initialisation and unsealing.

Key takeaways:

Use sync waves to control deployment order
Consider auto-unseal for production (Part 6)
Store initialisation data securely outside Git

Resources

The Guide to OpenBao - Enabling TLS on OpenShift - Part 4

Tue, 17 Feb 2026 00:00:00 +0000

In Part 3 we deployed OpenBao on OpenShift in HA mode with TLS disabled: the OpenShift Route terminates TLS at the edge, and traffic from the Route to the pods is plain HTTP. While this is ok for quick tests, for a production-ready deployment, you should consider TLS for the entire journey. This article explains why and how to enable TLS end-to-end using the cert-manager operator, what to consider, and the exact steps to achieve it.

Introduction

Part 3 uses tls_disable = 1 in the OpenBao listener and relies on the OpenShift Route for TLS. That gives encryption between the client and the Route, but not between the Route and the OpenBao pods or between pods (e.g. Raft). Enabling TLS on OpenBao itself adds encryption in transit everywhere and aligns with defense-in-depth and compliance requirements. After all, we are talking about a secrets management system here and should not compromise security.

This part assumes:

OpenBao is already deployed as in Part 3 (HA with Raft, Agent Injector enabled).
The cert-manager operator is installed and configured on your OpenShift cluster.

The article SSL Certificate Management for OpenShift describes the setup and usage of the cert-manager operator as an example.

Why Enable TLS?

Enabling TLS for OpenBao makes sense for several reasons:

Encryption in transit: Traffic between the Route and the pods, and between OpenBao peers (Raft), is encrypted. Secrets and tokens are never sent in plain text on the cluster network.
Defense in depth: Even if the Route or network is misconfigured, backend traffic remains protected.
Compliance: Many standards (e.g. PCI-DSS, SOC 2) require encryption in transit for sensitive data; TLS to the application (OpenBao) helps satisfy this.
Agent Injector: The injector webhook is called by the Kubernetes API server. Using TLS for the webhook (with a valid certificate) is required for production and when running multiple replicas.
Consistency: Using HTTPS everywhere simplifies client configuration and avoids mixing HTTP/HTTPS in the same environment.

What Must Be Considered?

Two TLS contexts: Each needs its own certificate and configuration.
1. OpenBao server (API and Raft)
2. Agent Injector (mutating webhook)
Certificate SANs: Server certificates must include all names used to reach OpenBao: Route host, internal service names (e.g. openbao.openbao.svc, openbao-0.openbao-internal.openbao.svc), 127.0.0.1 and ::1 for in-pod traffic, and any external DNS you use. The injector certificate must match the injector Service DNS name (e.g. openbao-agent-injector-svc.openbao.svc).
cert-manager: Using cert-manager gives automatic issuance and renewal. You need a ClusterIssuer (or an Issuer) in the OpenBao namespace.

In this example, we will use a self-signed CA for the OpenBao server and the Agent Injector. In a production environment, you should use a trusted CA.

OpenShift Route: With backend TLS enabled, you can keep the Route in reencrypt mode: the Route terminates TLS from the client and opens a new TLS connection to the pod. Alternatively, use passthrough if you want end-to-end TLS without re-encryption at the Route.
Raft join: After switching to TLS, retry_join and cluster addresses must use https:// and the correct hostnames. Existing unseal keys and root token are unchanged; only the transport is different.
Clients: CLI and applications must use https:// for BAO_ADDR and, if you use a private CA, BAO_CACERT (or the system trust store) so that the client trusts the server certificate.

Prerequisites

OpenBao HA deployment from Part 3 (namespace openbao, Helm release openbao).
cert-manager operator installed on OpenShift.
Sufficient rights to create Issuers, Certificates, and Secrets in the openbao namespace.

Overview of Steps

Create a Certificate Authority (CA) in the openbao namespace (or use an existing ClusterIssuer).
Issue a Certificate for the OpenBao server (API + Raft) and store it in a Secret.
Issue a Certificate for the Agent Injector and reference it in the Helm values.
Update Helm values to mount the server cert and CA, and configure the listener and Raft for TLS.
Upgrade the Helm release; initialize and unseal openbao-0 (new cluster) or re-unseal (existing).
Verify access via HTTPS and configure clients (BAO_ADDR, BAO_CACERT).

Step 1: Certificate Authority (CA) for OpenBao

If you already have a ClusterIssuer (e.g. Let’s Encrypt or an enterprise CA), you can use it for the server and injector certificates and skip this step. For a self-signed CA in the OpenBao namespace (typical for internal cluster TLS), create the following.

This self-signed CA is only for testing purposes. In a production environment, you should use a trusted CA, preferably your own.

Create a self-signed CA Issuer in the openbao namespace:

You might have your own CA or a ClusterIssuer already. You can use them, instead of creating a new one.
```
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: openbao-selfsigned
namespace: openbao (1)
spec:
selfSigned: {} (2)
```
1 The namespace where the OpenBao deployment is running.

2 The self-signed CA Issuer.

Create a self-signed CA Certificate in the openbao namespace:

This will now actually request the CA certificate from the self-signed CA Issuer. This process is fully automated by cert-manager because everything is self-signed. The requested certificate will be stored in the secret openbao-ca-secret and is available immediately.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: openbao-ca
namespace: openbao (1)
spec:
isCA: true (2)
commonName: OpenBao CA
secretName: openbao-ca-secret (3)
duration: 87660h (4)
privateKey: (5)
algorithm: ECDSA
size: 256 (6)
rotationPolicy: Always (6)
issuerRef:
name: openbao-selfsigned (7)
kind: Issuer (8)
group: cert-manager.io

1	The namespace where the OpenBao deployment is running.
2	The certificate is a CA certificate.
3	The name of the secret where the certificate and key will be stored.
4	The duration of the certificate. In this case 10 years.
5	The private key algorithm and size.
6	The rotation policy of the private key.
7	The issuer of the certificate.
8	The kind of the issuer. Can be Issuer or ClusterIssuer.

Step 2: Create an Issuer for the OpenBao Server and Agent Injector

Create an Issuer for the OpenBao Server and Agent Injector. This Issuer will reference the CA certificate and key stored in the secret openbao-ca-secret.

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: openbao-ca-issuer
namespace: openbao
spec:
ca:
secretName: openbao-ca-secret

The name of the secret where the CA certificate and key are stored.

Step 3: Certificate for the OpenBao Server

Now it is time to create the certificate for the OpenBao server. The server certificate must include every hostname used to reach OpenBao: the Route host, the headless service, and each Raft member. Adjust the dnsNames and optional uris to match your cluster and Route.

Create the following Certificate object:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: openbao-server-tls
namespace: openbao
spec:
secretName: openbao-server-tls (1)
duration: 8760h (2)
renewBefore: 720h (3)
commonName: openbao.openbao.svc (4)
dnsNames:
- openbao.apps.cluster.example.com # Route host (adjust to your domain) (5)
- openbao (6)
- openbao.openbao
- openbao.openbao.svc
- openbao.openbao.svc.cluster.local
- openbao-internal
- openbao-internal.openbao
- openbao-internal.openbao.svc
- openbao-internal.openbao.svc.cluster.local
- openbao-0.openbao-internal
- openbao-0.openbao-internal.openbao
- openbao-0.openbao-internal.openbao.svc
- openbao-0.openbao-internal.openbao.svc.cluster.local
- openbao-1.openbao-internal
- openbao-1.openbao-internal.openbao
- openbao-1.openbao-internal.openbao.svc
- openbao-2.openbao-internal
- openbao-2.openbao-internal.openbao
- openbao-2.openbao-internal.openbao.svc
ipAddresses: (7)
- 127.0.0.1
- "::1"
issuerRef:
name: openbao-ca-issuer (8)
kind: Issuer
group: cert-manager.io

1	The name of the secret where the certificate and key will be stored.
2	The duration of the certificate. In this case 1 year.
3	The duration before the certificate is renewed. In this case 30 days.
4	The common name of the certificate. This is the service name of the OpenBao server.
5	The Route host. Adjust to your domain.
6	The headless service names, all should be included.
7	Required for in-pod traffic: Readiness/liveness probes and local bao commands (e.g. raft join, operator unseal) connect to 127.0.0.1:8200. Without these IP SANs you get "tls: bad certificate" or "x509: cannot validate certificate for 127.0.0.1 because it doesn’t contain any IP SANs".
8	The issuer of the certificate, this time openbao-ca-issuer.

After a few moments the certificate will be ready and the secret will be created. The cert-manager will store the signed certificate and key in the Secret openbao-server-tls with keys tls.crt and tls.key. The Helm chart can mount this secret for the OpenBao listener.

Step 4: Certificate for the Agent Injector

The Agent Injector runs as a webhook; the Kubernetes API server calls it over TLS. The certificate must match the Service DNS name of the injector. Create a Certificate that references the same CA Issuer. In this example we use a short-lived certificate valid for 24 hours, renewed when 10% of the validity period remains.

Save as openbao-injector-cert.yaml:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: injector-certificate
namespace: openbao
spec:
secretName: injector-tls (1)
duration: 24h (2)
renewBefore: 144m
commonName: Agent Inject Cert (3)
dnsNames:
- openbao-agent-injector-svc (4)
- openbao-agent-injector-svc.openbao
- openbao-agent-injector-svc.openbao.svc
- openbao-agent-injector-svc.openbao.svc.cluster.local
issuerRef:
name: openbao-ca-issuer
kind: Issuer
group: cert-manager.io

1	The name of the secret where the certificate and key will be stored.
2	The duration of the certificate. In this case 24 hours. (renewal is done 10% before expiry)
3	The injector service name.
4	The injector service name is defined by the Helm chart. If you override the injector service name, adjust dnsNames accordingly.

Step 5: Helm Values for Server TLS

With all the certificates created, we can update the Helm values so that OpenBao uses the server certificate and listens with TLS. You need to:

Mount the Secret openbao-server-tls into the OpenBao pods.
Set the listener to use tls_cert_file and tls_key_file and disable tls_disable.
Switch Raft retry_join and cluster addresses to https://.
Add the environment variable BAO_CACERT to the OpenBao pods.

Before we start, we need to export the CA certificate from the secret openbao-ca-secret and save its value:

oc get secret openbao-ca-secret -n openbao -o jsonpath='{.data.ca\.crt}' | base64 -d

This will return the certificate like this:

-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

Save this, you will need it in the next step.

Now we need to create the Helm values file. This time we will enable TLS. Refer to Part 3 of this series to see what the initial values file looks like.

Create or update openbao-ha-values-tls.yaml (building on your Part 3 values):

global:
# Enable OpenShift-specific settings
openshift: true
# Required when TLS is enabled: tells the chart to use HTTPS for readiness/liveness
# probes and for in-pod API_ADDR (127.0.0.1:8200). Otherwise you get "client sent
# an HTTP request to an HTTPS server" from the probes.
tlsDisable: false (1)
server:
extraEnvironmentVars:
BAO_CACERT: /openbao/tls/openbao-server-tls/ca.crt (2)
# High Availability configuration
ha:
enabled: true
replicas: 3
# Raft storage configuration
raft:
enabled: true
setNodeId: true
config: |
ui = true
listener "tcp" {
tls_disable = 0 (3)
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/openbao/tls/openbao-server-tls/tls.crt" (4)
tls_key_file = "/openbao/tls/openbao-server-tls/tls.key" (5)
tls_min_version = "tls12" (6)
telemetry {
unauthenticated_metrics_access = "true"
}
}
storage "raft" {
path = "/openbao/data"
retry_join {
leader_api_addr = "https://openbao-0.openbao-internal:8200" (7)
leader_tls_servername = "openbao-0.openbao-internal"
leader_ca_cert_file = "/openbao/tls/openbao-server-tls/ca.crt" (8)
}
retry_join {
leader_api_addr = "https://openbao-1.openbao-internal:8200"
leader_tls_servername = "openbao-1.openbao-internal"
leader_ca_cert_file = "/openbao/tls/openbao-server-tls/ca.crt"
}
retry_join {
leader_api_addr = "https://openbao-2.openbao-internal:8200"
leader_tls_servername = "openbao-2.openbao-internal"
leader_ca_cert_file = "/openbao/tls/openbao-server-tls/ca.crt"
}
}
service_registration "kubernetes" {}
telemetry {
prometheus_retention_time = "30s"
disable_hostname = true
}
route:
enabled: true
host: openbao.apps.cluster.example.com
tls: (9)
# Route terminates client TLS; backend can use reencrypt or passthrough
termination: reencrypt
insecureEdgeTerminationPolicy: Redirect
destinationCACertificate: | (10)
-----BEGIN CERTIFICATE-----
# CA Certificate
-----END CERTIFICATE-----
extraVolumes: (11)
- type: secret
name: openbao-server-tls
path: /openbao/tls
readOnly: true
extraVolumeMounts: (12)
- name: openbao-server-tls
mountPath: /openbao/tls
readOnly: true
# Resource requests and limits
resources:
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 1Gi
cpu: 1000m
# Persistent volume for data
dataStorage:
enabled: true
size: 10Gi
# storageClass: "gp3-csi"
# Injector configuration
injector:
enabled: true
replicas: 2 # HA for the injector too
certs: (13)
secretName: injector-tls
# For a private CA: set caBundle to the CA cert (PEM) so the Kubernetes API server trusts the injector webhook. E.g. oc get secret openbao-ca-secret -n openbao -o jsonpath='{.data.ca\.crt}' | base64 -d
caBundle: "BASE64_ENCODED_CA_CERTIFICATE"
certName: tls.crt
keyName: tls.key
# UI configuration
ui:
enabled: true

1	global.tlsDisable: Set to false when the server listener uses TLS. This makes the chart use HTTPS for readiness/liveness probes and for the in-pod API_ADDR env var. If you leave it true (default), probes and local clients will use HTTP and you will see "client sent an HTTP request to an HTTPS server".
2	The environment variable BAO_CACERT is set to the CA certificate file path. This is helpful to execute the boa command inside the container.
3	The listener tls_disable is set to 0 to enable TLS.
4	The listener tls_cert_file is set to the certificate file path.
5	The listener tls_key_file is set to the key file path.
6	The listener tls_min_version is set to tls12.
7	The leader API address is set to the HTTPS address.
8	The leader CA certificate file is set to the CA certificate file path.
9	The Route tls termination is set to reencrypt.
10	The destination CA certificate is set to the CA certificate. Be sure not to add any extra lines or spaces.
11	The extra volumes are mounted at the /openbao/tls path.
12	The extra volume mounts are mounted at the /openbao/tls path.
13	The injector certs are set to the injector TLS secret name. The caBundle must be provided in base64 format.

The secret key names must match what cert-manager writes: tls.crt and tls.key. The OpenBao Helm chart mounts each entry in extraVolumes at path + name (e.g. with path: /openbao/tls and name: openbao-server-tls the secret is mounted at /openbao/tls/openbao-server-tls/). The listener tls_cert_file and tls_key_file must use that full path.

Be sure to change the Route host in the Helm values to the one you are using. In addition, make sure that the CA certificate is added correctly to the Route. Do not add any extra lines or spaces and do not forget the | after destinationCACertificate: and the -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- lines.

Step 6: Upgrade the Helm Release

Perform a Helm upgrade so the new volumes and configuration are applied. Pods will restart and pick up TLS; Raft will use HTTPS for join and replication.

helm upgrade openbao openbao/openbao \
--namespace openbao \
--values openbao-ha-values-tls.yaml

Watch the rollout. The first pod (openbao-0) will log "raft retry join initiated" and stay 0/1 Ready until it is initialized and unsealed (new cluster) or until it forms quorum (existing cluster).

New cluster: initialize and unseal openbao-0

openbao-0 will not become Ready until it is initialized and unsealed. Fetch the certificate, use port-forward and talk to OpenBao over HTTPS with the CA:

# 1. Save the CA cert (for BAO_CACERT)
oc get secret openbao-ca-secret -n openbao -o jsonpath='{.data.ca\.crt}' | base64 -d > openbao-ca.crt
# 2. Port-forward to openbao-0 (background)
oc port-forward openbao-0 8200:8200 -n openbao &
# 3. Use HTTPS and CA cert
export BAO_ADDR='https://127.0.0.1:8200'
export BAO_CACERT="$PWD/openbao-ca.crt"
# 4. Check status, then initialize (only once) and unseal 3 times
bao status
bao operator init -key-shares=5 -key-threshold=3 -format=json > openbao-init.json
bao operator unseal
bao operator unseal
bao operator unseal
bao status

After unsealing, openbao-0 becomes leader and goes 1/1 Ready. Then start openbao-1 and openbao-2 and join or unseal them as in Part 3 (use https://).

Perform the following steps for the other pods - join the raft cluster and unseal them (3 times):

oc exec -ti openbao-1 -- bao operator raft join https://openbao-0.openbao-internal:8200
# 3 times with 3 different unseal keys
oc exec -ti openbao-1 -- bao operator unseal
Unseal Key (will be hidden):

Repeat the same steps for openbao-2.

Existing cluster: re-unseal after restart

The existing cluster is already initialized. Re-unseal the pods (and re-join the raft cluster if necessary).

oc get pods -n openbao -w
oc exec -ti openbao-1 -- bao operator raft join https://openbao-0.openbao-internal:8200
oc exec -ti openbao-1 -- bao operator unseal # three times; same for openbao-2

Repeat the same steps for openbao-2.

Step 7: Verify and Use HTTPS from Clients

Verify server health over HTTPS (from inside the cluster):

curl -k https://openbao.apps.cluster.example.com/v1/sys/health | jq

Be sure to use the Route host in the Helm values.

From your workstation, use the Route URL with HTTPS. If the Route host is signed by your internal CA, set BAO_CACERT to the CA file:

export BAO_ADDR='https://openbao.apps.cluster.example.com'
export BAO_CACERT='/path/to/openbao-ca.crt'
bao status

Be sure to use the Route host in the Helm values.

Agent Injector: New pods that use the injector should start without webhook certificate errors. Check injector logs if you see TLS or certificate errors:

oc logs -n openbao -l app.kubernetes.io/name=openbao-agent-injector -f

Troubleshooting

MutatingWebhookConfiguration conflict (vault-k8s)

conflict occurred while applying object ... MutatingWebhookConfiguration: Apply failed with 1 conflict: conflict with "vault-k8s" using ... .webhooks[name="vault.hashicorp.com"].clientConfig.caBundle
this is a server-side apply (SSA) field ownership conflict. The OpenBao chart uses the same webhook name (vault.hashicorp.com) as the HashiCorp Vault agent injector for annotation compatibility. The clientConfig.caBundle field is still owned by a previous manager (e.g. a prior Vault Helm release or the Vault injector), so Helm cannot update it when you change the injector certificate.

Fix option 1 – Delete the webhook and re-upgrade (recommended)

Remove the MutatingWebhookConfiguration so Helm can recreate it and own all fields. There will be a short window where the injector webhook is missing (new pods requesting injection may fail until the upgrade completes).

# Replace RELEASE_NAME with your Helm release name (e.g. openbao)
RELEASE_NAME=openbao
oc delete mutatingwebhookconfiguration ${RELEASE_NAME}-agent-injector-cfg
# Re-run the upgrade
helm upgrade ${RELEASE_NAME} openbao/openbao \
--namespace openbao \
--values openbao-ha-values-tls.yaml

Fix option 2 – Force takeover of the conflicting field

If you cannot delete the webhook (e.g. in production), take over the caBundle field with server-side apply, then run the Helm upgrade again:

Export the MutatingWebhookConfiguration, set webhooks[0].clientConfig.caBundle to your injector CA (base64 PEM from openbao-ca-secret), then re-apply with --server-side --force-conflicts:

# Get the current object and the new CA bundle
oc get mutatingwebhookconfiguration openbao-agent-injector-cfg -o yaml > mwc.yaml
CA_BUNDLE=$(oc get secret openbao-ca-secret -n openbao -o jsonpath='{.data.ca\.crt}')
# Edit mwc.yaml: set .webhooks[0].clientConfig.caBundle to the value of CA_BUNDLE (no quotes in YAML).
# Then apply with force-conflicts so Helm can later manage it:
oc apply -f mwc.yaml --server-side --force-conflicts
# Re-run the Helm upgrade
helm upgrade openbao openbao/openbao --namespace openbao --values openbao-ha-values-tls.yaml

If you still have HashiCorp Vault’s agent injector installed on the same cluster, ensure only one injector is active for a given namespace (e.g. use namespace selectors) or uninstall the Vault injector to avoid two webhooks with the same name in different resources.

Route / UI – tls: bad record MAC

Pod logs show tls: bad record MAC from the router IP. The Route is likely using edge termination (HTTP to pod). Fix: Use reencrypt and set destinationCACertificate (Step 5).

oc get route openbao -n openbao -o jsonpath='{.spec.tls.termination}'

If the OpenBao UI does not load and the pod logs show:

http: TLS handshake error from 10.x.x.x:xxxxx: local error: tls: bad record MAC

the traffic is coming from the OpenShift router (the IP is typically a cluster pod IP).

Cause: The Route is likely using edge termination: the router terminates TLS at the edge and sends plain HTTP to the pod. The pod expects HTTPS, so the TLS layer receives non-TLS data and reports "bad record MAC".

Fix: Use reencrypt (or passthrough) and set destinationCACertificate so the router talks HTTPS to the pod. See Step 5 above. Quick check:

oc get route openbao -n openbao -o jsonpath='{.spec.tls.termination}'
# Must be "reencrypt" or "passthrough", not "edge"

If the output is edge, patch the Route to reencrypt and set spec.tls.destinationCACertificate to the CA PEM (Step 5).

TLS handshake / 127.0.0.1 certificate errors

If you see in the pod logs:

remote error: tls: bad certificate (from 127.0.0.1)

or when running:

oc exec -ti openbao-0 — bao operator raft join https://…;: x509: cannot validate certificate for 127.0.0.1 because it doesn’t contain any IP SANs

The server certificate does not include 127.0.0.1 (and optionally ::1) as Subject Alternative Names. Readiness/liveness probes and in-pod bao commands connect to the listener on 127.0.0.1, so the certificate must include these IP SANs.

Fix: Add ipAddresses to the OpenBao server Certificate and let cert-manager re-issue:

# Edit the Certificate (or re-apply the YAML from Step 3 with ipAddresses added)
oc edit certificate openbao-server-tls -n openbao
# Add under spec:
# ipAddresses:
# - 127.0.0.1
# - "::1"
# cert-manager will issue a new cert; wait until the secret is updated
oc get certificate openbao-server-tls -n openbao
# Restart OpenBao pods so they load the new cert
oc rollout restart statefulset/openbao -n openbao

Resources

The Guide to OpenBao - OpenShift Deployment with Helm - Part 3

Fri, 13 Feb 2026 00:00:00 +0000

After understanding standalone installation in Part 2, it is time to deploy OpenBao on OpenShift/Kubernetes using the official Helm chart. This approach provides high availability, Kubernetes-native management, and seamless integration with the OpenShift ecosystem.

Introduction

Deploying OpenBao on OpenShift/Kubernetes offers several advantages:

High Availability: Multiple replicas with automatic failover
Kubernetes-native: Managed by standard Kubernetes primitives
Persistent Storage: Data survives pod restarts via PVCs
Integration: Works with Kubernetes service accounts for authentication
Scalability: Easy to scale and manage

The official OpenBao Helm chart supports multiple deployment modes:

Dev: Single server, in-memory storage (testing only)
Standalone: Single server, persistent storage
HA: Multiple servers with Raft consensus (recommended)
External: Connect to external OpenBao cluster

Integrations

Currently, OpenBao supports the following integrations that help seamlessly load secrets into applications without the need to modify the application code:

Agent Injector: A mutating webhook that automatically injects a sidecar container that retrieves and renews secrets.
CSI Provider: A (vendor neutral) CSI driver that mounts secrets as volumes.

Prerequisites

Before deploying, ensure you have:

OpenShift 4.12+ or Kubernetes 1.30+
Helm 3.6+
oc or kubectl CLI configured
The OpenBao CLI (bao) for initialization and unsealing (see OpenBao installation)
Cluster-admin privileges (for initial setup)
A storage class that supports ReadWriteOnce PVCs

# Verify prerequisites
oc version
helm version
# Check available storage classes
oc get storageclass

Adding the Helm Repository

First, add the OpenBao Helm repository:

# Add the OpenBao Helm repository
helm repo add openbao https://openbao.github.io/openbao-helm
# Update repository cache
helm repo update
# Search for available charts
helm search repo openbao
# Expected output:
# NAME CHART VERSION APP VERSION DESCRIPTION
# openbao/openbao 0.x.x 2.x.x Official OpenBao Helm chart

According to the official documentation, the Helm chart is new and under significant development. It should always be run with --dry-run before any install or upgrade to verify changes.

Creating the Namespace

Create a dedicated namespace for OpenBao:

While I am using the oc CLI, you can also use the kubectl CLI as in-place replacement.

oc new-project openbao

Deployment Mode: High Availability (Recommended)

For production environments, deploy in HA mode with Raft. This is the recommended deployment mode for production and is straightforward to achieve on Kubernetes and OpenShift. The following values definitions will use OpenShift specific settings. I will mark them in the callouts.

We will first create a values file, deploy openbao and then discuss what must be done to activate all OpenBao pods.

Create HA Values File

Create openbao-ha-values.yaml:

The values file is based on the official values file from that chart, but only modified values or important changes are listed here.

global:
# Enable OpenShift-specific settings
openshift: true (1)
server:
# High Availability configuration
ha:
enabled: true (2)
replicas: 3 (3)
# Raft storage configuration
raft:
enabled: true (4)
setNodeId: true
config: | (5)
ui = true
listener "tcp" {
tls_disable = 1
address = "[::]:8200"
cluster_address = "[::]:8201"
telemetry {
unauthenticated_metrics_access = "true"
}
}
storage "raft" {
path = "/openbao/data"
retry_join {
leader_api_addr = "http://openbao-0.openbao-internal:8200"
}
retry_join {
leader_api_addr = "http://openbao-1.openbao-internal:8200"
}
retry_join {
leader_api_addr = "http://openbao-2.openbao-internal:8200"
}
}
service_registration "kubernetes" {}
telemetry {
prometheus_retention_time = "30s"
disable_hostname = true
}
route:
enabled: true (6)
host: openbao.apps.cluster.example.com (7)
tls:
termination: edge
# Resource requests and limits
resources:
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 1Gi
cpu: 1000m
# Persistent volume for data
dataStorage: (8)
enabled: true
size: 10Gi
# storageClass: "gp3-csi"
# Injector configuration
injector:
enabled: true
replicas: 2 # HA for the injector too (9)
# UI configuration
ui: (10)
enabled: true

1	OpenShift specific: Activate OpenShift Mode: Critical setting, if you install on OpenShift. It adjusts the Helm chart to use Routes instead of Ingress and modifies RoleBindings to work with OpenShift’s stricter authentication.
2	High Availability (HA): Deploys OpenBao as a StatefulSet rather than a Deployment.
3	Raft Consensus Quorum: Sets the cluster size to 3. Raft requires an odd number of nodes to handle leader elections and avoid split-brain scenarios. This cluster can survive the loss of 1 node.
4	Integrated Raft Storage: Enables the internal Raft storage backend, removing the need for external dependencies like Consul or Etcd.
5	Server Configuration (HCL): The actual OpenBao server configuration file. Note that tls_disable = 1 is used because the OpenShift Route handles TLS termination at the edge, passing unencrypted traffic to the pod.
6	OpenShift specific: Route: Tells Helm to create an OpenShift Route object automatically.
7	OpenShift specific: Host: The external DNS address where users and applications will access the OpenBao API and UI.
8	Persistent Storage: Allocates a 10Gi Persistent Volume Claim (PVC) for each of the 3 pods to store encrypted data and Raft logs.
9	Injector Redundancy: Runs 2 replicas of the sidecar injector. If the injector service is down, new application pods attempting to start with secrets will fail.
10	Web UI: Enables the graphical dashboard service.

Deploy HA Cluster

# Deploy with HA values
helm install openbao openbao/openbao \
--namespace openbao \
--values openbao-ha-values.yaml

Verify the Deployment

oc get pods -n openbao

This will result in the following output:

NAME READY STATUS RESTARTS AGE
openbao-0 0/1 Running 0 60s (1)
openbao-agent-injector-xxx 1/1 Running 0 60s
openbao-agent-injector-yyy 1/1 Running 0 60s

1	Only 1 openbao pod (instead of 3) is running, and the pod is not in "ready" state.

As you can see, only one OpenBao pod exists so far, and it is not in "ready" state. This is because OpenBao is not yet initialized and unsealed. Once that is done, the other pods will appear and join the cluster.

The OpenBao pods show 0/1 ready because they are sealed and need initialization.

This is also indicated in the logs of the openbao pod:

2026-02-13T14:44:52.587Z [ERROR] core: failed to get raft challenge: leader_addr=http://openbao-0.openbao-internal.openbao.svc:8200
error=
| error during raft bootstrap init call: Error making API request.
|
| URL: PUT http://openbao-0.openbao-internal.openbao.svc:8200/v1/sys/storage/raft/bootstrap/challenge
| Code: 503. Errors:
|
| * Vault is sealed (1)
2026-02-13T14:44:52.588Z [ERROR] core: failed to get raft challenge: leader_addr=http://openbao-2.openbao-internal.openbao.svc:8200 error="error during raft bootstrap init call: Put \"http://openbao-2.openbao-internal.openbao.svc:8200/v1/sys/storage/raft/bootstrap/challenge\": dial tcp: lookup openbao-2.openbao-internal.openbao.svc on 172.30.0.10:53: no such host" (2)
2026-02-13T14:44:52.589Z [ERROR] core: failed to get raft challenge: leader_addr=http://openbao-1.openbao-internal.openbao.svc:8200 error="error during raft bootstrap init call: Put \"http://openbao-1.openbao-internal.openbao.svc:8200/v1/sys/storage/raft/bootstrap/challenge\": dial tcp: lookup openbao-1.openbao-internal.openbao.svc on 172.30.0.10:53: no such host"
2026-02-13T14:44:52.589Z [ERROR] core: failed to retry join raft cluster: retry=2s err="failed to get raft challenge"

1	Vault is sealed: This means that the OpenBao service is not yet initialized and unsealed.
2	No such host: This means that the OpenBao service is not yet available.

Initializing and Unsealing OpenBao

After deployment, OpenBao needs to be initialized and unsealed. This is done on the first pod. Once this is done, the other pods will appear and can join the cluster. We will create a local portforwarding to the first pod to initialize it.

Initialize the Cluster

Create a local port forwarding to the first pod

oc port-forward openbao-0 8200:8200 -n openbao &

Set environment variable
```
export BAO_ADDR='http://127.0.0.1:8200'
```

Check status

bao status

This will result in the following output:

Key Value
--- -----
Seal Type shamir
Initialized false (1)
Sealed true (2)
Total Shares 0
Threshold 0
Unseal Progress 0/0
Unseal Nonce n/a
Version 2.5.0
Build Date 2026-02-04T16:19:33Z
Storage Type raft
HA Enabled true (3)

1	Not yet initialized
2	Vault is sealed
3	High Availability is enabled

Initialize the cluster with 5 key shares and 3 threshold
```
bao operator init -key-shares=5 -key-threshold=3 -format=json > openbao-init.json
```
This will create the file openbao-init.json with the unseal keys and root token.
```
cat openbao-init.json
```

Take care of the openbao-init.json file. It contains the unseal keys and root token!

Unseal Pod openbao-0

After initialization, we need to unseal the first pod. This is done by providing 3 different unseal keys. (threshold is 3)

# Unseal openbao-0 (3 times with different keys)
bao operator unseal # Enter first key
bao operator unseal # Enter second key
bao operator unseal # Enter third key

This unseals openbao-0, which can be verified with the command bao status.

Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false (1)
Total Shares 5
Threshold 3
Version 2.5.0
Build Date 2026-02-04T16:19:33Z
Storage Type raft
Cluster Name vault-cluster-80c01167
Cluster ID b81ecb85-9751-655a-95b7-69463dd13241
HA Enabled true
HA Cluster https://openbao-0.openbao-internal:8201
HA Mode active
Active Since 2026-02-13T14:46:13.292643992Z
Raft Committed Index 29
Raft Applied Index 29

1	Vault is unsealed

This makes openbao-0 ready and openbao-1 is trying to start:

oc get pods
NAME READY STATUS RESTARTS AGE
openbao-0 1/1 Running 0 2m10s (1)
openbao-1 0/1 Running 0 14s (2)
openbao-agent-injector-98769cf97-r4stk 1/1 Running 0 2m12s
openbao-agent-injector-98769cf97-xldgm 1/1 Running 0 2m12s

1	openbao-0 is ready
2	openbao-1 is trying to start and wants to join the Raft cluster

Activate Pod openbao-1

Since OpenBao is already initialized, we can skip the initialization step. However, we must join the Raft cluster.

oc exec -ti openbao-1 -- bao operator raft join http://openbao-0.openbao-internal:8200

Once done, we can unseal openbao-1. We need to provide 3 different unseal keys again. Execute the following command 3 times:

oc exec -ti openbao-1 -- bao operator unseal
Unseal Key (will be hidden):

Now openbao-1 is ready and openbao-2 is trying to start:

oc get pods
NAME READY STATUS RESTARTS AGE
openbao-0 1/1 Running 0 3m48s
openbao-1 1/1 Running 0 112s
openbao-2 0/1 Running 0 18s
openbao-agent-injector-98769cf97-r4stk 1/1 Running 0 3m50s
openbao-agent-injector-98769cf97-xldgm 1/1 Running 0 3m50s

Activate Pod openbao-2

Now we need to repeat the previous steps for openbao-2.

oc exec -ti openbao-2 -- bao operator raft join http://openbao-0.openbao-internal:8200
# Run 3 times (enter a different unseal key each time):
oc exec -ti openbao-2 -- bao operator unseal

Verify Raft Cluster

You can verify the Raft cluster by logging in with the root token and then checking the Raft peer list.

oc exec -ti openbao-0 -- bao login

Check the peer list:

# Check Raft peer list
oc exec -ti openbao-0 -- bao operator raft list-peers
Node Address State Voter
---- ------- ----- -----
openbao-0 openbao-0.openbao-internal:8201 leader true
openbao-1 openbao-1.openbao-internal:8201 follower true
openbao-2 openbao-2.openbao-internal:8201 follower true

Accessing the UI

Once unsealed, access the OpenBao UI:

Via Route (Production)

oc get route openbao -n openbao
# Open browser: https://openbao.apps.cluster.example.com

Figure 1. OpenBao Login Form

Upgrading OpenBao

Keep your secrets management up to date. To upgrade an existing deployment:

# Update Helm repository
helm repo update
# Check available versions
helm search repo openbao --versions
# Upgrade with your values file
helm upgrade openbao openbao/openbao \
--namespace openbao \
--values openbao-ha-values.yaml
# Watch the rolling update
oc get pods -n openbao -w

After upgrade, pods may need to be unsealed again if they restart.

Troubleshooting

Pods Not Starting

# Check pod status
oc describe pod openbao-0 -n openbao
# Check pod logs
oc logs openbao-0 -n openbao
# Check events
oc get events -n openbao --sort-by='.lastTimestamp'

PVC Issues

# Check PVC status
oc get pvc -n openbao
# If pending, check storage class
oc describe pvc data-openbao-0 -n openbao

Raft Join Failures

If pods cannot join the Raft cluster:

# Check internal DNS resolution
oc exec -it openbao-0 -n openbao -- nslookup openbao-internal
# Check connectivity between pods
oc exec -it openbao-0 -n openbao -- wget -O- http://openbao-1.openbao-internal:8200/v1/sys/health

What Should Be Considered Next?

Securely store the unseal keys and root token
Configure pod anti-affinity for true HA
Consider auto-unseal for operational ease (upcoming article)
Put everything into a GitOps pipeline

Conclusion

You now have OpenBao running on OpenShift in high-availability mode. This deployment:

Survives pod failures and restarts
Uses Raft for distributed consensus
Integrates with OpenShift security model
Is ready for production use (after unsealing automation)

Key points to remember:

Use HA mode for production
Store unseal keys securely
Configure pod anti-affinity for true HA
Consider auto-unseal for operational ease (upcoming article)

Resources

Hosted Control Planes behind a Proxy

Mon, 15 Dec 2025 00:00:00 +0000

Recently, I encountered a problem deploying a Hosted Control Plane (HCP) at a customer site. The installation started successfully—etcd came up fine—but then it just stopped. The virtual machines were created, but they never joined the cluster. No OVN or Multus pods ever started. The only meaningful message in the cluster-version-operator pod logs was:

I1204 08:22:35.473783 1 status.go:185] Synchronizing status errs=field.ErrorList(nil) status=&cvo.SyncWorkerStatus{Generation:1, Failure:(*payload.UpdateError)(0xc0006313b0), Done:575, Total:623,

This message did appear over and over again.

Problem Summary

etcd started successfully, but the installation stalled
API server was running
VMs started but never joined the cluster
No OVN or Multus pods ever started
The installation was stuck in a loop

Troubleshooting

Troubleshooting was not straightforward. The cluster-version-operator logs provided the only clue. However, the VMs were already running, so we could log in to them—and there it was, the reason for the stalled installation.

Connect to a VM

To connect to a VM, use the virtctl command. This connects you to the machine as the core user:

You can download virtctl from the OpenShift Downloads page in the OpenShift web console.

You need the SSH key for the core user.

virtctl ssh -n clusters-my-hosted-cluster core@vmi/my-node-pool-dsj7z-sss8w (1)

1	Replace my-hosted-cluster with the name of your hosted cluster and my-node-pool-dsj7z-sss8w with the name of your VM.

Verify the Problem

Once logged in, check the journalctl -xf output. In case of a proxy issue, you’ll see an error like this:

> Dec 10 06:23:09 my-node-pool2-gwvjh-rwtx8 sh[2143]: time="2025-12-10T06:23:09Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:500704de3ef374e61417cc14eda99585450c317d72f454dda0dadd5dda1ba57a: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 3.209.93.201:443: i/o timeout"

So we have a timeout when trying to pull the image from the container registry. This is a classic proxy issue. When you try curl or manual pulls you will get the same message.

Configuring the Proxy for the Hosted Control Plane

The big question is: How do we configure the proxy for the Hosted Control Plane?

This is not well documented yet—in fact, it’s barely documented at all.

The Hosted Control Plane is managed through a custom resource called HostedCluster, and that’s exactly where we configure the proxy. The upstream documentation at HyperShift Documentation explains that you can add a configuration section to the resource. Let’s do that:

Update the HostedCluster resource with the following configuration:

spec:
configuration:
proxy:
httpProxy: http://proxy.example.com:8080 (1)
httpsProxy: https://proxy.example.com:8080 (2)
noProxy: .cluster.local,.svc,10.128.0.0/14,127.0.0.1,localhost (3)
trustedCA:
name: user-ca-bundle (4)

1	HTTP proxy URL
2	HTTPS proxy URL
3	Comma-separated list of domains, IPs, or CIDRs to exclude from proxy routing
4	OPTIONAL: Name of a ConfigMap containing a custom CA certificate bundle

If you’re using a custom CA, create the ConfigMap beforehand or alongside the HostedCluster resource. The ConfigMap must contain a key named ca-bundle.crt with your CA certificate(s).

But wait there is more…

If you are using such a proxy, which is injecting it’s own certificate, then you probably saw the following: The Release Image is not available when you try to deploy a new Hosted Control Plane. The drop down in the UI is empty and you can’t select a release image.

This happens because the Pod that is trying to fetch the image is not able to connect to Github.

You can test this with the following commands:

oc rsh cluster-image-set-controller-XXXXX -n multicluster-engine (1)
curl -v https://github.com/stolostron/acm-hive-openshift-releases.git/info/refs?service=git-upload-pack (2)

1	Replace XXXXX with the name of the Pod cluster-image-set-controller in the multicluster-engine namespace
2	Try to execute the curl command to see if you can connect to Github

If this command fails with a certificate error, then you need to add the certificate to the Pod/Deployment. A Configmap called trusted-ca-bundle should already exist in the multicluster-engine namespace. If not, it must be created with the certificate chain of your Proxy (key: ca-bundle.crt)

The following command will add the ConfigMap to the deployment.

oc -n multicluster-engine set volume deployment/cluster-image-set-controller --add --type configmap --configmap-name trusted-ca-bundle --name trusted-ca-bundle --mount-path /etc/pki/tls/certs/ --overwrite

Wait a couple of minutes after the Pods has been restarted. It will try to download the release images from Github again.

You can check the logs of the Pod to see if the download was successful.

oc logs cluster-image-set-controller-XXXXX -n multicluster-engine (1)

1	Replace XXXXX with the name of the Pod cluster-image-set-controller in the multicluster-engine namespace

That should do it, you should now be able to select a release image and deploy a new Hosted Control Plane.

Figure 1. Drow Down with Release Images

Summary

That’s actually everything you need for proxy configuration with Hosted Control Planes. Hopefully, the official OpenShift documentation will be updated soon to include this information. red Hat created two tickets to track the progress of this:

The Guide to OpenBao - Standalone Installation - Part 2

Thu, 12 Feb 2026 00:00:00 +0000

In the previous article, we introduced OpenBao and its core concepts. Now it is time to get our hands dirty with a standalone installation. This approach is useful for testing, development environments, edge deployments, or scenarios where Kubernetes is not available.

Introduction

While OpenBao shines in Kubernetes environments, understanding the standalone installation helps you grasp the fundamentals. This knowledge is valuable whether you are:

Learning OpenBao before deploying to production
Running OpenBao outside of Kubernetes (edge, legacy systems)
Debugging issues in containerized deployments
Setting up a development environment

Installation Methods Overview

OpenBao can be installed through multiple methods:

Method

Best For

Complexity

Package managers (apt, dnf, brew)

Production Linux/macOS systems

Low

Container images (Podman/Docker)

Quick testing, isolated environments

Low

Binary download

Air-gapped environments

Low

Source compilation

Custom builds, development

Medium

For this article, we will focus on macOS for local testing with binary and container image (using Podman) and Red Hat Enterprise Linux to set up an example production-ready server. The deployment on the Kubernetes environment will be discussed in the next article.

Method 1: Package Manager Installation

RHEL/Fedora/CentOS

On RHEL or CentOS, before you can install OpenBao you need to install the EPEL repository. To do so, you first need to enable the Code Ready Repository. The following commands will do the trick. Be sure that your system is registered, in case of RHEL.

If you get the error "Repositories disabled by configuration." you need to tell the subscription manager that you want to manage the repositories. This can be done permanently or temporarily. You can use the command: sudo subscription-manager config --rhsm.manage_repos=1 to do so.

RHEL

Enable Code Ready Repository

sudo subscription-manager repos --enable codeready-builder-for-rhel-9-$(arch)-rpms

Install EPEL Repository

sudo dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm

Install OpenBao
```
sudo dnf install -y openbao
```

CentOS

Enable Code Ready Repository on CentOS

sudo dnf config-manager --set-enabled crb

Install EPEL Repository

sudo dnf install epel-release epel-next-release

Install OpenBao
```
sudo dnf install -y openbao
```

macOS (Homebrew)

Install OpenBao
```
brew install openbao
```

Verify Installation

After installation, verify OpenBao is available:

bao version
# Output:
OpenBao v2.5.0 (bcbb6036ec2b747bceb98c7706ce9b974faa1b23), built 2026-02-04T15:57:17Z (cgo)

The OpenBao CLI command is bao, not vault. This distinguishes it from HashiCorp Vault.

Start Development Server

To start a development environment, you can use the following command.

This is NOT suitable for production, it is just a test to evaluate the basic concepts of OpenBao.

bao server -dev -dev-root-token-id="dev-only-token"

This will start the server. The UI is accessible at http://localhost:8200 where you can login using the root token dev-only-token.

Method 2: Container Image

For quick testing or isolated environments, container images are ideal. Luckily, OpenBao offers several types of containers suitable for any environment. We will use the image hosted on quay.io, which is based on RHEL UBI and can be found at: quay.io/openbao/openbao-ubi

Using Podman

The following command will fetch the image and start the container.

This is NOT suitable for production, it is just a test to evaluate the basic concepts of OpenBao.

# Run in dev mode (for testing only!)
podman run --rm -d \
--name openbao-dev \
-p 8200:8200 \
-e 'BAO_DEV_ROOT_TOKEN_ID=dev-only-token' \
-e 'BAO_DEV_LISTEN_ADDRESS=0.0.0.0:8200' \
quay.io/openbao/openbao-ubi:latest server -dev

# Verify it is running
podman logs -f openbao-dev

This will start the server. The UI is accessible at http://localhost:8200 where you can login using the root token dev-only-token.

If you prefer to use Docker, simply replace podman with docker in the commands.

Dev Mode: Quick Start for Testing

Dev mode is the fastest way to start using OpenBao for learning and testing.

The characteristics in this mode are:

In-memory storage (data lost on restart)
Automatically initialized and unsealed
Root token printed to stdout
TLS disabled
Single server (no HA)

Let’s create an example secret and try to retrieve it again.

Authenticate against OpenBao

export VAULT_TOKEN="dev-only-token"
export BAO_ADDR='http://127.0.0.1:8200'

Create Secret

curl \
--header "X-Vault-Token: $VAULT_TOKEN" \
--header "Content-Type: application/json" \
--request POST \
--data '{"data": {"password": "OpenBao123"}}' \
$BAO_ADDR/v1/secret/data/my-secret-password &&
echo "Secret written successfully."

Retrieve Secret

curl --header "X-Vault-Token: $VAULT_TOKEN" \
$BAO_ADDR/v1/secret/data/my-secret-password | jq '.data.data'

You should see the password that was created before:

{
"password": "OpenBao123"
}

Check Status of OpenBao Server

bao status

This will give you the status of your running OpenBao instance.

Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 2.5.0
Build Date 2026-02-04T15:57:17Z
Storage Type inmem
Cluster Name vault-cluster-421b2431
Cluster ID 6d42dcd2-e399-211e-999a-49b1874cc8ce
HA Enabled false

Hardening your System

OpenBao does a good job to secure your secrets, however, memory paging (or swap) can undermine the protection. Your OS should either have swap disabled completely or encrypt the swap space.

As I am testing on macOS, the swap space is encrypted out of the box.

However, OpenBao has documented what must be done for various operating systems at Post-installation hardening

Production Standalone Setup

For a proper standalone installation, follow these steps:

Step 1: Create Configuration File

Create /etc/openbao.d/openbao.hcl:

# Full configuration for standalone OpenBao server
# Cluster name for identification
cluster_name = "openbao-standalone"
# Storage backend using integrated Raft
storage "raft" {
path = "/var/lib/openbao/data"
node_id = "node1"
}
# HTTP listener (for internal communication)
listener "tcp" {
address = "0.0.0.0:8200"
cluster_address = "0.0.0.0:8201"
tls_disable = false
tls_cert_file = "/etc/openbao.d/tls/tls.crt"
tls_key_file = "/etc/openbao.d/tls/tls.key"
}
# API address for clients
api_addr = "https://openbao.example.com:8200"
# Cluster address for raft communication
cluster_addr = "https://openbao.example.com:8201"
# UI enabled
ui = true
# Logging
log_level = "info"
log_file = "/var/log/openbao/openbao.log"
# Disable memory locking (enable in production if possible)
disable_mlock = true
# Telemetry (optional)
telemetry {
prometheus_retention_time = "30s"
disable_hostname = true
}

Step 2: Generate TLS Certificates

For production, you should use proper certificates. For testing, create self-signed ones:

This is only for testing purposes. In production, you should use proper certificates.

First, prepare the TLS directory if it does not exist
```
sudo mkdir -p /etc/openbao.d/tls
```

Then create a configuration file for the TLS certificates:

# Create TLS config
cat <<EOF > openbao.cnf
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no
[req_distinguished_name]
CN = openbao.example.com
[v3_req]
subjectAltName = @alt_names
[alt_names]
DNS.1 = openbao.example.com
IP.1 = 127.0.0.1 (1)
EOF

1	The IP address of the server. In this case we are using localhost, but you can add your IPs here.

Then generate the certificate:

# Generate private key
sudo openssl genrsa -out /etc/openbao.d/tls/tls.key 4096
# Generate certificate signing request
sudo openssl req -new -key /etc/openbao.d/tls/tls.key -out /etc/openbao.d/tls/tls.csr -subj "/CN=openbao.example.com"
# Generate self-signed certificate
sudo openssl x509 -req -days 365 -in /etc/openbao.d/tls/tls.csr -signkey /etc/openbao.d/tls/tls.key -out /etc/openbao.d/tls/tls.crt -extfile openbao.cnf -extensions v3_req
# Set permissions
sudo chown -R openbao:openbao /etc/openbao.d/tls
sudo chmod 600 /etc/openbao.d/tls/tls.key
sudo chmod 755 /etc/openbao.d/tls

Step 3: Create Systemd Service

Create /etc/systemd/system/openbao.service:

[Unit]
Description=OpenBao Secret Management
Documentation=https://openbao.org/docs
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/openbao.d/openbao.hcl
[Service]
User=openbao
Group=openbao
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/bin/bao server -config=/etc/openbao.d/openbao.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
LimitNOFILE=65536
LimitMEMLOCK=infinity
[Install]
WantedBy=multi-user.target

Step 4: Start the Service

# Reload systemd
sudo systemctl daemon-reload
# Enable and start OpenBao
sudo systemctl enable openbao
sudo systemctl start openbao
# Check status
sudo systemctl status openbao
# View logs
sudo journalctl -u openbao -f

Initialize and Unseal OpenBao

After starting OpenBao for the first time, it needs to be initialized and unsealed.

Set Environment Variables

We will do the following commands as root user.

Be sure that the hostname is resolvable. In this case I am using the test domain: openbao.example.com.

# Set the OpenBao address
export BAO_ADDR='https://openbao.example.com:8200'
# If using self-signed certificates
export BAO_CACERT='/etc/openbao.d/tls/tls.crt'

Check Status

You will see that OpenBao is running but not yet initialized. This will be the next step.

bao status
Key Value
--- -----
Seal Type shamir
Initialized false (1)
Sealed true
Total Shares 0
Threshold 0
Unseal Progress 0/0
Unseal Nonce n/a
Version 2.4.4-1.el9
Build Date 2025-11-24
Storage Type file
HA Enabled false

1	Not yet initialized

Initialize OpenBao

Before we use OpenBao, we need to initialize it. This will create the unseal keys and the root token. The unseal keys are used to … well, unseal the OpenBao service. The root token is used as a master key to authenticate against the OpenBao service. This token has access to ALL secrets. Treat this root token with care.

We will initialize OpenBao with default options of 5 key shares and a threshold of 3. This means that we need 3 different unseal keys to unseal the OpenBao service.

bao operator init -key-shares=5 -key-threshold=3 -format=json > /root/openbao-init.json

Store the unseal keys and root token securely! Anyone with these can access all secrets.

The output looks like:

{
"unseal_keys_b64": [
"key1...",
"key2...",
"key3...",
"key4...",
"key5..."
],
"unseal_keys_hex": [...],
"unseal_shares": 5,
"unseal_threshold": 3,
"recovery_keys_b64": [],
"recovery_keys_hex": [],
"root_token": "sbr.xxxxxxxxxxxx"
}

Unseal OpenBao

Now everything is set up and we can use the unseal keys to unseal the OpenBao service.

You need to provide 3 (threshold) different unseal keys:

# First key
bao operator unseal
# Enter first unseal key
# Second key
bao operator unseal
# Enter second unseal key
# Third key
bao operator unseal
# Enter third unseal key

After providing enough keys:

bao status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false (1)
Total Shares 5
Threshold 3
Version 2.4.4-1.el9
Build Date 2025-11-24
Storage Type file
Cluster Name openbao-standalone
Cluster ID d07874ef-df52-e45f-1723-ade333c6d609
HA Enabled false

1	Now unsealed and ready

Now we can login with the root token to the OpenBao service.

Login with Root Token

Since we have not created any users yet, we will use the root token to authenticate.

# Enter the root token from initialization
bao login

Basic Verification

Let us verify the installation works:

List enabled secrets engines

bao secrets list

This lists the enabled secrets engines.

Path Type Accessor Description
---- ---- -------- -----------
cubbyhole/ cubbyhole cubbyhole_84c3d2a5 per-token private secret storage
identity/ identity identity_61809171 identity store
sys/ system system_034ae772 system endpoints used for control, policy and debugging

List enabled auth methods

bao auth list

This lists the enabled authentication methods.

Path Type Accessor Description Version
---- ---- -------- ----------- -------
token/ token auth_token_c7feca17 token based credentials n/a

Create a test secret
```
bao secrets enable -path=secret kv-v2
```
This enables the KV (key/value) secrets engine at the path secret.
```
bao kv put secret/test message="Hello from OpenBao"
```
This creates a test secret at the path secret/test.
```
bao kv get secret/test
```
This retrieves the test secret from the path secret/test.

That’s it for the basic verification. You can now start to use OpenBao in your production environment. Let’s see what we can do in the UI.

Visiting the UI

If you watched the previous commands closely, you will have noticed that the UI has been enabled in the configuration file:

# UI enabled
ui = true

The UI is accessible at http://openbao.example.com:8200. You can login with the root token from the initialization.

Figure 1. OpenBao Login Form

Here you will see several options to explore. For now we are interested in the Secrets Engine section.

Figure 2. OpenBao Secrets Engine

We have created the path "secret" and inside it a secret called "test". We can retrieve it now:

Figure 3. OpenBao Secret Retrieval

Security Hardening Checklist

Before using OpenBao in production you should always consider the following checklist:

Item

Recommendation

TLS

Always enable TLS with valid certificates

Root Token

Revoke root token after initial setup and create admin users instead

Unseal Keys

Distribute to different people/locations, consider auto-unseal

Network

Restrict access with firewall rules

Audit

Enable audit logging

Backups

Regular Raft snapshots

Updates

Keep OpenBao updated

What is Coming Next?

In Part 3, we will deploy OpenBao on OpenShift/Kubernetes using the official Helm chart. This provides:

High availability out of the box
Kubernetes-native management
Integration with OpenShift security features
Persistent storage via PVCs

Conclusion

You now have a working standalone OpenBao installation. This forms the foundation for understanding how OpenBao operates. While standalone mode is useful for testing and edge cases, most production deployments will use Kubernetes, which we will cover next.

Key takeaways:

OpenBao can run standalone or in containers
Dev mode is for testing only
Production requires proper TLS, initialization, and security hardening
Unseal keys must be stored securely

Resources

The Guide to OpenBao - Introduction - Part 1

Wed, 11 Feb 2026 00:00:00 +0000

I finally had some time to dig into Secret Management. For my demo environments, SealedSecrets is usually enough to quickly test something. But if you want to deploy a real application with Secret Management, you need to think of a more permanent solution.

This article is the first of a series of articles about OpenBao, a HashiCorp Vault fork. Today, we will explore what OpenBao is, why it was created, and when you should consider using it for your secret management needs. If you are familiar with HashiCorp Vault, you will find many similarities, but also some important differences that we will discuss.

Introduction

In general, sensitive information in any system should be stored in a secure way. When it comes to OpenShift or Kubernetes, this is especially true, since the secrets are stored in the etcd database. Even if etcd is encrypted at rest, anybody can decode a given base64 string which is stored in the Secret.

Base64 is not an encryption format. It is an encoding format.

For example, the string Thomas encoded as base64 is VGhvbWFzCg==. This is simply masked plain text and it is not secure to share these values, especially not on Git. To make your CI/CD pipelines or GitOps process secure, you need to think of a secure way to manage your Secrets.

This is where OpenBao comes in.

What is OpenBao?

OpenBao is an identity-based secrets and encryption management system. It is an open-source, community driven fork of HashiCorp Vault and provides secure storage, fine-grained access control, and lifecycle management for secrets such as API or SSH keys, passwords, certificates, and encryption keys.

The core OpenBao workflow consists of four stages:

Authentication: Verifying the identity of the client to determine if they are who they say they are. Once authenticated, a token is created and associated with a so-called policy.
Validation: Checking if the client has the required permissions
Authorization: Granting access based on policies that provide or deny access to certain paths and operations in OpenBao.
Access: Limiting what the client can do with the secret

At its core, OpenBao:

Stores and encrypts sensitive information at rest and in transit
Provides fine-grained access controls (ACL)
Supports dynamic secret generation
Offers a comprehensive audit trail (must be enabled)
Enables encryption as a service

The History: Why OpenBao Exists

OpenBao is a community-driven, open-source fork of HashiCorp Vault. But why was it created?

In August 2023, HashiCorp announced a significant change to the licensing of their products, including Vault. They moved from the Mozilla Public License 2.0 (MPL 2.0) to the Business Source License 1.1 (BSL 1.1).

The BSL 1.1 license includes restrictions on competitive use:

Competitors could no longer use Vault’s code to offer competing services
Cloud providers and managed service providers faced restrictions
The open-source community lost the freedom to fork and commercialize

The Fork

In response to this license change, the Linux Foundation announced the OpenBao project in December 2023. OpenBao is:

A fork of HashiCorp Vault (from version 1.14.x)
Licensed under the OSI-approved Mozilla Public License 2.0
Governed by a community-driven model
Maintained independently of HashiCorp
Continues to enable innovation without license restrictions

OpenBao vs. HashiCorp Vault: Key Differences

While OpenBao shares its heritage with Vault, there are several important differences:

Licensing

Aspect

OpenBao

HashiCorp Vault

License

MPL 2.0 (OSI-approved open source)

BSL 1.1 (with staged conversion)

Governance

Community-driven (Linux Foundation)

HashiCorp/IBM controlled

Commercial restrictions

None

Restrictions on competitive use

Enterprise Features

Community-driven additions

Paid Enterprise tier

Support

Community support

Paid support available

Branding

bao CLI, OpenBao naming

vault CLI, Vault naming

Technical Differences

Token Format - OpenBao uses shorter tokens in the format sbr.[random], while Vault uses longer tokens (hvs., hvb., hvr. prefixes followed by long random strings). Old Vault tokens are still accepted according to their TTLs, but newly issued tokens follow the OpenBao format.
Plugin Ecosystem - OpenBao comes with fewer built-in plugins by default, focusing on OSI-licensed integrations. Proprietary cloud vendor plugins have been moved to external repositories.
Storage Backend - OpenBao has simplified its storage options, primarily supporting Raft as the recommended backend.
API Compatibility - OpenBao’s API is designed to be compatible with Vault, meaning existing clients and integrations should work without modification. However, some edge cases may require updates.

Core Concepts

Before diving into installation and configuration in the next articles, it is essential to understand the fundamental concepts of OpenBao:

Secrets Engines

Secrets engines are components that store, generate, or encrypt data. OpenBao supports multiple types:

KV (Key-Value): Simple static secret storage with optional versioning
PKI: Certificate Authority for generating TLS certificates
Database: Dynamic credential generation for databases
Transit: Encryption as a Service (EaaS)
SSH: SSH key signing and OTP generation

Authentication Methods

Authentication methods verify client identity before granting access:

Kubernetes: Uses Kubernetes service account tokens. Essential for Kubernetes/OpenShift deployments.
OIDC: OpenID Connect for user authentication, like Keycloak, Okta, or Azure AD.
LDAP: Directory service authentication, like Active Directory, OpenLDAP, or Microsoft AD.
AppRole: Machine-oriented authentication for applications. Ideal for CI/CD pipelines.
Token: Direct token-based authentication. The root token is created during initialization.

Storage Backend

OpenBao encrypts all data before writing to storage. Supported backends include:

Integrated Raft (recommended): Built-in distributed storage
Consul: HashiCorp’s service discovery and KV store (less common with OpenBao)
File: Single-node deployments only
PostgreSQL/MySQL: Database-backed storage

Policies

Policies define what a client can do after authentication. They use path-based access control (deny-by-default mode):

path "secret/data/myapp/*" {
capabilities = ["read", "list"]
}
path "pki/issue/my-role" {
capabilities = ["create", "update"]
}
path "database/creds/myapp-role" {
capabilities = ["read"]
}

Tokens

Tokens are the primary authentication credential in OpenBao. After successful authentication, clients receive a token that:

Has a TTL (Time To Live)
Is associated with one or more policies
Can be renewed (if renewable)
Can create child tokens

Seal/Unseal

OpenBao starts in a sealed state where it cannot access encrypted data. The unseal process requires multiple key shares (using Shamir’s Secret Sharing) or auto-unseal mechanisms to decrypt the master key.

Leases

Most secrets in OpenBao have an associated lease - a duration after which the secret expires. This enables:

Automatic secret rotation
Revocation of compromised credentials
Audit trail of secret usage

Use Cases

Dynamic Database Credentials

Instead of storing static database passwords:

Application authenticates to OpenBao
Requests database credentials
OpenBao creates a temporary database user
Returns credentials with a short TTL
Credentials automatically expire

Benefits: No long-lived credentials, automatic rotation, per-application isolation.

Certificate Management with PKI

Instead of manually managing TLS certificates:

Configure OpenBao as an intermediate CA
Applications request certificates on demand
Short-lived certificates (hours/days instead of years)
Automatic renewal before expiration

Benefits: No certificate sprawl, automated rotation, reduced attack surface.

Encryption as a Service

Instead of implementing encryption in each application:

Applications send plaintext to OpenBao’s Transit engine
OpenBao encrypts with managed keys
Applications store ciphertext
Decryption requests go through OpenBao

Benefits: Centralized key management, separation of duties, key rotation without re-encryption.

Kubernetes Secret Injection

Instead of storing secrets in Kubernetes Secrets:

Deploy OpenBao with Kubernetes auth
Configure injector or External Secrets Operator
Pods automatically receive secrets at startup
Secrets never stored in etcd

Benefits: Secrets not in cluster, dynamic injection, centralized management.

When to Use OpenBao

OpenBao is an excellent choice when you need:

Centralized Secret Management - If you have secrets scattered across configuration files, environment variables, and various secret stores, OpenBao provides a single source of truth.
Dynamic Secrets - For use cases where you need short-lived, automatically rotated credentials (e.g., database passwords), OpenBao can generate them on-demand.
Encryption as a Service - If applications need encryption capabilities without managing encryption keys, OpenBao’s Transit engine provides this functionality.
Certificate Management - OpenBao’s PKI engine can act as a Certificate Authority, issuing and managing TLS certificates.
Compliance Requirements - For environments with strict audit requirements, OpenBao provides comprehensive audit logging of all secret access.

When NOT to Use OpenBao

OpenBao might be overkill for:

Simple, Static Secrets - If you only have a few static secrets that rarely change, simpler solutions like Sealed Secrets or External Secrets Operator with a basic backend might suffice.
Small Teams with Limited Resources - OpenBao requires operational expertise to maintain. If you do not have the resources to operate it properly, consider managed alternatives.
Single-Application Deployments - If you have a single application with minimal secret requirements, the complexity of OpenBao may not be justified.

Architecture Overview

A typical OpenBao deployment consists of:

Figure 1. Architecture Overview

Multiple Nodes: For high availability (HA), OpenBao runs as a cluster
Raft Consensus: Leader election and data replication
Persistent Storage: Encrypted data stored on persistent volumes
Load Balancer: Distributes client requests

Conclusion

OpenBao provides a powerful, open-source solution for secret management that addresses the challenges of modern cloud-native environments. Its fork from HashiCorp Vault means it benefits from years of development while remaining truly open source.

In the next article, we will start with a standalone installation to understand the fundamentals before moving to Kubernetes deployments.

Resources

RBAC Overview — an OpenShift Console plugin

Wed, 10 Jun 2026 00:00:00 +0000

Answering the question "who can do what on this cluster?" usually means juggling oc commands, YAML dumps, and several console pages. RoleBindings live in namespaces, ClusterRoleBindings are cluster-scoped, OpenShift adds Users and Groups on top of Kubernetes subjects, and SecurityContextConstraints have their own authorisation model. For a quick audit before a change window, that is a lot of clicking and I always struggled to find a quick and reliable way to get and overview. For example, I always wanted to see immediately which users have cluster-admin rights or which service accounts are on the privileged SCC.

The RBAC Overview is a dynamic OpenShift Console plugin that pulls these views into one place. This article walks through what it does, how to install or test it, and how I use the tabs in practice. This open source plugin is available on GitHub

Why another RBAC view?

The console already lists RoleBindings, ClusterRoleBindings, Users, and Groups. That is sufficient when you know exactly which resource you need. It is less helpful when you start from a question:

Who has cluster-admin outside the usual system groups?
What can user alice do in the namespace openshift-gitops?
Which groups may get pods in the namespace default?
Which service accounts are on the privileged SCC?

The RBAC Overview plugins tries to fix these questions. Each tab uses the Kubernetes and OpenShift authorization APIs (SelfSubjectReview, ResourceAccessReview, SubjectAccessReview, and list permissions) rather than re-implementing policy evaluation in the browser.

What you get

Navigation: The plugin hooks into the Administrator perspective → User Management → RBAC Overview (entry appears above Users).

As direct URL you can use: <cluster-url>/rbac-overview.

The following tabs are available:

Tab Question it answers

Who can

Who may perform a certain action or whether one subject is allowed to do certain actions. It basically mirrors the oc adm policy who-can command and presents the results in a more user-friendly way. The tab also includes a Show my access button for the logged-in user, which shows what the logged in user can do on the cluster.

Cluster admins

Cluster-wide and namespace-scoped bindings to sensitive roles (cluster-admin, admin, or roles you configure). This helps to see which users have elevated rights and which namespaces are affected.

Namespace access

All bindings that affect one namespace — local RoleBindings and applicable ClusterRoleBindings.

Subjects

Pick a user, group, or service account and inspect its bindings. This helps to inspect a certain subject and see which roles are bound to it.

Role access

Pick a ClusterRole and list every ClusterRoleBinding and RoleBinding that references it — the inverse of the Subjects tab when you start from a role name instead of a user or group.

SCC access

All SCCs with key fields, direct users / groups / serviceAccounts on each SCC, and RBAC grants to use securitycontextconstraints. This helps to see which users, groups, and service accounts are on which SCCs and which RBAC grants are in place.

Architecture in brief

The plugin follows the usual dynamic plugin pattern:

Webpack builds static assets (plugin-manifest.json, JavaScript bundles, locales).
A container image serves those files over HTTPS on port 9443 (required by the ConsolePlugin API).
Helm installs Deployment, Service, ConsolePlugin, ConfigMap (plugin-config.json), and optionally a Job that enables the plugin on the console. (Plugins must be enabled by the console operator.)

The runtime image uses Project Hummingbird (registry.access.redhat.com/hi/nginx). I tried this as they are expected to be super-small distroless images, with the goal to have zero CVEs … and the size of the image is really only 19MB.

The nginx that is used in this image is serving dist/ with a serving certificate injected by OpenShift. There is no application logic in the container beyond static files.

Configuration for sensitive roles on the Cluster admins tab is delivered via plugin-config.json (Helm values → ConfigMap). Default entries include cluster-admin and admin; you can add names or label ClusterRoles with rbac-overview.io/elevated: "true".

Prerequisites

OpenShift Container Platform or OKD 4.10+ (dynamic console plugins)
For cluster install: Helm 3, oc, Podman or Docker, and permission to push your image registry
For local development: Node.js 22+, npm, and oc login with a bearer token (not certificate-only kubeconfig)

The plugin is tested against recent 4.20+ consoles. Match the console SDK generation to your cluster (the repository currently targets @openshift-console/dynamic-plugin-sdk@4.21-latest).

Install on a cluster

Install with Helm

Log in to the cluster, then install the chart. The published and public plugin image is quay.io/tjungbau/openshift-console-rbac-overview:1.0.5 (chart version 1.0.5).

From the repository:

oc login https://<api-url>:6443 -u <user> -p '<password>'
git clone https://github.com/tjungbauer/openshift-console-rbac-overview.git
cd openshift-console-rbac-overview
helm upgrade --install rbac-overview ./chart/rbac-overview \
--namespace rbac-overview \
--create-namespace \
--set namespace.create=false \
--set image.tag=1.0.5 \
--set image.pullPolicy=Always \
--wait

Or add the published Helm repository (no clone required):

helm repo add rbac-overview https://tjungbauer.github.io/openshift-console-rbac-overview
helm repo update
helm upgrade --install rbac-overview rbac-overview/rbac-overview \
--namespace rbac-overview \
--create-namespace \
--set namespace.create=false \
--set image.tag=1.0.5 \
--wait

The chart defaults to quay.io/tjungbau/openshift-console-rbac-overview; set image.tag only when you need a specific release.

Do not combine helm --create-namespace with --set namespace.create=true. Helm 4 reports an ownership error on the Namespace object. Use --create-namespace and namespace.create=false as shown above. I will clean this up in the future.

Build your own image

If you prefer to create your own image, for example in an air-gapped environment, you are free to do so. The runtime image is a thin nginx layer on top of webpack output — there is no npm step inside the container. This helps keep the image small and avoids unnecessary dependencies.

From the repository root:

npm install
npm run build # writes dist/ (plugin-manifest.json, bundles, locales)
podman build --platform=linux/amd64 \ (1)
-t <your-registry-repository>/:<tag> .
podman login <your-registry-repository> # if your repository is private
podman push <your-registry-repository>/:<tag>

1	Use your own registry, repository and version tag.

Then point Helm at your image:

helm upgrade --install rbac-overview ./chart/rbac-overview \
--namespace rbac-overview \
--create-namespace \
--set namespace.create=false \
--set image.repository=<your-registry-repository> \ (1)
--set image.tag=<tag> \ (2)
--set image.pullPolicy=Always \
--wait

1	Use your own registry and repository.
2	Use your own version tag.

The repository also ships scripts/build-install.sh, which runs npm run build, builds and pushes the image, installs the chart, and runs verification checks in one pass:

chmod +x scripts/build-install.sh
QUAY_USER=<your-quay-user> ./scripts/build-install.sh

Override VERSION or PLATFORM when the tag should differ from package.json or when you build on Apple Silicon for a linux/amd64 cluster.

The script was created due to my own lazyness and it has quay.io hardcoded.

After install, confirm the plugin is enabled:

oc get console.operator cluster -o jsonpath='{.spec.plugins}{"\n"}'
oc -n rbac-overview get deploy,consoleplugin

Open the console, switch to Administrator, and reload User Management → RBAC Overview.

Local development

If you like to test the plugin locally first, you can do so by running the following commands - no image build required:

# terminal 1
npm install
npm run start
# terminal 2
oc login ... # must provide a token: oc whoami --show-token
npm run start-console # console on http://localhost:9000

Open http://localhost:9000/rbac-overview. The console loads the plugin from localhost:9001 while API calls use your oc session.

On Apple Silicon, set CONSOLE_IMAGE_PLATFORM=linux/amd64 before npm run start-console.

Using the plugin

Who can

The default tab mirrors oc adm policy who-can for namespaced and cluster-scoped resources.

Query mode offers:

Who can — list subjects allowed to perform the action
Can subject — check one user, group, or service account

Set Scope, Verb, Resource, optional API group, Subresource, Resource name, and for namespaced queries a real Namespace.

The button Show my access lists groups, visible namespaces, and role bindings for the logged-in user. This becomes handy when someone asks what can I actually see on this cluster?

From My access you can jump to Can subject with your user pre-filled, or open the Subjects tab on yourself.

Cluster admins

This tab highlights bindings to roles you treat as elevated (configured in plugin-config.json or via the rbac-overview.io/elevated label on ClusterRoles).

Sub tabs:

Cluster-wide admins — ClusterRoleBindings to sensitive ClusterRoles.
Namespace elevated — RoleBindings in any namespace that reference sensitive roles.

Use View rules on a row to expand policy rules below the table.

Namespace access

Select a namespace, then browse:

In namespace — RoleBindings defined in that namespace.
Cluster-wide — ClusterRoleBindings whose ClusterRoles apply there.

The default filter shows User and Group subjects; add ServiceAccount when you care about workload identities.

Subjects

If you are interested to query a specific subject (user, group etc.) you can use this tab. The list merges:

subjects found in RoleBindings and ClusterRoleBindings across the cluster, and
Users, Groups, and ServiceAccounts returned by the OpenShift user API (where your account may list them).

Select an entry to see binding details on the right, open linked resources, or continue in Who can → Can subject.

The error message that appears in the screenshot for the selected user is due to the fact that there are no bindings for this user. This is normal and expected.

Role access

The Subjects tab answers who has what bound to them? Role access flips the question: who is bound to this ClusterRole? That is useful when you audit a built-in or custom ClusterRole (view, edit, cluster-admin, or a role you ship with an operator) and need every ClusterRoleBinding and RoleBinding that references it in one table.

Pick a ClusterRole from the dropdown. The table lists matching bindings with scope, namespace (for namespaced RoleBindings), role kind, subjects, and View rules to expand the ClusterRole policy inline.

The screenshot shows cluster-admin with 30 bindings — a mix of human users, the cluster-admins group, and OpenShift system groups such as system:openshift:operator:kube-apiserver-operator.

SCC access

To get a quick overview about SCC and who is on which SCC was always tricky.

OpenShift SCCs authorise subjects in two ways:

direct fields on the SCC object
RBAC to use securitycontextconstraints.

This tab shows every SCC and their configuration.

All SCCs — every SCC you can list, with priority, privileged/host settings, run-as-user and SELinux strategy, volumes, and direct subject count.
Direct authorizations — users, groups, and service accounts listed on each SCC.
RBAC use grants — ClusterRoleBindings that grant use on SCC resources.

This tab does not replace a full SCC compliance review, but it saves time when you are tracing why a pod was allowed a privileged context.

Limitations

Who can requires the same elevated create permissions as oc adm policy who-can.
Large clusters: tables paginate, but initial binding scans can take a moment on first tab open.

Summary

RBAC Overview does not replace oc or your GitOps RBAC manifests. It gives operators and auditors a single console entry point for the questions that come up every week — who is admin, what does this namespace grant, what is bound to this service account, who can touch this verb/resource, and which identities sit on sensitive SCCs.

The plugin was created with the help of AI (I always want to be open about that), especially the initial layout, some styling elements like dropdowns and buttons and the README … which I never want to write.

If you try it, feedback and issues are welcome on the repository.

References

Cert-Manager Policy Approver in OpenShift

Tue, 03 Jun 2025 00:00:00 +0000

One of the most commonly deployed operators in OpenShift environments is the Cert-Manager Operator. It automates the management of TLS certificates for applications running within the cluster, including their issuance and renewal.

The tool supports a variety of certificate issuers by default, including ACME, Vault, and self-signed certificates. Whenever a certificate is needed, Cert-Manager will automatically create a CertificateRequest resource that contains the details of the certificate. This resource is then processed by the appropriate issuer to generate the actual TLS certificate. The approval process in this case is usually fully automated, meaning that the certificate is issued without any manual intervention.

But what if you want to have more control? What if certificate issuance must follow strict organizational policies, such as requiring a specifc country code or organization name? This is where the CertificateRequestPolicy resource, a resource provided by the Approver Policy, comes into play.

This article walks through configuring the Cert-Manager Approver Policy in OpenShift to enforce granular policies on certificate requests.

Prerequisites

Before you begin, ensure you have the following:

OpenShift 4.16 or higher with cluster-admin access
Cert-Manager Operator installed

The installation of Cert-Manager itself is discussed in the article: Managing Certificates using GitOps approach.

The Cert-Manager Operator does not currently support the Approver Policy by default. You need to install the Approver Policy manually using a Helm Chart. There is a feature request to include the Approver Policy in the Cert-Manager Operator in the future.

Adding Approver Policy Chart as a dependency

Source: Cert-Manager Deployment

The above Chart contains the necessary resources to deploy the Cert-Manager itself and Cert-Manager Approver Policy in OpenShift. To deploy the Approver Policy alongside Cert-Manager, add it as a dependency in your Chart.yaml:

[...]
- name: cert-manager-approver-policy
version: v0.19.0
repository: https://charts.jetstack.io
[...]

This is the official Helm Chart for the Cert-Manager Approver Policy tool provided by Jetstack. The version used in this example is v0.19.0, but you can use a newer version if available.

Configuration of the Helm Chart

The initial values.yaml file was extended to:

include the configuration for the Approver Policy Chart
the configuration for a CertificateRequestPolicy object
with some specific modifications to the CertManager resource itself

Disabling Cert-Manager’s Auto-Approver

The first step we need to do is to disable the auto-approver of the Cert-Manager. If this is not done, there will be a race condition between the auto-approver and the Approver Policy, which will lead to unexpected results. These changes are done by:

cert-manager:
certManager:
enable_patch: true (1)
unsupportedConfigOverrides:
controller:
args:
- '--controllers=*,-certificaterequests-approver' (2)

1	This enables the patching of the CertManager resource, which tells the chart to overwrite (patch) the automatically generated CertManager resource with the custom configuration.
2	This disables the auto-approver Controller of the Cert-Manager.

As the name suggests, this is currently an unsupported configuration, but it is necessary to disable the auto-approver for the Cert-Manager. In the future versions of the Cert-Manager, this might change, and the auto-approver might be supported out of the box.

In my case, the full CertManager resource looks like this:

apiVersion: operator.openshift.io/v1alpha1
kind: CertManager
metadata:
annotations:
name: cluster
spec:
controllerConfig:
overrideArgs: (1)
- '--dns01-recursive-nameservers-only'
- '--dns01-recursive-nameservers=ns-362.awsdns-45.com:53,ns-930.awsdns-52.net:53'
logLevel: Normal
managementState: Managed
operatorLogLevel: Normal
unsupportedConfigOverrides: (2)
controller:
args:
- '--controllers=*,-certificaterequests-approver'

1	Settings to support AWS Nameservers for DNS01 challenges.
2	This disables the auto-approver Controller of the Cert-Manager.

Configuration of the Approver Policy

The second step is to configure the Approver Policy chart. This chart will deploy the necessary resources, most importantly a Deployment that will start the Pods which will process the CertificateRequestPolicy resources later on.

My configuration for that chart looks like this (some default values are omitted for brevity):

cert-manager-approver-policy:
crds: (1)
# This option decides if the CRDs should be installed
# as part of the Helm installation.
enabled: true
# This option makes it so that the "helm.sh/resource-policy": keep
# annotation is added to the CRD. This will prevent Helm from uninstalling
# the CRD when the Helm release is uninstalled.
# WARNING: when the CRDs are removed, all cert-manager-approver-policy custom resources
# (CertificateRequestPolicy) will be removed too by the garbage collector.
keep: true
# Number of replicas of approver-policy to run.
replicaCount: 1 (2)
image: (3)
# Target image repository.
repository: quay.io/jetstack/cert-manager-approver-policy
# Kubernetes imagePullPolicy on Deployment.
pullPolicy: IfNotPresent
tag: v0.19.0
app:
# List of signer names that approver-policy will be given permission to
# approve and deny. CertificateRequests referencing these signer names can be
# processed by approver-policy. Defaults to an empty array, allowing approval
# for all signers.
approveSignerNames: (4)
- 'issuers.cert-manager.io/*'
- 'clusterissuers.cert-manager.io/*'

1	This enables the installation of the CRDs that are required for the Approver Policy.
2	The number of replicas of the Approver Policy Deployment. In most cases, one replica is enough.
3	The image configuration for the Approver Policy. The image is pulled from the Jetstack Quay.io repository.
4	The list of signer names that the Approver Policy will be allowed to approve. In this case, it is configured to allow all issuers and clusterissuers.

The approveSignerNames are, if configured, an important setting, especially if you want to add custom (cluster)issuers. In such a case, you need to add the name of the custom issuer to this list. Otherwise the Approver Policy will not be able to approve the CertificateRequests for that issuer.

Creating a CertificateRequestPolicy and Role(Binding)

The final step in our configuration is to define a CertificateRequestPolicy resource that will define the policy for the certificate requests. This resource will be processed by the Approver Policy and will determine if a certificate request is approved or denied based on the defined criteria.

The following example shows a CertificateRequestPolicy that will:

Allow certificate requests with any common name, DNS names, IP addresses, URIs, and email addresses.
Require DNS names to be set.
Require the subject to contain a specific organization (MyOrganization) and country code (AT).
Allow usages for server auth and client auth.
Set constraints for the certificate duration (1h-24h) and private key algorithm (RSA) and size (2048-4096).
Allow all issuers by using an empty selector.

role: cert-manager-policy:global-approver (1)
serviceAccount: cert-manager (2)
cert_manager_Namespace: cert-manager (3)
policies:
- name: my-approver-policy
enabled: true
allowed:
commonName:
required: false
value: "*"
validations: []
dnsNames: (4)
required: true
values:
- "*"
validations: []
ipAddresses:
required: false
values: ["*"]
validations: []
uris:
required: false
values:
- "*"
validations: []
emailAddresses:
required: false
values:
- "*"
validations: []
# isCA: false
subject:
organizations: (5)
required: true
values:
- "MyOrganization"
validations:
- rule: self.matches("MyOrganization")
message: Organization must be MyOrganization
countries:
required: true
values:
- AT
validations:
- rule: self.matches("AT")
message: Country code must be AT
usages:
- "server auth"
- "client auth"
constraints: (6)
minDuration: 1h
maxDuration: 24h
privateKey:
algorithm: RSA
minSize: 2048
maxSize: 4096
selector:
issuerRef: {} (7)

1	The role that is used to approve the certificate requests. This role must be created in the OpenShift cluster and must have the necessary permissions to approve certificate requests.
2	The service account that is used by the Approver Policy to process the certificate requests. This service account must have the necessary permissions to access the CertificateRequest resources.
3	The namespace where the Cert-Manager is deployed. This is usually the `cert-manager` namespace, but you can change it if you have a different namespace.
4	The DNS names are required to be set for the certificate request.
5	The subject must contain the organization MyOrganization and the country code AT.
6	The constraints for the certificate request, such as the minimum and maximum duration, private key algorithm, and size.
7	The selector is empty, which means that the policy applies to all issuers. If you want to limit the policy to specific issuers, you can specify the issuerRef here.

Rendered CertificateRequestPolicy and Role(Binding)

The above configuration will create a CertificateRequestPolicy resource that looks like this:

Expand me...

---
# Source: cert-manager/templates/ClusterRole-Approver-Policy-approving.yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: "cert-manager-policy:global-approver"
labels:
helm.sh/chart: cert-manager-2.0.0
app.kubernetes.io/name: cert-manager
app.kubernetes.io/instance: release-name
app.kubernetes.io/managed-by: Helm
rules:
- verbs:
- use
apiGroups:
- policy.cert-manager.io
resources:
- certificaterequestpolicies
resourceNames:
- my-approver-policy
---
# Source: cert-manager/charts/cert-manager-approver-policy/templates/clusterrolebinding.yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
labels:
app.kubernetes.io/name: cert-manager-approver-policy
helm.sh/chart: cert-manager-approver-policy-v0.19.0
app.kubernetes.io/instance: release-name
app.kubernetes.io/version: "v0.19.0"
app.kubernetes.io/managed-by: Helm
name: cert-manager-approver-policy
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cert-manager-approver-policy
subjects:
- kind: ServiceAccount
name: cert-manager-approver-policy
namespace: default
---
# Source: cert-manager/templates/CertificateRequestPolicy.yaml
apiVersion: policy.cert-manager.io/v1alpha1
kind: CertificateRequestPolicy
metadata:
name: my-approver-policy
annotations:
argocd.argoproj.io/sync-wave: "10"
labels:
helm.sh/chart: cert-manager-2.0.0
app.kubernetes.io/name: cert-manager
app.kubernetes.io/instance: release-name
app.kubernetes.io/managed-by: Helm
spec:
allowed:
commonName:
required: false
value: "*"
validations: []
dnsNames:
required: true
values:
- "*"
validations: []
emailAddresses:
required: false
values:
- "*"
validations: []
ipAddresses:
required: false
values:
- "*"
validations: []
uris:
required: false
values:
- "*"
validations: []
isCA: false
subject:
organizations:
required: true
values:
- "MyOrganization"
validations:
- rule: "self.matches(\"MyOrganization\")"
message: "Organization must be MyOrganization"
countries:
required: true
values:
- "AT"
validations:
- rule: "self.matches(\"AT\")"
message: "Country code must be AT"
organizationalUnits:
required: false
values: ["*"]
validations: []
localities:
required: false
values: ["*"]
validations: []
provinces:
required: false
values: ["*"]
validations: []
streetAddresses:
required: false
values: ["*"]
validations: []
postalCodes:
required: false
values: ["*"]
validations: []
serialNumber:
required: false
value: "*"
validations: []
usages:
- "server auth"
- "client auth"
constraints:
minDuration: 1h
maxDuration: 24h
privateKey:
algorithm: RSA
minSize: 2048
maxSize: 4096
selector:
issuerRef: {}

One important note is about the ClusterRole and ClusterRoleBinding that are created by the Helm Chart. The role looks like the following and is required to allow the Approver Policy to approve certificate requests. This small bit, puzzled me for a while:

rules:
- verbs:
- use
apiGroups:
- policy.cert-manager.io
resources:
- certificaterequestpolicies
resourceNames:
- my-approver-policy

With the above configuration we are good to go. The Helm Chart can be deployed to the OpenShift cluster (for example, using Argo CD), and the CertificateRequestPolicy will be created automatically.

A new Pod is running in the cert-manager namespace:

❯ oc get pods -n cert-manager | grep approver
NAME READY STATUS RESTARTS AGE
cert-manager-approver-policy-xxxxx 1/1 Running 0 XXm (1)

1	The Pod `cert-manager-approver-policy-xxxxx` is the Pod that is responsible for processing the CertificateRequestPolicy resources.

Testing the Policy

Test 1 - Valid Certificate Request

Now it is time to test the policy. We need to create a Certificate and monitor the output of our approval pod.

As a reminder, the policy we created requires the following:

The subject must contain the organization MyOrganization
The subject must contain the country code AT.
The keysize must be at least 2048 bits. (max 4096 bits)
The duration must be between 1 hour and 24 hours.
The usage must be server auth or client auth.

Let’s create this example Certificate in the myproject namespace:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: test-certificate1
namespace: myproject
spec:
dnsNames:
- test1.apps.ocp.aws.ispworld.at
duration: 24h
issuerRef:
kind: ClusterIssuer
name: letsencrypt-prod
privateKey:
algorithm: RSA
encoding: PKCS1
rotationPolicy: Always
secretName: test1
subject:
organizations:
- MyOrganization
countries:
- AT
usages:
- server auth

In the log of the Approver Policy Pod, we should see the following output:

time=2025-06-03T16:07:58.656Z level=DEBUG+3 msg="Approved by CertificateRequestPolicy: \"my-approver-policy\"" logger=controller-manager/events type=Normal object="{Kind:CertificateRequest Namespace:myproject Name:test-certificate1-1 [...]}" reason=Approved

This indicates that the CertificateRequest was approved by the Approver Policy. The policy was able to validate the subject, keysize, duration, and usage of the certificate request and approved it accordingly. The certificate has been created successfully in the myproject namespace, and the secret test1 contains the TLS certificate and private key.

❯ oc get secret test1 -n myproject -o yaml
apiVersion: v1
data:
tls.crt: ...
tls.key: ...
kind: Secret
metadata:
labels:
controller.cert-manager.io/fao: "true"
name: test1
namespace: myproject
type: kubernetes.io/tls

Test 2 - Invalid Certificate Request

That was easy, but what happens if we create a CertificateRequest that does not meet the policy requirements? Let’s try to create a Certificate without the required organization or a wrong country code:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: test-certificate2
namespace: myproject
spec:
dnsNames:
- test2.apps.ocp.aws.ispworld.at
duration: 24h
issuerRef:
kind: ClusterIssuer
name: letsencrypt-prod
privateKey:
algorithm: RSA
encoding: PKCS1
rotationPolicy: Always
secretName: test2
subject: (1)
countries:
- XX
usages:
- server auth

1	The subject does not contain the required organization MyOrganization and the country code is set to XX, which is not allowed by the policy.

This will lead to the following error in the log of the Approver Policy Pod:

time=2025-06-03T16:16:02.233Z level=DEBUG+3 msg="No policy approved this request: [my-approver-policy: [spec.allowed.subject.organizations.required: Required value: true, spec.allowed.subject.countries.values: Invalid value: []string{\"XX\"}: AT, spec.allowed.subject.countries.validations[0]: Invalid value: \"XX\": Country code must be AT]]" logger=controller-manager/events type=Warning object="{Kind:CertificateRequest Namespace:myproject Name:test-certificate2-1 ..." reason=Denied

It complains that the subject does not meet the policy requirements and therefore the CertificateRequest was denied.

Ok we have a policy, but whats next?

The above example shows how the Cert-Manager Approver Policy can be configured and deployed, even if it is not yet supported by the Cert-Manager Operator. However, we only scratched the surface of what is possible with the Approver Policy. You can create more complex policies that include additional validations, such as checking the validity of the DNS names, IP addresses, or URIs. You can also create policies that require specific email addresses or organizational units in the subject.

You can even create fine-grained policies that apply to specific issuers or namespaces by using the selector field in the CertificateRequestPolicy resource. This allows you to create policies that are tailored to your specific requirements and use cases.

The best references can be found here:

Official documentation: Cert-Manager Approver Policy
Example Policies: Cert-Manager Approver Policy Examples

Conclusion

The Cert-Manager Approver Policy is a powerful tool that allows you to implement custom policies for certificate requests in OpenShift. It provides a way to control the issuance of TLS certificates based on specific criteria, such as the subject, key size, duration, and usage of the certificate. While not yet officially supported by the Cert-Manager Operator, it can be easily integrated into your OpenShift environment using a Helm Chart. Future support is currently being discussed, and it is expected that the Approver Policy will be included in the Cert-Manager Operator in the future.

Single log out from Keycloak and OpenShift

Thu, 22 May 2025 00:00:00 +0000

The following 1-minute article is a follow-up to my previous article about how to use Keycloak as an authentication provider for OpenShift. In this article, I will show you how to configure Keycloak and OpenShift for Single Log Out (SLO). This means that when you log out from Keycloak, you will also be logged out from OpenShift automatically. This requires some additional configuration in Keycloak and OpenShift, but it is not too complicated.

Prerequisites

The following prerequisites are required to follow this article:

OpenShift 4.16 or higher with cluster-admin privileges
Keycloak installed and configured, as described in the Step by Step - Using Keycloak Authentication in OpenShift

Keycloak Configuration

Configure the logout URL in Keycloak. This is done by adding a new Valid post logout redirect URIs to the Keycloak client configuration. In this case, we want to call the OpenShift logout URL. Which is: https://oauth-openshift.apps.<your-cluster-name>/logout

This is done in the client settings:

OpenShift Configuration

Now we need to configure OpenShift to use the logout URL. This is done by adding a new Post Logout Redirect URI to the OpenShift Console configuration.

Modify the existing Console resource named cluster and add the logoutRedirect parameter to the authentication section. The important pieces of the logout URL are the client_id and the post_logout_redirect_uri. The client ID must be the same as the Keycloak client ID, and the post logout redirect URI must be the OpenShift logout URL. (Also change the <keycloakURL> and <realm> to your Keycloak URL and realm name.)

Actually, instead of client_id, the id_token_hint should be used. But OpenShift does not store the token, so we are using the client ID instead.

apiVersion: config.openshift.io/v1
kind: Console
metadata:
name: cluster
spec:
authentication:
logoutRedirect: 'https://<keycloakURL>/realms/<realm>/protocol/openid-connect/logout?client_id=<realm>&post_logout_redirect_uri=https://console-openshift-console.apps.ocp.aws.ispworld.at' (1)

1	The logout URL for OpenShift. This is the URL that will be called when you log out from Keycloak. The URL must contain the client ID and the post logout redirect URI (console of OpenShift).

Wait a few moments until the OpenShift Console Operator applies the new configuration.

Testing the configuration

Now that the configuration is done, we can test the Single Log Out functionality.

We are logged into OpenShift already as user testuser
Now we log out from OpenShift using the Log out button in the upper right corner.
This will redirect us to the Keycloak logout page where we need to confirm the logout.
This will log us out from Keycloak and redirect us back to the OpenShift logout page.

This is it. You are now logged out from both Keycloak and OpenShift. You can also check the Sessions in Keycloak to see that the session for the user is terminated.

Conclusion

As promised, this was a short article about how to configure Keycloak and OpenShift for Single Log Out. This is a beneficial feature if you want to ensure that users are logged out from all applications when they log out from Keycloak. It is also a good security practice to ensure that users are logged out from all applications when they are done using them.

Step by Step - Using Keycloak Authentication in OpenShift

Sat, 17 May 2025 00:00:00 +0000

I was recently asked about how to use Keycloak as an authentication provider for OpenShift. How to install Keycloak using the Operator and how to configure Keycloak and OpenShift so that users can log in to OpenShift using OpenID. I have to admit that the exact steps are not easy to find, so I decided to write a blog post about it, describing each step in detail. This time I will not use GitOps, but the OpenShift and Keycloak Web Console to show the steps, because before we put it into GitOps, we need to understand what is actually happening.

This article tries to explain every step required so that a user can authenticate to OpenShift using Keycloak as an Identity Provider (IDP) and that Groups from Keycloak are imported into OpenShift. This article does not cover a production grade installation of Keycloak, but only a test installation, so you can see how it works. For production, you might want to consider a proper database (maybe external, but at least with a backup), high availability, etc.).

Prerequisites

The following prerequisites are required to follow this article:

OpenShift 4.16 or higher with cluster-admin privileges

Installing Keycloak Operator

The very first step is to install the Keycloak Operator. This can be done using the OpenShift Web Console or the CLI or GitOps. I will show the steps using the OpenShift Web Console.

Installing the Keycloak Operator via GitOps can be seen in this GitHub repository. By the time you read this article, this repository will install and configure a Keycloak instance automatically using a local EXAMPLE Postgres database. It does not yet import any Realm or additional configuration, but I will try to add this in the future when time allows. Oh, and it is just a demo and not production ready, so please do not use it in production.

Log in to the OpenShift Web Console as a user with cluster-admin privileges.
In the OpenShift Web Console, navigate to the Operators → OperatorHub.
In the OperatorHub, search for Red Hat build of Keycloak and select the Operator from the list.
Install the latest version of the Keycloak Operator.

You can keep the default settings for the installation, but I recommend using a specific namespace. In this example, I will use the namespace keycloak for the installation.
Press the Install button to install the Keycloak Operator. After a few minutes, the Keycloak Operator should be installed and running in the OpenShift cluster. You can check the status of the Operator in the Installed Operators view.

Create a local example Postgres Database

Demo only! This is not production ready and should not be used in production.

As a next step, we need to create a local Postgres database for Keycloak. Secrets are used to store the database credentials, and a simple StatefulSet is used to create the database. The database will be created in the same namespace as the Keycloak Operator.

First, let’s create a Secret for the database credentials. In OpenShift select the keycloak namespace and navigate to the Secrets view. Create a new Secret with the following test configuration:
```
kind: Secret
apiVersion: v1
metadata:
name: keycloak-db-secret
namespace: keycloak
stringData:
password: thisisonly4testingNOT4prod (1)
username: testuser
type: Opaque
```
1 The password for the database. This is just a test password and should not be used in production.

Next, we need to create a StatefulSet for the Postgres database, again in the keycloak namespace. Again, as I cannot mention that enough, this is just a test configuration and should not be used in production.

apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgresql-db
namespace: keycloak
spec:
serviceName: postgresql-db-service
selector:
matchLabels:
app: postgresql-db
replicas: 1 (1)
template:
metadata:
labels:
app: postgresql-db
spec:
containers:
- name: postgresql-db
image: postgres:15 (2)
volumeMounts:
- mountPath: /data
name: psql
env: (3)
- name: POSTGRES_USER
value: testuser
- name: POSTGRES_PASSWORD
value: thisisonly4testingNOT4prod
- name: PGDATA
value: /data/pgdata
- name: POSTGRES_DB
value: keycloak
volumeClaimTemplates: (4)
- metadata:
name: psql
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "gp3-csi"
resources:
requests:
storage: 10Gi

1	The number of replicas for the database. In this example, we are using a single replica.
2	The image for the database, postgres version 15 in this example.
3	The environment variables for the database. The username and password are the same as in the Secret we created before, and clear text
4	The volume for the database. In this example, the StatefulSet uses a volume claim template to create a volume with the size of 10 GB for the database. The volume is created using the `gp3-csi` storage class. You can use any other storage class that is available in your OpenShift cluster or even remove this line and use the default class instead.

Finally, we need to create a Service for the database so that the Keycloak Operator can access the database. Again, in the keycloak namespace.
```
apiVersion: v1
kind: Service
metadata:
name: postgres-db
namespace: keycloak
spec:
selector:
app: postgresql-db (1)
type: LoadBalancer
ports:
- port: 5432
targetPort: 5432
```
1 The selector for the Service. This must match the label of the StatefulSet we created before.

Creating a Keycloak Instance

Now that the Keycloak Operator is installed and our example database is running, we can create a Keycloak instance.

In the OpenShift Web Console, navigate to the Installed Operators view and select the Keycloak Operator. (Maybe you need to select the keycloak namespace first.)
In the Keycloak Operator view, create a new instance of Keycloak and switch to the YAML view.

The fun part here is that the YAML example the Operator provides is actually wrong and does not work. Something that kept me busy for a while.

Replace the YAML with the following configuration:

apiVersion: k8s.keycloak.org/v2alpha1
kind: Keycloak
metadata:
name: keycloak (1)
namespace: keycloak
labels:
app: sso
spec:
db: (2)
host: postgres-db
passwordSecret:
key: password
name: keycloak-db-secret
usernameSecret:
key: username
name: keycloak-db-secret
vendor: postgres
hostname: (3)
hostname: sso.apps.ocp.aws.ispworld.at
http: (4)
tlsSecret: keycloak-certificate
instances: 1 (5)

1	The name and the namespace of the Keycloak instance.
2	The database configuration. In this example, we are using a local Postgres database. You can also use an external database, but you need to configure the connection string accordingly.
3	Hostname of our Keycloak instance.
4	The TLS secret for the Keycloak instance. You need to create a TLS secret with the certificate and key for the hostname. This is where the example YAML is wrong. It tries to put tlsSecret under spec, but it should be under http.
5	The number of instances of Keycloak. In this example, we are using a single instance.

What about the SSL Certificate?

The Keycloak Operator does not create a certificate for the Keycloak instance. You need to create a certificate manually and store it in a secret. The Operator will use this secret to create the TLS certificate for the Keycloak instance. In the example above we are referencing a secret called keycloak-certificate in the keycloak namespace. This secret was created using the Cert Manager Operator. For example, you can use the following configuration to create a certificate for the Keycloak instance.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: keycloak-certificate
namespace: keycloak
spec:
dnsNames:
- sso.apps.ocp.aws.ispworld.at (1)
duration: 2160h0m0s
issuerRef:
kind: ClusterIssuer
name: letsencrypt-prod (2)
privateKey:
algorithm: RSA
encoding: PKCS1
rotationPolicy: Always
secretName: keycloak-certificate (3)

1	The DNS name the Certificate is valid for. This should be the same as the hostname in the Keycloak instance.
2	The issuer for the certificate. In this example, we are using the LetsEncrypt ClusterIssuer.
3	The name of the secret where the certificate is stored. This should be the same as the TLS secret in the Keycloak instance.

I strongly recommend using the Cert Manager Operator to automatically request and approve the certificate. However, if you do not have this automation in place, you can use a self-signed certificate. This certificate must be created manually and stored as a secret. For example, you can use the following command to create a self-signed certificate and store it in a secret:

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=test.keycloak.org/O=Test Keycloak./C=US"
oc create secret -n keycloak tls keycloak-certificate2 --cert=tls.crt --key=tls.key

Login in to Keycloak

Once the Keycloak instance is created and all Pods (1) are running, you can log in to the Keycloak Admin Console using the following URL: https://sso.apps.ocp.aws.ispworld.at This is the hostname we configured in the keycloak instance.

To authenticate, you need to fetch the initial password for the admin user. This password is stored in a secret called keycloak-initial-admin

You can use the following command to fetch the password:

oc extract secret/keycloak-initial-admin -n keycloak --to=-

or you can use the OpenShift Web Console to view the secret.

Once authenticated, you should see the Keycloak Admin Console:

The first thing you should do is to change the password for the admin user. I trust you know how to do this :)

Configure Keycloak to be used by OpenShift

The next steps are to configure Keycloak to be used as an Identity Provider (IDP) for OpenShift. This is done by creating a new Realm and a new Client in Keycloak. The following steps will show you the minimum configuration required to use Keycloak as an IDP for OpenShift. It does not cover all the options and features of Keycloak (and there are a lot), but it should be enough to get you started.

The full documentation for Keycloak can be found at Keycloak Documentation.

Create a new Realm and Client

In the Realm Dropdown (upper left corner) select Create new Realm
Create a new Realm called openshift (Enabled, of course) and press Create.
Now, inside the Realm openshift, select Clients and press the Create client button.
Create a new Client with the following configuration. Name it, for example, openshift.
- Be sure the Client type is set to OpenID Connect
Enable Client authentication. The rest can be left as default.
Add the following redirect URL and Web origin and press Save.:
- Redirect URL: https://oauth-openshift.apps.<your-cluster-name>/oauth2callback/* … redirecting everything under oauth2callback
- Web origin: https://oauth-openshift.apps.<your-cluster-name>;

Create a new User and a Group

In the openshift Realm, select Groups, press the Create group button and create a group called, for example, openshift-users.
In the openshift Realm, select Users and press the Add user button. Be sure to join the group openshift-users.

No more configuration is needed for the (test) user at this point.
Set the password for the user. Select the user we have just created, select the Credentials tab and press Set password. Set the password to Temporary to force the user to change the password on the first login.

Configure a Group Mapper

The above configuration is enough to log in to OpenShift using Keycloak as an IDP (except that we need to configure OpenShift itself). However, we also want to import the groups from Keycloak into OpenShift. This configuration was not easy to find, and is done by creating a Group Mapper in Keycloak.

In the openshift Realm, select Clients scopes and select the profile scope:
Select the Mappers tab and Add a mapper By configuration:
Select the Group Membership mapper.

Configure the mapper with the following settings:

Mapper Type: Group Membership
Name: openshift-groups
Token Claim Name: groups → This is the name of the claim that will be used to map the groups from Keycloak to OpenShift.
Full group path: OFF
Add to ID token: ON
Add to access token: ON

Add to userinfo: ON

Disable the Full groupo path option, otherwise the group name will be prefixed with a /. Moreover, be sure that you set the Token Claim Name correctly to the claim we will configure in OpenShift (groups).

Configure OpenShift to use Keycloak as an IDP

Now that Keycloak is configured, we need to configure OpenShift to use Keycloak as an IDP. This is done by creating a new Identity Provider in OpenShift. Before we do this, we need to create a new OAuth client secret for OpenShift in the Namespace openshift-config The secret will be used to authenticate OpenShift with Keycloak. When we created the keycloak client, we enabled the Client authentication option. This created a client secret we need to use in OpenShift.

In Keycloak, select the openshift client and select the Credentials tab and copy the Client secret.
Back in OpenShift, navigate to the openshift-config namespace and select the Secrets view. Create a new secret with the following configuration:
```
kind: Secret
apiVersion: v1
metadata:
name: openid-client-secret
namespace: openshift-config
stringData:
clientSecret: <you client secret from Keycloak> (1)
type: Opaque
```
1 The client secret we copied from Keycloak. This is the secret we will use to authenticate OpenShift with Keycloak.
Now we need to create a new Identity Provider in OpenShift. In the OpenShift Web Console, navigate to the Administration → Cluster Settings → Configuration search for OAuth and select the YAML view. Here the following must be created or added to an existing OAuth configuration:

[...]
spec:
identityProviders:
- mappingMethod: claim
name: rhsso (1)
openID:
claims: (2)
email:
- email
groups:
- groups
name:
- name
preferredUsername:
- preferred_username
clientID: openshift (3)
clientSecret:
name: openid-client-secret (4)
extraScopes: []
issuer: 'https://sso.apps.ocp.aws.ispworld.at/realms/openshift' (5)
type: OpenID (6)

1	The name of the Identity Provider. This is the name that will be displayed in the OpenShift login screen.
2	The claims that will be used to map the user to OpenShift. In this example, we are using the email, groups, name and preferred_username claims from Keycloak.
3	The client ID we created in Keycloak. This is the client ID that will be used to authenticate OpenShift with Keycloak.
4	The name of the secret we created in the openshift-config namespace.
5	The issuer URL for Keycloak. It is <hostname of keycloak>/realms/<realm name>.
6	The type of the Identity Provider. In this example, we are using OpenID Connect.

The above configuration will trigger a restart of the authentication Pods in OpenShift. Wait until all Pods have been restarted and the Operator is running again.

Test the configuration

Now it is time to test the configuration. Open a new browser window (or incognito window), navigate to the OpenShift login page and try to log in using Keycloak as an IDP rhsso.

If you have multiple IDPs configured, it is important to select the correct IDP. rhsso in this example.

By selecting the rhsso Identity Provider, you should be redirected to the Keycloak login page.

If you selected Temporary for the password, you will now be asked to change the password on the first login.
After a successful login, you should see the OpenShift Web Console, and you should be logged in as the user you created in Keycloak.
In OpenShift you will see the user created. In the Identities column you will see that it starts with rhsso, indicating that the user was authenticated using the rhsso Identity Provider.
And finally, if you navigate to User Management → Groups, you should see the group openshift-users that was created in Keycloak.

Conclusion

In this article, I have shown how to install and configure Keycloak in OpenShift for authentication. I have also shown how to configure Keycloak to be used as an Identity Provider for OpenShift and how to import groups from Keycloak into OpenShift. The biggest two challenges were to find the correct callback URL and to configure the Group Mapper in Keycloak. The rest was pretty straightforward.

What’s next? With the groups now mapped into OpenShift, you can now create RoleBindings and ClusterRoleBindings to assign the appropriate roles to the users in Keycloak. This is quite nice, as I do not need to create Users manually in OpenShift anymore (previously used HTPasswd) but instead use Keycloak as the single source of truth for users and groups. All I need to configure is the RoleBindings and ClusterRoleBindings in OpenShift.

I hope this article was helpful and you learned something new. Remember, this is just a test configuration. In production you should use a proper database and keycloak setup (high Availability, backup, etc.). If you have any questions or comments, please feel free to reach out to me on LinkedIn, Email or via GitHub issues.

References

Introducing AdminNetworkPolicies

Wed, 06 Nov 2024 00:00:00 +0000

Classic Kubernetes/OpenShift offer a feature called NetworkPolicy that allows users to control the traffic to and from their assigned Namespace. NetworkPolicies are designed to give project owners or tenants the ability to protect their own namespace. Sometimes, however, I worked with customers where the cluster administrators or a dedicated (network) team need to enforce these policies.

Since the NetworkPolicy API is namespace-scoped, it is not possible to enforce policies across namespaces. The only solution was to create custom (project) admin and edit roles, and remove the ability of creating, modifying or deleting NetworkPolicy objects. Technically, this is possible and easily done. But shifts the whole network security to cluster administrators.

Luckily, this is where AdminNetworkPolicy (ANP) and BaselineAdminNetworkPolicy (BANP) comes into play.

AdminNetworkPolicy (ANP) and BaselineAdminNetworkPolicy (BANP)

This article demonstrates the configuration of the new AdminNetworkPolicy and BaselineAdminNetworkPolicy objects using the Helm Chart admin-networkpolicies. The NetworkPolicy object is not covered in this article.

ANP and BANP are designed for cluster administrators to protect the entire cluster by creating cluster-scoped policies. They are not replacing NetworkPolicies, but instead create a tier model and can be used together. Administrators can use ANPs to enforce non-overridable policies that take precedence over NetworkPolicy objects. Administrators can use BANP to set up and enforce optional cluster-scoped network policy rules that are overridable by users using NetworkPolicy objects when necessary. When used together, ANP, BANP, and network policy can achieve full multi-tenant isolation that administrators can use to secure their cluster.

The three resources create a 3-Tier Access Control List (ACL) that is evaluated in descending order:

Tier 1 - AdminNetworkPolicy (ANP): If the traffic matches an allow or deny rule, then any existing NetworkPolicy and BaselineAdminNetworkPolicy (BANP) objects in the cluster are skipped from evaluation. If a pass rule is matched, then the evaluation is handed over to the next tier (NetworkPolicy). This means, that Cluster Administrators can enforce policies that cannot be overwritten by users (allow/deny rules) or pass the evaluation to the Network Policy, where the project owners can decide further.
Tier 2 - NetworkPolicy (NP): If the traffic passed the ANP then the NetworkPolicy is evaluating the traffic. The NetworkPolicy resources are controlled by the project owners by default.
Tier 3 - BaselineAdminNetworkPolicy (BANP): If the traffic passed the ANP and the NetworkPolicy, then the BANP is evaluating the traffic. These objects are controlled by the cluster administrators again and are cluster scoped. There can only be one BANP (named "default") configured on the cluster.

AdminNetworkPolicy

An AdminNetworkPolicy (ANP) is a cluster-scoped resource, that allow cluster administrators to secure the network traffic before NetworkPolicies in the namespaces are evaluated. These rules cannot be overwritten by project owners or developers and allow the administrators to enforce the security. Use cases could be, for example:

You want to enforce only specific egress endpoints (e.g. only allow traffic to the specific database servers)
You want to be sure that traffic from OpenShift monitoring is always allowed
You want to allow the management of NetworkPolicies to project owners and do not want to take care for them or during the onboarding.

The ANP allows cluster administrators to define:

A priority value that determines the order of its evaluation. The lower the value the higher the precedence.
A set of pods that consists of a set of namespaces or namespace on which the policy is applied.
A list of ingress rules to be applied for all ingress traffic towards the subject.
A list of egress rules to be applied for all egress traffic from the subject.

AdminNetworkPolicy Actions for Rules

The AdminNetworkPolicy allows three actions for the rules:

Allow: The traffic is allowed, and no further rules are evaluated.
Deny: The traffic is denied, and no further rules are evaluated.
Pass: The traffic is passed to the next tier (NetworkPolicy).

Subject of a Policy

In any ANP (or BANP) a subject can be defined and they specify the pods to which this AdminNetworkPolicy applies. (Note that host-networked pods are not included in subject.selection.) There are two ways to define the subject:

namespaces: The namespaces block is used to select pods via namespace selectors. Here, matchLabels or matchExpressions can be used to limit the namespaces.
pods: The Pods-Array is used to select pods via namespace AND pod selectors. Here namespaceSelector and podSelector can be set to limit the Pods.

If subject is not defined, the policy applies to all pods and namespaces in the cluster.

In my Helm chart the options are supported like the following snippets show:

Select Namespaces with matchExpressions

Values in the Helm Chart:

anp:
- name: sample-anp-rule-1
enabled: true
priority: 50
subject:
matchNamespaces: (1)
matchExpressions: (2)
- key: kubernetes.io/metadata.name
operator: NotIn
values:
- kube-system
- openshift*
- default
- kubde-info

1	matchNamespaces is used to select namespaces
2	matchExpressions is used to select namespaces with matchExpressions. In this example all namespaces that do not match (operator == NotIn) the values, so all namespaces except "kube-system, kube-info, default and openshift*" are selected.

This will result in the following AdminNetworkPolicy snippet:

 subject:
namespaces:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: NotIn
values:
- "kube-system"
- "openshift*"
- "default"
- "kubde-info"

Select Namespaces with matchLabels

Values in the Helm Chart:

anp:
- name: sample-anp-rule-1
enabled: true
priority: 5
subject:
matchNamespaces: (1)
matchLabels: (2)
apps: my-apps
tenant: my-tenant

1	matchNamespaces is used to select namespaces
2	matchLabels is used to select namespaces based on labels. In this example, all namespaces that have the labels "apps: my-apps" and "tenant: my-tenant" are selected.

This will result in the following AdminNetworkPolicy snippet:

spec:
priority: 5
subject:
namespaces:
matchLabels:
apps: "my-apps"
tenant: "my-tenant"

Select Pods with podSelectors and namespaceSelectors

Values in the Helm Chart:

anp:
- name: sample-anp-rule-1
enabled: true
priority: 5
subject:
matchPods:
- pods: (1)
namespaceSelector: (2)
labels:
kubernetes.io/metadata.name: openshift-dns
podSelector: (3)
labels:
app: dns

1	matchPods is used to select pods. Here a list of pods can be defined.
2	namespaceSelector is used to select namespaces based on labels. In this example all namespaces that have the label "kubernetes.io/metadata.name: openshift-dns" are selected.
3	podSelector is used to select pods based on labels. In this example all pods that have the label "app: dns" are selected.

This will result in the following AdminNetworkPolicy snippet:

 subject:
- pods:
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: openshift-dns
podSelector:
matchLabels:
app: dns

BaselineAdminNetworkPolicy

BaselineAdminNetworkPolicy (BANP) is a cluster-scoped resource, that allow cluster administrators to secure the network traffic after NetworkPolicies in the namespaces have been evaluated. These rules can be overwritten by project owners or developers using NetworkPolicies.

BANP is a singleton resource, meaning it can be defined only one time. Therefore, its name must be default. Moreover, the priority field is not required here.

Use cases could be, for example:

Creating default rules, such as blocking any intra-cluster traffic by default. Users will need to explicitly use NetworkPolicy objects to allow known traffic.

A BANP allows administrators to specify:

A subject that consists of a set of namespaces or namespace.
A list of ingress rules to be applied for all ingress traffic towards the subject.
A list of egress rules to be applied for all egress traffic from the subject.

BaselineAdminNetworkPolicy Actions for Rules

The BaselineAdminNetworkPolicy allows two actions for the rules. They are like the AdminNetworkPolicy, except for the pass action, which does not make sense here as BANP is the last tier (nowhere to pass).

Allow: The traffic is allowed, and no further rules are evaluated.
Deny: The traffic is denied, and no further rules are evaluated.

Examples Examples Examples

The following examples are taken directly from Kubernetes Blog: Getting started with the AdminNetworkPolicy API and Official OpenShift Documentation. Verify the values-file of the Helm Chart for the further examples.

I will show, how to configure them using the Helm Chart admin-networkpolicies and the actual result. The chart is already configured with these examples and prepared to be used with GitOps/Argo CD.

Example 1: Allow all traffic from the OpenShift monitoring namespace

Typically, it makes sense to allow the traffic from OpenShift Monitoring to all namespaces. After all, monitoring is useful :) The following example shows the possible configuration for the Helm Chart, which will render a valid ANP resource for us. It will allow ALL (including OpenShift internal Namespaces) traffic from the OpenShift monitoring namespace (labeled as kubernetes.io/metadata.name: monitoring).

---
anp:
- name: sample-anp-rule-1 (1)
enabled: true (2)
syncwave: 10
priority: 5 (3)
subject: {} (4)
ingress: (5)
- name: allow-ingress-from-monitoring (6)
enabled: true (7)
action: Allow (8)
peers: (9)
- type: namespaces
labels:
kubernetes.io/metadata.name: monitoring

1	Name of the ANP
2	Enable or disable the ANP. If disabled, the ANP will not be created. (Default is `false`)
3	Priority of the ANP. The lower the value the higher the precedence. (Default is `50`)
4	Subject of the ANP. In this case, it is empty, which means all namespaces including OpenShift internal namespaces.
5	Ingress rules of the ANP. Here a list of ingress rules for this ANP can be defined
6	Name of the ingress rule
7	Enable or disable the ingress rule. If disabled, the particular ingress rule will not be created. (Default is `false`)
8	Action of the ingress rule. In this case, it is `Allow`, which means all traffic from the OpenShift monitoring namespace will be allowed. Other options are described at AdminNetworkPolicy Actions for Rules.
9	Peers of the ingress rule. In this case, all namespaces labeled as `kubernetes.io/metadata.name: monitoring` are allowed to access all namespaces.

The ANP that will be created is the following. It is a valid ANP resource and can be applied to the cluster. (Typically applied by Argo CD) As described above it will allow incoming access from the OpenShift monitoring namespace to all namespaces.

---
apiVersion: policy.networking.k8s.io/v1alpha1
kind: AdminNetworkPolicy
metadata:
name: "sample-anp-rule-1"
labels:
helm.sh/chart: admin-networkpolicies-1.0.2
app.kubernetes.io/name: admin-networkpolicies
app.kubernetes.io/instance: release-name
app.kubernetes.io/managed-by: Helm
annotations:
argocd.argoproj.io/sync-wave: "10"
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
priority: 5
subject:
namespaces: {}
ingress:
- name: "allow-ingress-from-monitoring"
action: "Allow"
from:
- namespaces:
matchLabels:
kubernetes.io/metadata.name: "monitoring"

Example 2: Allow all traffic from labeled namespaces

As a second example, we want to allow all traffic from namespaces that are labeled with tenant: restricted to all namespaces that are labeled with anp: cluster-control-anp. This is useful, if you want to restrict access to certain namespaces. However, the rule action is configured as Pass which means that the traffic will be allowed but might be further restricted by a NetworkPolicy in the tenant namespace.

---
anp:
- name: sample-anp-rule-2
enabled: true
priority: 5
subject:
matchNamespaces: (1)
matchLabels:
anp: cluster-control-anp (2)
ingress:
- name: pass-from-restricted-tenants
enabled: true
action: Pass (3)
peers:
- type: namespaces (4)
labels:
tenant: restricted

1	Subject of the ANP. In this case, we select based on labels.
2	Label selector for the namespaces. In this case, all namespaces that are labeled with `anp: cluster-control-anp` are subject of this ANP.
3	Action of the ingress rule. In this case, it is `Pass`, which means the traffic is allowed, but might be restricted by NetworkPolicies in the tenant namespace. Other options are described at AdminNetworkPolicy Actions for Rules.
4	Peers of the ingress rule. In this case, all namespaces labeled as `tenant: restricted` are allowed to access all namespaces.

---
apiVersion: policy.networking.k8s.io/v1alpha1
kind: AdminNetworkPolicy
metadata:
name: "sample-anp-rule-2"
labels:
helm.sh/chart: admin-networkpolicies-1.0.2
app.kubernetes.io/name: admin-networkpolicies
app.kubernetes.io/instance: release-name
app.kubernetes.io/managed-by: Helm
annotations:
argocd.argoproj.io/sync-wave: "10"
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
priority: 5
subject:
namespaces:
matchLabels:
anp: "cluster-control-anp"
ingress:
- name: "pass-from-restricted-tenants"
action: "Pass"
from:
- namespaces:
matchLabels:
tenant: "restricted"

Example 3: Show possible peers settings

The most important settings for the rules are the peers settings. The following examples show the snippets of possible peers. For further information, please refer to the example in the values file: Helm Chart Values with further examples

The following rules are examples of EGRESS rules.

Allow egress traffic to namespaces labeled splunk on ports 80 and 443:

 peers:
- type: namespaces
labels:
tenant: splunk
ports:
- protocol: TCP
portNumber: 80
- portName: https

Allow egress traffic to nodes where the key "node-role.kubernetes.io/control-plane" exists on the port 6443:

 peers:
- type: nodes
expr:
- key: node-role.kubernetes.io/control-plane
operator: Exists
ports:
- protocol: TCP
portNumber: 6443

Allow egress traffic to pods labeled "app: dns" in the namespace openshift-dns on the port 53 and 5353:

 peers:
- type: pods
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: openshift-dns
podSelector:
matchLabels:
app: dns
ports:
- protocol: TCP
port: 5353
- protocol: TCP
port: 53
- protocol: UDP
port: 53
- protocol: UDP
port: 5353

Allow egress traffic to a list of IPs:

 peers:
- type: networks
ips:
- 172.29.0.0/30
- 10.0.54.0/19
- 10.0.56.38/32
- 10.0.69.0/24

Allows egress traffic to a list of domain names (*.kubernetes.io and kubernetes.io)

 peers:
- type: domainNames
domains:
- '*.kubernetes.io'
- kubernetes.io

Deny all egress traffic. This should be the last rule when full egress traffic shall be disabled. This might also be put into the BANP.

 - name: default-deny
enabled: true
action: Deny
peers:
- type: networks
ips:
- 0.0.0.0/0

Example 4: BaselineAdminNetworkPolicy

The BANP is more or less identical to ANP, except that you cannot define a "name" and a "priority". The following example creates a BANP that allows incoming and outgoing traffic to namespaces labeled "tenant-1".

banp: (1)
- enabled: true (2)
syncwave: 10
subject:
matchNamespaces:
matchLabels:
kubernetes.io/metadata.name: example.name
ingress:
- name: "deny-all-ingress-from-tenant-1"
enabled: true
action: Deny
peers:
- type: pods
namespaceSelector:
matchLabels:
custom-banp: tenant-1
podSelector:
matchLabels:
custom-banp: tenant-1
egress:
- name: allow-all-egress-to-tenant-1
enabled: true
action: Allow
peers:
- type: pods
namespaceSelector:
matchLabels:
custom-banp: tenant-1
podSelector:
matchLabels:
custom-banp: tenant-1

1	Using the key banp (instead of anp)
2	No name or priority are defined here.

Further Information

[Ep.6] Setup & Configure Advanced Cluster Security

Sun, 28 Apr 2024 00:00:00 +0000

Today I want to demonstrate the deployment and configuration of Advanced Cluster Security (ACS) using a GitOps approach. The required operator shall be installed, verified if it is running and then ACS shall be initialized. This initialization contains the deployment of several components:

Central - as UI and as a main component of ACS
SecuredClusters - installs a Scanner, Controller pods etc.
Console link into OpenShift UI - to directly access the ACS Central UI
Job to create an initialization bundle to install the Secured Cluster
Job to configure authentication using OpenShift

Let’s start …

Prerequisites

Argo CD (OpenShift GitOps) deployed
App-Of-Apps deployed

Introduction

The main components of ACS are the two custom resources: Central and SecuredCluster. The recommended way to deploy ACS is to use the Operator (alternatives would be the command line or Helm Chart). When I first came across ACS I thought about how to automate the full deployment. I did not want to install the Operator, then the Central, then manually create a so-called init-bundle, then deploy the Secured Cluster, then find the route that has been used, then find the secret that stores the initial administrator credentials and then, finally, log into ACS and activate OpenShift authentication.

As you can see there are a lot of tasks to do before I can start a customer demo.

At the same time, I started to dig into GitOps and I thought this would be a good option to create my very first Helm Chart.

Long story short, I have now a Helm Chart (actually three, because I outsourced some of the features into sub-charts) that automatically does all these things above. Once I am synchronizing my Argo CD Application everything will happen automatically.

The Configure App-of-Apps installed an Argo CD Application called in-cluster-setup-acs:

Figure 1. Argo CD Application: setup-acs

This Argo CD Application used the following path to find the Helm Chart: setup-acs

This Helm chart is a wrapper chart that uses sub-charts as dependencies to install and configure the operator as well as to do some OpenShift Jobs on top, for example, creating a ConsoleLink or creating an init-bundle.

Please check out the article Setup Compliance Operator on why I am using a wrapper chart.

Installing Advanced Cluster Security

Analysing Chart.yaml

Let’s examine the Chart.yaml file to see which dependencies are used:

The file looks like the following. Three sub-charts are defined as required to deploy and configure the ACS. This is pretty much the same as it was for the Setup Compliance Operator.

apiVersion: v2
name: setup-acs
description: Deploys Advanced Cluster Security (ACS) on target cluster. If enabled Central will be deployed too.
version: 1.0.0
dependencies:
- name: rhacs-setup (1)
version: ~1.0.0 (2)
repository: https://charts.stderr.at/
- name: helper-operator (3)
version: ~1.0.23
repository: https://charts.stderr.at/
- name: helper-status-checker (4)
version: ~4.0.0
repository: https://charts.stderr.at/
condition: helper-status-checker.enabled (5)

1	Dependency: RHACS Setup
2	Version that will be used. The "~" means that the latest version of 1.0.X will be used.
3	Dependency: Helper Operator
4	Dependency: Helper Status Checker
5	Only use this dependency when "enabled" is set

Verify the READMEs of the different Charts for detailed information on how to configure them.

Configuration of the Chart

To configure Advanced Cluster Security the values file of the wrapper Chart must be prepared accordingly.

The important thing here is, that any value that should be bypassed to a sub-chart is defined under the name of the sub-chart. For example, everything under helper-operator: will be sent to the helper-operator Chart and is used there for its configuration.

Check out the example values file I use to configure ACS and the README to find further information about the possible settings that can be done.

Let’s check the main example, to quickly start:

Installing and verifying the Operator

The first thing to do is to deploy the Operator and to verify if the Operator installation finished successfully.

The two Helm Charts helper-operator and helper-status-checker are responsible to do so.

They are configured as follows:

helper-operator:
operators:
rhacs-operator: (1)
enabled: true (2)
syncwave: '0' (3)
namespace:
name: rhacs-operator (4)
create: true
subscription:
channel: stable (5)
approval: Automatic
operatorName: rhacs-operator
source: redhat-operators
sourceNamespace: openshift-marketplace
operatorgroup: (6)
create: true
# rhacs does not support to monitor own namespace,
# therefore the spec in the OperatorGroup must be empty
notownnamespace: true
# Subchart helper-status-checker
# checks if ACS operator is ready
helper-status-checker:
enabled: true (7)
checks: (8)
- operatorName: rhacs-operator (9)
namespace:
name: rhacs-operator (10)
syncwave: 3
serviceAccount:
name: "status-checker-acs" (11)

1	Key that can be freely defined. Theoretically, you can deploy multiple operators at once.
2	Is this Operator enabled yes/no.
3	Syncwave for the Operator deployment. (Subscription and OperatorGroup etc.)
4	The Namespace where the Operator shall be deployed and if this namespace shall be created.
5	Configuration of the Subscription resource.
6	Configuration of the OperatorGroup
7	Enable status checker or not. Default: false
8	List of operators to check. Typically, only one is checked, but there could be more.
9	Name of the Operator to check (same as for helper-operator)
10	Namespace where the Operator has been installed (same as for helper-operator)
11	Name of the ServiceAccount that will be created to check the status.

Verify the READMEs at Helper Operator and Helper Operator Status Checker to find additional possible configurations.

Also verify the separate article Operator Installation with Argo CD to understand why I am verifying the status of the Operator installation.

Configuring Advanced Cluster Security

Besides the deployment of the Operator, the configuration of ACS is the most important part here. The ACS Operator provides two custom resources: Central and SecuredCluster. On the Central cluster both CRDs are required. On any other (spoke) cluster, the SecuredCluster resource is enough.

In the following example, I am going to configure both Central and SecuredCluster. Since the values file is quite huge I removed most of the additional comments, to keep this article short and readable. You can read the example values file or the README at Advanced Cluster Security Chart to find additional possible configurations. Especially, if you like to add tolerations or set resource limits.

#########################################
# Settings for Advanced Cluster Security
#########################################
rhacs-setup:
rhacs:
namespace: (1)
name: stackrox
syncwave: '0'
descr: 'Red Hat Advanced Cluster Security'
################
# CENTRAL of ACS
################
# Settings for the Central of ACS
central: (2)
enabled: true
syncwave: '3'
egress:
connectivityPolicy: Online
###############
# CENTRAL DB
###############
# Settings for Central DB, which is responsible for data persistence.
db: (3)
# -- Set Central DB resources.requests for a DEMO environment to save resources.
resources:
requests:
cpu: '1'
memory: '1Gi'
# -- If you want this component to only run on specific nodes, you can
# configure tolerations of tainted nodes.
tolerations: {}
# - effect: NoSchedule
# key: infra
# operator: Equal
# value: reserved
# - effect: NoSchedule
# key: infra
# operator: Equal
# value: reserved
###############
# SCANNER
###############
scanner: (4)
enabled: true
analyzer:
# The following settings are NOT suitable for a production environment
autoscaling:
status: "Disabled"
max: 1
min: 1
# When autoscaling is disabled, the number of replicas will always be
# configured to match this value.
replicas: 1
tolerations: {}
###############
# SCANNER DB
###############
db:
tolerations: {}
#################
# SECURED CLUSTER
#################
secured_cluster: (5)
enabled: true
syncwave: '4'
clustername: local-cluster
sensor:
tolerations: {}
admissioncontrol:
listenOn:
creates: true
events: true
updates: true
tolerations: {}
# -- Basic settings for ACS authentication
# This configuration is done by a Job, that will configure the OpenShift oauth for ACS.
basic_acs_settings: (6)
auth_provider: 'OpenShift'
auth_provider_type: 'openshift'
min_access_role: 'None'
syncwave: 5
####################################################
# Additional settings for Central and possible Jobs
####################################################
job_vars: (7)
max_attempts: 20
job_init_bundle: (8)
enabled: true
syncwave: '3'
consolelink: (9)
enabled: true
syncwave: '3'
location: ApplicationMenu
text: Advanced Cluster Security
section: Observability

1	Create the Namespace stackrox and install the ACS resources there.
2	Enable the Central during Syncwave 3 and set the connectivityPolicy to Online
3	The Central DB and its configuration. Here the resource requests are modified to allow a small installation on the DEMO environment. Also, tolerations might be set here, as well a PVCs etc.
4	Settings for the Scanner and its databases. Again, tolerations might be configurated here, but also, not shown in this example, resource limits and requests and other settings. Since I am configuring for a DEMO environment, I disabled the autoscaler and set the replica to 1.
5	The SecuredCluster is the 2nd CRD that is provided by ACS Operator. It is installed after the Central (thus a higher Syncwave). The most important setting here is the clustername. In our "local" example, the name is set to local-cluster.
6	Some basic settings, that will configure the OpenShift authentication and the minimum role for authenticated users (None)
7	Some default settings for Jobs that are started by this Helm chart.
8	The Job that initializes the creation of the init-bundle
9	The Job and its configuration to generate a direct link to ACS in the OpenShift UI.

With this ACS is about to be installed on the cluster. Let’s see what will happen during the synchronization.

Deploying Advanced Cluster Security (ACS)

Let’s hit the sync button inside OpenShift GitOps. This will start the whole process, walking through the syncwaves and the hooks that have been defined.

Figure 2. Syncing Argo CD

Since hooks are used, you must sync the whole Argo CD Application. As you can see inside Argo CD, the hooks are not shown, because they will only appear when their time has come (and will disappear afterward again). This means, that if you perform a selective sync, Argo CD does not know when it should start such a hook and they are never triggered.

The Operator installation is now started. At the moment the Operator has the status Installing. Currently, no CRDs (Central or SecuredCluster) are available yet. If we would just let Argo CD continue, it would try to create the Central configuration, based on a CRD which does not yet exist. Thus, the syncing process will fail and therefore the status-checker is going to verify if the installation was truly successful.

Figure 3. Operator is installed

The Status Checker is a simple Pod that is triggered by a Kubernetes Job. If waits until the status of the Operator is Succeeded. Until this is the case, Argo CD waits before it continues with the synchronization. (It waits until the hook ends the Job)

Figure 4. Status Checker

In the logs file of the Pod, we can see that the Operator is ready.

Figure 5. Status Checker Logs

And indeed, the status of the Operator is now Succeeded. Now it is time for Argo CD to continue the synchronization.

Figure 6. Operator Ready

The next step is to create the Central CRD. This will deploy the UI of ACS and the local Scanner. You can also see two other Jobs that have been created by the Helm Charts create_cluster_link and create_cluster_init_bundle. They will finish when the Central becomes ready.

Until the Central becomes ready, these two additional Jobs may show errors. Do not worry, OpenShift will reschedule them.

Figure 7. Installing Central

You can also see two other Jobs that have been created by the Helm Charts create_cluster_link and create_cluster_init_bundle. They will finish when the Central becomes ready.

Until the Central becomes ready, these two additional Jobs may show errors. Do not worry, OpenShift will reschedule them.

Figure 8. Init Job is waiting for Central

Once the Central has been deployed, the second CRD SecuredCluster will be added. This will trigger the installation of the Collectors and other components.

Figure 9. Installing SecuredCluster

Eventually, all Pods are running at the end. The additional Jobs are completed and ACS is ready to take care of the cluster security.

Figure 10. All Pods running

We can now use the console link that was created.

Figure 11. ACS ConsoleLink

If you disabled the creation of the ConsoleLink you would need to find the Route that ACS Operator created. Honestly, I do not know why the Operator does not create such ConsoleLink out of the box.

Since I am lazy I created another Job that automatically configures authentication via OpenShift for ACS. This way, we can simply use our OpenShift credentials to login.

Figure 12. ACS Login

And that’s it, we can now use ACS, which was deployed fully automatically.

Figure 13. Advanced Cluster Security

Conclusion

As you can see Advanced Cluster Security was completely installed. This included the Operator, the Central, the creation and installation of an init-bundle, the creation of a ConsoleLink, the configuration of the SecuredCluster CRD and the initial configuration of the auth provider inside ACS. You can now start using ACS or add additional clusters.

Speaking of additional clusters: The create-cluster-init-bundle created three certificates: collector-tls, sensor-tls and admission-control-tls. They are required so that the SecuredClusters can communicate with the Central. You could now create a separate init-bundle for each SecuredCluster, which is not really easy to automate, or you simply take these created secrets and put them into your GitOps and re-use them for any other SecuredCluster.

All these steps and configurations seem quite complicated but honestly, it is straightforward. I install any Operator using Helper Operator and in most cases. I also use Helper Operator Status Checker to find additional possible configurations. Both require simple configuration only, which you would need to know anyway when you create the Subscription object manually. Once done, you can repeat this for any other Operator.

The real magic happens when the Operator is configured at the same time because this is very individual to the Operator.

[Ep.5] Setup & Configure Compliance Operator

Thu, 25 Apr 2024 00:00:00 +0000

In the previous articles, we have discussed the Git repository folder structure and the configuration of the App-Of-Apps. Now it is time to deploy our first configuration. One of the first things I usually deploy is the Compliance Operator. This Operator is recommended for any cluster and can be deployed without any addition to the Subscription.

In this article, I will describe how it is installed and how the Helm Chart is configured.

Prerequisites

Argo CD (OpenShift GitOps) deployed
App-Of-Apps deployed

Introduction

As a reminder, at Git repository folder structure I described my preferred folder structure. I would like to deploy the Compliance Operator in the Management Cluster now. All my examples can be found at GitHub repository OpenShift Clusterconfig GitOps. The folder clusters/management-cluster/setup-compliance-operator is the one I am interested in.

Inside this folder, you will find another Helm Chart. The Helm Chart has no local templates, instead, it uses dependencies to call other (sub-) charts. However, the values.yaml is the main part to configure everything.

In case you want to have any local template, that you do NOT want to integrate into one of the sub-charts, you can easily do so, by storing them in the templates folder.

Why "empty" Helm Charts?

Actually, it would be possible to use the Helm Chart of the Chart repository directly, without creating a separate chart, that does nothing else than using dependency charts.

The reasons why I am using such an "empty" Chart are the following (in no particular order):

With that way it is possible to add templates (i.e. SealedSecrets) and modify the values-file without packaging and releasing a new Chart version every time you change a small thing.
The Multi-Source Option, which allows you to use a Helm Chart from repository A and a values file from repository B is still a TechPreview feature (Argo CD 2.10). I am using this for the App-of-Apps already, but I did not do this for all charts. This feature is on the list for Argo CD version 2.11 to become globally available.

As an alternative, it is also possible to mix Kustomize and Helm. That way you only need a kustomization.yaml file and reference to a Helm Chart. In the folder clusters/management-cluster/ingresscontroller I have such an example.

Installing Compliance Operator

Analysing Chart.yaml

As any Helm Chart a Chart.yaml file exists, that stores the basic information. The most important ones for now are the dependencies.

The file looks like the following. Three sub-charts are defined as required to deploy and configure the Compliance Operator.

apiVersion: v2
name: setup-compliance-operator
description: Deploy and configure the Compliance Operator
version: 1.0.1
dependencies:
- name: compliance-operator-full-stack (1)
version: ~1.0.0 (2)
repository: https://charts.stderr.at/
- name: helper-operator (3)
version: ~1.0.21
repository: https://charts.stderr.at/
- name: helper-status-checker (4)
version: ~4.0.0
repository: https://charts.stderr.at/
condition: helper-status-checker.enabled (5)

1	Dependency: Compliance Operator Full Stack
2	Version that will be used. The "~" means that the latest version of 1.0.X will be used.
3	Dependency: Helper Operator
4	Dependency: Helper Status Checker
5	Only use this dependency when "enabled" is set

Verify the READMEs of the different Charts for detailed information on how to configure them.

As you can see three other Helm Charts are used to actually deploy and configure the Operator.

Configuration of the Chart

To configure the Compliance Operator, the values files must be prepared accordingly.

The following is a full example of the values I typically use.

# Install Operator Compliance Operator
# Deploys Operator --> Subscription and Operatorgroup
helper-operator:
operators:
compliance-operator:
enabled: true
syncwave: '0'
namespace:
name: openshift-compliance
create: true
subscription:
channel: stable
approval: Automatic
operatorName: compliance-operator
source: redhat-operators
sourceNamespace: openshift-marketplace
operatorgroup:
create: true
notownnamespace: true
# Verify if the Operator has been successfully deployed
helper-status-checker:
enabled: true
checks:
- operatorName: compliance-operator
namespace:
name: openshift-compliance
serviceAccount:
name: "status-checker-compliance"
# Setting for the Compliance Operator
compliance-operator-full-stack:
compliance:
namespace:
name: openshift-compliance
syncwave: '0'
descr: 'Red Hat Compliance'
scansettingbinding:
enabled: true
syncwave: '3'
profiles:
- name: ocp4-cis-node
kind: Profile # Could be Profile or TailedProfile
- name: ocp4-cis
kind: Profile
scansetting: default

Let us walk through the settings in more detail.

Installing the Operator

The first thing to do is to deploy the Operator. Two resources are relevant to install an Operator:

Subscription
OperatorGroup

Both objects should be deployed at the very beginning of Argo CD synchronisation. This is done by setting the Syncwave to 0.

The main settings are the operatorName, the channel (which is the version of the operator) and the approval (which defines if the Operator is updated automatically or manually).

In addition, a Namespace object is deployed, because this Operator should run in its very own namespace.

This will start the Operator installation process.

helper-operator:
operators:
compliance-operator: (1)
enabled: true (2)
syncwave: '0' (3)
namespace:
name: openshift-compliance (4)
create: true
subscription: (5)
channel: stable # Version of the Operator
approval: Automatic # Automatic or Manual
operatorName: compliance-operator # Name of the Operator
source: redhat-operators
sourceNamespace: openshift-marketplace
operatorgroup: (6)
create: true
notownnamespace: true

1	Key that can be freely defined. Theoretically, you can deploy multiple operators at once.
2	Is this Operator enabled yes/no.
3	Syncwave for the Operator deployment. (Subscription and OperatorGroup etc.)
4	The Namespace where the Operator shall be deployed and if this namespace shall be created.
5	Configuration of the Subscription resource.
6	Configuration of the OperatorGroup

Verify the README at Helper Operator to find additional possible configurations.

Verify the Status of the Operator

After Argo CD creates the subscription and operatorgroup resources (and namespace), OpenShift will start the installation of the Operator. This installation will take a while but Argo CD does not see this. All it sees is that the Subscription resource is available and it tries to continue with the configuration of the Operator. Here it will fail because the CRDs are not available yet.

Therefore, I created a mechanism to verify if an Operator is ready or not.

Also verify the separate article Operator Installation with Argo CD that addresses the problem in more detail.

All it does is to start a small Job inside OpenShift and to verify the status of the Operator installation. If everything is fine, the Job will end successfully and Argo CD will continue with the next syncwave. Argo CD Hook and syncwaves are required here. The Job should be started after the Subscription/OperatorGroup resources have been created, which means any syncwave after "0".

The following annotations will be used by the Job:

 argocd.argoproj.io/hook: Sync (1)
argocd.argoproj.io/hook-delete-policy: HookSucceeded (2)
argocd.argoproj.io/sync-wave: {{ .syncwave | default 1 | quote }} (3)

1	Hooks are ways to run scripts before, during, and after a Sync operation.
2	Deletes the OpenShift Job again. The hook resource is deleted after the hook succeeded (e.g. Job/Workflow completed successfully).
3	Syncwave: can be configured. Must be after helper-operator (default 0) and before the Operator is configured further. Default value is 1.

The configuration for hepler_status_checker will look like the following:

# Verify if the Operator has been successfully deployed
helper-status-checker:
enabled: true (1)
checks: (2)
- operatorName: compliance-operator (3)
namespace:
name: openshift-compliance (4)
serviceAccount:
name: "status-checker-compliance" (5)

1	Enable status checker or not. Default: false
2	List of operators to check. Typically, only one is checked, but there could be more.
3	Name of the Operator to check (same as for helper-operator)
4	Namespace where the Operator has been installed (same as for helper-operator)
5	Name of the ServiceAccount that will be created to check the status.

Verify the README at Helper Operator Status Checker to find additional possible configurations.

Configuring Compliance Operator

Finally, the Operator has been deployed and has been verified. Now the time is right to configure the Operator with any configuration we would like. This means, using CRDs to do whatever the Operator offers.

This is reflected in the following part of the values file. All these settings are handed over to the sub-chart compliance-operator-full-stack.

Verify the README at Compliance Operator Chart to find additional possible configurations. Especially, if you like to do Tailored Profiles.

The compliance operator requires a so-called ScanSettingBinding that uses Profiles which are used to check the cluster compliance once a day. In this case, I am using CIS Benchmarks. There are two profiles:

ocp4-cis-node: will check the node operating system for missing but suggested configuration.
ocp4-cis: will check the OpenShift cluster for missing but suggested configuration.

# Setting for the Compliance Operator
compliance-operator-full-stack: (1)
compliance:
namespace:
name: openshift-compliance (2)
syncwave: '0'
descr: 'Red Hat Compliance'
scansettingbinding: (3)
enabled: true
syncwave: '3'
profiles: (4)
- name: ocp4-cis-node
kind: Profile # Could be Profile or TailedProfile
- name: ocp4-cis
kind: Profile
scansetting: default

1	Handing everything that comes below to the sub-chart compliance-operator-full-stack
2	Namespace where the configuration should be deployed. The Syncwave at this point could be omitted.
3	The configuration for the ScanSettingBinding. It is enabled (default = false) and has a Syncwave AFTER the helper-status-checker.
4	The list of profiles that shall be used. These must exist. The Compliance Operator offers several profiles. I usually use these two for full CIS compliance check.

Conclusion

With this configuration, the Compliance Operator will not only be installed but also configured with the same Argo CD Application. All you need to do is to synchronize Argo CD and let the magic happen. After a few minutes, everything should be in sync.

Figure 1. Sync Compliance Operator

Inside OpenShift the Operator is configured and starts doing its job:

Figure 2. Configured Compliance Operator

This concludes the deployment of the Compliance Operator. For further information about the Operator itself, please read the documentation or articles:

Also, be sure to check out the READMEs of the different Charts:

If you have any questions or problems, feel free to create a GitHub issue at any time.

Secrets Management - Vault on OpenShift

Tue, 16 Aug 2022 00:00:00 +0000

Sensitive information in OpenShift or Kubernetes is stored as a so-called Secret. The management of these Secrets is one of the most important questions, when it comes to OpenShift. Secrets in Kubernetes are encoded in base64. This is not an encryption format. Even if etcd is encrypted at rest, everybody can decode a given base64 string which is stored in the Secret.

For example: The string Thomas encoded as base64 is VGhvbWFzCg==. This is simply a masked plain text and it is not secure to share these values, especially not on Git. To make your CI/CD pipelines or Gitops process secure, you need to think of a secure way to manage your Secrets. Thus, your Secret objects must be encrypted somehow. HashiCorp Vault is one option to achieve this requirement.

HashiCorp Vault

HashiCorp Vault is a solution to solve our Secret management problem. It stores and encrypts our sensitive information at rest and in transit and enables you to create fine grained access controls (ACL). This defines who has access to a specific secret. Application A should only get access to secret A and so on. However, that is just the tip of the iceberg. Vault can do much more. If you are interested in further details, check out the introduction video by Armon, the Co-Founder of Hashicorp Vault at: https://learn.hashicorp.com/tutorials/vault/getting-started-intro?in=vault/getting-started

For this article we will keep it simple and cover:

Installing HashiCorp Vault to OpenShift
Integrating the plugin "Kubernetes Authentication" to access the Secrets
Accessing a static secret stored in HashiCorp Vault by an example application

Installing Vault

The easiest way to install Vault is by using the supported Helm chart.

Add the Helm repository to your repo:

helm repo add hashicorp https://helm.releases.hashicorp.com

Update the repository to get the latest updates.

helm repo update

Before we install Vault, we need to create a values file to set certain variables. Let’s create a file called "overwrite-values.yaml" with the following content

global:
openshift: true
server:
ha:
enabled: true
replicas: 3
raft:
enabled: true

This will tell HashiCorp Vault the environment we are going to install is (OpenShift), enables high availability with 3 replicas and enables RAFT (see below for some introduction).

Finally, let us deploy HashiCorp Vault into the namespace vault using the values file we have created in the previous step:

helm upgrade --install vault hashicorp/vault --values bootstrap/vault/overwrite-values.yaml --namespace=vault --create-namespace

This will start the agent-injector and 3 vault-pods:

oc get pods -n vault
NAME READY STATUS RESTARTS AGE
vault-0 0/1 Running 0 36s
vault-1 0/1 Running 0 36s
vault-2 0/1 Running 0 36s
vault-agent-injector-74c848f67b-sq4dq 1/1 Running 0 37s

vault agent injector: detecting applications with annotations that require vault agent which will get injected
vault-0: vault server

The vault servers remain in not-ready state until Vault has been unsealed by you.

When you check the logs of one of the pods you will see:

oc logs vault-0 -n vault
...
2022-08-11T12:31:22.497Z [INFO] core: security barrier not initialized
2022-08-11T12:31:22.497Z [INFO] core: seal configuration missing, not initialized

Vault always starts uninitialized and sealed, to protect the secrets. You need to give at least three different unseal-keys in order to be able to use Vault.

Vault’s seal mechanism uses Shamir’s secret sharing. This is a manual process to secure the cluster if it restarts. You can use auto-unseal for specific cloud providers to bypass the manual requirement.

Raft Storage

During the deployment we have enabled Raft which is the integrated HA storage for Vault. Vault can use several different styles of storage backends, but when it comes to HA and Kubernetes, Raft is easiest one since it has no other dependencies, and it is well known inside OpenShift. Etcd is using Raft as well. It is a distributed Consensus Algorithm where multiple members form a cluster and elect one leader. The leader has the responsibility to replicate everything to the followers. Since this leader election requires a majority an odd number of cluster members must be available to ensure there is a minimum number left if a member is failing.

Initialize and Unseal Vault

As mentioned, the Vault Pods are not fully available yet, since Vault it currently sealed and cannot be used. The first thing to do is to initialize and unseal it.

The following commands will login into one Pod and execute commands from there.

Let’s get the status of our Vault fist:

oc -n vault exec -it vault-0 -- vault operator init -status

This will return the message: Vault is not initialized

The following command will initialize Vault for further usage:

oc exec -ti -n vault vault-0 -- vault operator init -format=json > unseal.json

This will create the file unseal.json locally on your machine. Keep this file secure It contains by default 5 key shards and the root token you will need to unseal Vault and authenticate as root.

Keep this file secure, it contains keys to unseal Vault and the root_token to authenticate against.

Never share this file. It will look like this:

{
"unseal_keys_b64": [
"key_1",
"key_2",
"key_3",
"key_4",
"key_5"
],
"unseal_keys_hex": [
"key_hex_1",
"key_hex_2",
"key_hex_3",
"key_hex_4",
"key_hex_5"
],
"unseal_shares": 5,
"unseal_threshold": 3,
"recovery_keys_b64": [],
"recovery_keys_hex": [],
"recovery_keys_shares": 5,
"recovery_keys_threshold": 3,
"root_token": "root.token"
}

With the initialization in place Vault is put into a sealed mode. This means Vault cannot decrypt secrets at this moment. To unseal Vault you need the unseal key, which is split into multiple shards using Shamir’s secret sharing. A certain number of individual shards (default 3) must be provided to reconstruct the unseal key.

To unseal Vault lets login to our Pod "vault-0" and unseal it. Use the following command and provide one of the keys:

oc exec -ti -n vault vault-0 -- vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true (1)
Sealed true (2)
Total Shares 5
Threshold 3
Unseal Progress 1/3 (3)
Unseal Nonce 08b01535-be15-e865-251c-f948ed0661c9
Version 1.11.2
Build Date 2022-07-29T09:48:47Z
Storage Type raft
HA Enabled true

1	Vault is initialized
2	Vault is still sealed
3	The unseal progress: Currently 1 out of 3 keys have been provided

Use the same command another 2 times using different keys to complete the unseal process:

At the end the following output should be shown:

Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false (1)
Total Shares 5
Threshold 3
Version 1.11.2
Build Date 2022-07-29T09:48:47Z
Storage Type raft
Cluster Name vault-cluster-f7402e5b (2)
Cluster ID aff648f0-b3a2-1fdd-12f6-492842b08b2b
HA Enabled true
HA Cluster https://vault-0.vault-internal:8201
HA Mode active (3)
Active Since 2022-08-16T07:20:45.828215961Z
Raft Committed Index 36
Raft Applied Index 36

1	Vault is now unsealed
2	Name of our cluster
3	High availability is enabled

Vault-0 is now initialized, but there are 2 other members in our HA cluster which must be added. Let vault-1 and vault-2 join the cluster and perform the same unseal process as previously: use 3 different keys:

oc exec -ti vault-1 -n vault -- vault operator raft join http://vault-0.vault-internal:8200
# 3 times....
oc exec -ti vault-1 -n vault -- vault operator unseal
oc exec -ti vault-2 -n vault -- vault operator raft join http://vault-0.vault-internal:8200
# 3 times...
oc exec -ti vault-2 -n vault -- vault operator unseal

Verify Vault Cluster

To verify if the Raft cluster has successfully been initialized, run the following.

First, login using the root_token, that was created above, on the vault-0 pod.

oc exec -ti vault-0 -n vault -- vault login
Token (will be hidden): <root_token>

oc exec -ti vault-0 -n vault -- vault operator raft list-peers

This should return:

Node Address State Voter
---- ------- ----- -----
16ec7490-f621-42ea-976d-5f054cfaeecc vault-0.vault-internal:8201 leader true
60ba2885-432a-c7d3-d280-a824f0acce42 vault-1.vault-internal:8201 follower true
bcc4f551-79bc-47e5-d01b-97fc12d1afa5 vault-2.vault-internal:8201 follower true

As you can see Vault-0 is the leader while the other two members are followers.

Configure Kubernetes Authentication

There are multiple ways how an application can interact with Vault. One example is to use Tokens. This is quite easy but has the disadvantage that it does require additional steps of managing the life cycle of such token, moreover they might be shared, which is not what we want.

HashiCorp Vault supports different authentication methods. One of which is the Kubernetes Auth Method that must be enabled before we can use. The Kubernetes Auth Method makes use of Jason Web Tokens (JWT)s that are bound to a Service Account. When we tell Vault that a Service Account is fine to authenticate, then a Deployment using this account is able to authenticate and request Secrets.

Vault has a plugin ecosystem, which allows to enable certain plugins. To enable Kubernetes Auth Method use the following process:

Login vault-0 pods oc exec -it vault-0 -n vault — /bin/sh
execute the command: vault auth enable kubernetes which returns: Success! Enabled kubernetes auth method at: kubernetes/
Set up the Kubernetes configuration to use Vault’s service account JWT.

the address to the OpenShift API (KUBERNETES_PORT_443_TCP_ADDR) is automatically available via an environment variable.

vault write auth/kubernetes/config issuer="" \
token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Success! Data written to: auth/kubernetes/config

With this step authentication against OpenShift is enabled.

Configure a Secret

With the Kubernetes Auth Method in place we can configure a secret to test our setup. We will use an example application called expenses that has a MySQL database. The static password to bootstrap this database shall be stored in Vault. A plugin called key-value secrets engine will be used to achieve this.

There are other plugins that are specifically designed to automatically rotate secrets. For example, it is possible to dynamically create user credentials für MySQL.

You can list available engines by using the command: oc -n vault exec -it vault-0 — vault secrets list

There are currently two versions of this key/value engine:

KV Version 1: does not versionize the key/values, thus updates will overwrite the old values.
KV Version 2: does versionize the key/value pairs

In our example we will use version 2.

Like the authentication method, we need to enable the secrets engine:

oc -n vault exec -it vault-0 -- vault secrets enable \
-path=expense/static \ (1)
-version=2 \ (2)
kv (3)

1	API path where our secrets are stored
2	Version 2
3	name of our engine

You can list the enabled engines with the following command:

oc -n vault exec -it vault-0 -- vault secrets list
Path Type Accessor Description
---- ---- -------- -----------
cubbyhole/ cubbyhole cubbyhole_f1e955f9 per-token private secret storage
expense/static/ kv kv_6db09e5d n/a
identity/ identity identity_ca05e6ab identity store (1)
sys/ system system_e34a76c3 system endpoints used for control, policy and debugging

1	Enabled KV secrets engine using the path expense/static/

Now lets put a secret into our store. We will store our super-secure MySQL password into expense/static/mysql

MYSQL_DB_PASSWORD=mysuperpassword$
oc -n vault exec -it vault-0 -- vault kv put expense/static/mysql db_login_password=${MYSQL_DB_PASSWORD}

This command will store the key db_login_password with the database as value. We can get the secret by calling:

oc -n vault exec -it vault-0 -- vault kv get expense/static/mysql
====== Secret Path ======
expense/static/data/mysql (1)
======= Metadata =======
Key Value
--- -----
created_time 2022-08-17T06:08:05.839663508Z
custom_metadata <nil>
deletion_time n/a
destroyed false
version 1
========== Data ==========
Key Value
--- -----
db_login_password mysuperpassword$ (2)

1	The data path of our secret
2	our password

Configuring policies

The Secret is now stored at expense/static/mysql but there is no policy in place. Everybody who is authenticated and is calling this path will get to see the secrets. Luckily, one or more policies can be assigned to the authentication method. A policy defines capabilities that allow you to perform certain actions.

The following capabilities are known:

create - to create new data
read - to read data
delete - to delete data
list - to list data

Policies can be written either in JSON or HCL (HashiCorp Configuration Language). Let’s create a file with the following content:

path "expense/static/data/mysql" {
capabilities = ["read", "list"]
}

KV Version2 stores the secrets in a path with the prefix data/

This will limit my access to read and list only.

Write the policy:

cat my-policy.hcl | oc -n vault exec -it vault-0 -- vault policy write expense-db-mysql -

Next, we are going to bind the Vault secret to a service account and a namespace. Both objects will be created later, when we deploy the application.

oc -n vault exec -it vault-0 -- vault write auth/kubernetes/role/expense-db-mysql \ (1)
bound_service_account_names=expense-db-mysql \ (2)
bound_service_account_namespaces=expenses \ (3)
policies=expense-db-mysql \ (4)
ttl=1h (5)

1	Path or our new role
2	Name of the service account we will create and that will be used by the application
3	Name of the namespace we will create
4	Name of the policy we created earlier
5	The token is valid for 1 hour, after this period the service account must re-authenticate

Let’s start an Application

Now we will create our MySQL application into the namespace expenses. Use the following command to create the namespace and the application containing the objects Deployment, ServiceAccount (expense-db-mysql) and Service.

See at the Github Page for a full yaml specification of the three objects.

oc new-project expenses
oc apply -f https://raw.githubusercontent.com/joatmon08/vault-argocd/part-1/database/deployment.yaml

The deployment will start a Pod with a sidecar container vault-agent. This sidecar is automatically created and must not be defined inside the Deployment specification. Instead, some annotations in the Deployment define what the container should be automatically injected and also where to find our secret:

 annotations:
vault.hashicorp.com/agent-inject: "true" (1)
vault.hashicorp.com/role: "expense-db-mysql" (2)
vault.hashicorp.com/agent-inject-secret-db: "expense/static/data/mysql" (3)
vault.hashicorp.com/agent-inject-template-db: | (4)
{{ with secret "expense/static/data/mysql" -}}
export MYSQL_ROOT_PASSWORD="{{ .Data.data.db_login_password }}"
{{- end }}
...
spec:
serviceAccountName: expense-db-mysql (5)

1	Defines that the vault-agent side car container shall be automatically injected. This is the most important annotation.
2	Name of the role that was created created previously
3	The agent will inject the data from expense/static/data/mysql and stores it in a file db The file name is everything that comes after vault.hashicorp.com/agent-inject-secret-
4	Configuration… the template that defines how the secret will be rendered
5	The service account name we bound our secret to, using the Consul language. In this case the MySQL password is simply exported

The vault-agent is requesting the database password from Vault and provides it to the application where it is stored at /vault/secrets/db

oc -n expenses exec -it $(oc get pods -l=app=expense-db-mysql -o jsonpath='{.items[0].metadata.name}') -c expense-db-mysql -- cat /vault/secrets/db
# output
export MYSQL_ROOT_PASSWORD="mysuperpassword$"

The Deployment sources this file when it starts and MySQL will take this information to configure itself.

TIP: Using vault CLI on your local environment

All above commands that are dealing with Vault commands, first login to a pod and then execure the commands from there.

If you have the Vault CLI installed on your local machine, you can open a port forwarding to your Vault cluster at OpenShift and execute the commands locally:

oc port-forward -n vault svc/vault 8200
export VAULT_ADDR=http://localhost:8200
vault login
...

Thanks

Thanks to the wonderful Rosemary Wang and her Github repository: https://github.com/joatmon08/vault-argocd/tree/part-1

Also check out the Youtube Video: GitOps Guide to the Galaxy (Ep 31) | GitOps With Vault Part 1 in which Rosemary and Christian discuss this setup

Automated ETCD Backup

Sat, 29 Jan 2022 00:00:00 +0000

Securing ETCD is one of the major Day-2 tasks for a Kubernetes cluster. This article will explain how to create a backup using OpenShift Cronjob.

There is absolutely no warranty. Verify your backups regularly and perform restore tests.

Prerequisites

The following is required:

OpenShift Cluster 4.x
Integrated Storage, might be NFS or anything. Best practice would be a RWX enabled storage.

Configure Project & Cronjob

Create the following objects in OpenShift. This fill create:

A Project called ocp-etcd-backup
A PersistentVolumeClaim to store the backups. Change to your appropriate StorageClass and accessMode
A ServiceAccount called openshift-backup
A dedicated ClusterRole which is able to start (debug pods)
A ClusterRoleBinding between the created ServiceAccount and the customer ClusterRole
A 2nd ClusterRoleBinding, which gives our ServiceAccount the permission to start privileged containers. This is required to start a debug pod on a control plane node.
A CronJob which performs the backup … see Callouts for inline explanations.

A helm chart, which would create these objects below, can be found at: https://github.com/tjungbauer/ocp-auto-backup. This is probably a better way to manage the variables via the values.yaml file.

kind: Namespace
apiVersion: v1
metadata:
name: ocp-etcd-backup
annotations:
openshift.io/description: Openshift Backup Automation Tool
openshift.io/display-name: Backup ETCD Automation
openshift.io/node-selector: ''
spec: {}
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: etcd-backup-pvc
namespace: ocp-etcd-backup
spec:
accessModes:
- ReadWriteOnce (1)
resources:
requests:
storage: 100Gi
storageClassName: gp2
volumeMode: Filesystem
---
kind: ServiceAccount
apiVersion: v1
metadata:
name: openshift-backup
namespace: ocp-etcd-backup
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cluster-etcd-backup
rules:
- apiGroups: [""]
resources:
- "nodes"
verbs: ["get", "list"]
- apiGroups: [""]
resources:
- "pods"
- "pods/log"
verbs: ["get", "list", "create", "delete", "watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: openshift-backup
subjects:
- kind: ServiceAccount
name: openshift-backup
namespace: ocp-etcd-backup
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-etcd-backup
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: etcd-backup-scc-privileged
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:openshift:scc:privileged
subjects:
- kind: ServiceAccount
name: openshift-backup
namespace: ocp-etcd-backup
---
kind: CronJob
apiVersion: batch/v1
metadata:
name: cronjob-etcd-backup
namespace: ocp-etcd-backup
labels:
purpose: etcd-backup
spec:
schedule: '*/5 * * * *' (2)
startingDeadlineSeconds: 200
concurrencyPolicy: Forbid
suspend: false
jobTemplate:
metadata:
creationTimestamp: null
spec:
backoffLimit: 0
template:
metadata:
creationTimestamp: null
spec:
nodeSelector:
node-role.kubernetes.io/master: '' (3)
restartPolicy: Never
activeDeadlineSeconds: 200
serviceAccountName: openshift-backup
schedulerName: default-scheduler
hostNetwork: true
terminationGracePeriodSeconds: 30
securityContext: {}
containers:
- resources:
requests:
cpu: 300m
memory: 250Mi
terminationMessagePath: /dev/termination-log
name: etcd-backup
command: (4)
- /bin/bash
- '-c'
- >-
oc get no -l node-role.kubernetes.io/master --no-headers -o
name | grep `hostname` | head -n 1 | xargs -I {} -- oc debug
{} -- bash -c 'chroot /host sudo -E
/usr/local/bin/cluster-backup.sh /home/core/backup' ; echo
'Moving Local Master Backups to target directory (from
/home/core/backup to mounted PVC)'; mv /home/core/backup/*
/etcd-backup/; echo 'Deleting files older than 30 days' ; find
/etcd-backup/ -type f -mtime +30 -exec rm {} \;
securityContext:
privileged: true
runAsUser: 0
imagePullPolicy: IfNotPresent
volumeMounts:
- name: temp-backup
mountPath: /home/core/backup (5)
- name: etcd-backup
mountPath: /etcd-backup (6)
terminationMessagePolicy: FallbackToLogsOnError
image: registry.redhat.io/openshift4/ose-cli
serviceAccount: openshift-backup
volumes:
- name: temp-backup
hostPath:
path: /home/core/backup
type: ''
- name: etcd-backup
persistentVolumeClaim:
claimName: etcd-backup-pvc
dnsPolicy: ClusterFirst
tolerations:
- operator: Exists
effect: NoSchedule
- operator: Exists
effect: NoExecute
successfulJobsHistoryLimit: 5
failedJobsHistoryLimit: 5

1	RWO is used here, since I have no other available storage on my test cluster.
2	How often shall the job be executed. Here, every 5 minutes.
3	Bind the job to "Master" nodes.
4	Command to be executed… It fetches the actual local master nodename and starts a debugging Pod there. The backup script is called and moves the backup to /home/core/backup which is a folder on the control plane itself. The move command will move the backups from the local folder to the actual backup target volume. Finally, it will remove backups older than 30 days.
5	Mounted /home/core/backup on the master nodes, here the command will store the backups before they are moved
6	Target destination for the etcd backup on the mounted PVC

Start a Job

If you do not want to wait until the CronJob is triggered, you can manually start the Job using the following commands:

oc create job backup --from=cronjob/cronjob-etcd-backup -n ocp-etcd-backup

This will start a Pod which will do the backup:

Starting pod/ip-10-0-196-187us-east-2computeinternal-debug ...
To use host binaries, run `chroot /host`
found latest kube-apiserver: /etc/kubernetes/static-pod-resources/kube-apiserver-pod-15
found latest kube-controller-manager: /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-10
found latest kube-scheduler: /etc/kubernetes/static-pod-resources/kube-scheduler-pod-9
found latest etcd: /etc/kubernetes/static-pod-resources/etcd-pod-3
etcdctl is already installed
{"level":"info","ts":1638199790.980932,"caller":"snapshot/v3_snapshot.go:119","msg":"created temporary db file","path":"/home/core/backup/snapshot_2021-11-29_152949.db.part"}
{"level":"info","ts":"2021-11-29T15:29:50.991Z","caller":"clientv3/maintenance.go:200","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":1638199790.9912837,"caller":"snapshot/v3_snapshot.go:127","msg":"fetching snapshot","endpoint":"https://10.0.196.187:2379"}
{"level":"info","ts":"2021-11-29T15:29:53.306Z","caller":"clientv3/maintenance.go:208","msg":"completed snapshot read; closing"}
Snapshot saved at /home/core/backup/snapshot_2021-11-29_152949.db
{"level":"info","ts":1638199793.3482974,"caller":"snapshot/v3_snapshot.go:142","msg":"fetched snapshot","endpoint":"https://10.0.196.187:2379","size":"180 MB","took":2.367303503}
{"level":"info","ts":1638199793.348459,"caller":"snapshot/v3_snapshot.go:152","msg":"saved","path":"/home/core/backup/snapshot_2021-11-29_152949.db"}
{"hash":1180914745,"revision":10182252,"totalKey":19360,"totalSize":179896320}
snapshot db and kube resources are successfully saved to /home/core/backup
Removing debug pod ...
Moving Local Master Backups to target directory (from /home/core/backup to mounted PVC)

Verifying the Backup

Let’s start a dummy Pod which can access the PVC to verify if the backup is really there.

apiVersion: v1
kind: Pod
metadata:
name: verify-etcd-backup
spec:
containers:
- name: verify-etcd-backup
image: registry.access.redhat.com/ubi8/ubi
command: ["sleep", "3000"]
volumeMounts:
- name: etcd-backup
mountPath: /etcd-backup
volumes:
- name: etcd-backup
persistentVolumeClaim:
claimName: etcd-backup-pvc

Logging into that Pod will show the available backups stored at /etcd-backup which is the mounted PVC.

oc rsh -n ocp-etcd-backup verify-etcd-backup ls -la etcd-backup
total 1406196
drwxr-xr-x. 3 root root 4096 Nov 29 17:00 .
dr-xr-xr-x. 1 root root 25 Nov 29 17:06 ..
drwx------. 2 root root 16384 Nov 29 15:21 lost+found
-rw-------. 1 root root 179896352 Nov 29 15:21 snapshot_2021-11-29_152150.db
-rw-------. 1 root root 179896352 Nov 29 15:29 snapshot_2021-11-29_152949.db
-rw-------. 1 root root 179896352 Nov 29 15:32 snapshot_2021-11-29_153159.db
-rw-------. 1 root root 179896352 Nov 29 15:36 snapshot_2021-11-29_153618.db
-rw-------. 1 root root 179896352 Nov 29 15:55 snapshot_2021-11-29_155513.db
-rw-------. 1 root root 179896352 Nov 29 16:00 snapshot_2021-11-29_160020.db
-rw-------. 1 root root 179896352 Nov 29 16:55 snapshot_2021-11-29_165521.db
-rw-------. 1 root root 179896352 Nov 29 17:00 snapshot_2021-11-29_170020.db
-rw-------. 1 root root 89875 Nov 29 15:21 static_kuberesources_2021-11-29_152150.tar.gz
-rw-------. 1 root root 89875 Nov 29 15:29 static_kuberesources_2021-11-29_152949.tar.gz
-rw-------. 1 root root 89875 Nov 29 15:32 static_kuberesources_2021-11-29_153159.tar.gz
-rw-------. 1 root root 89875 Nov 29 15:36 static_kuberesources_2021-11-29_153618.tar.gz
-rw-------. 1 root root 89875 Nov 29 15:55 static_kuberesources_2021-11-29_155513.tar.gz
-rw-------. 1 root root 89875 Nov 29 16:00 static_kuberesources_2021-11-29_160020.tar.gz
-rw-------. 1 root root 89875 Nov 29 16:55 static_kuberesources_2021-11-29_165521.tar.gz
-rw-------. 1 root root 89875 Nov 29 17:00 static_kuberesources_2021-11-29_170020.tar.gz

1	Allowed domains
2	Allow subdomains
3	Key size
4	Key type, typically rsa or ec (elliptic curve)

1	The namespace where the OpenBao deployment is running.
2	The self-signed CA Issuer.

Security on TechBlog about OpenShift/Ansible/Satellite and much more

Creating an auditd rule to monitor Ansible Automation Platform static secrets

Ansible Automation Platform local SECRET_KEY

Configuring Linux auditd

Summary

The Guide to OpenBao - Secrets Engines KV - Part 8

Introduction

List of Secrets Engines

KV (Key-Value) Secrets Engine

KV Version 1

KV Version 2

KV Version 1 vs Version 2

Enabling KV Engine

Storing Secrets

Reading Secrets

Managing Versions

Custom Metadata

Policies for KV v2

Conclusion

Resources

The Guide to OpenBao - Secrets Engines PKI - Part 9

Quick Navigation

Introduction

PKI Secrets Engine

Setting Up PKI Secrets Engine (Root CA)

Setting Up an Intermediate CA

PKI Policies

Integration with cert-manager

Enable the Kubernetes auth method

Configure the Kubernetes auth method

In-cluster example: reviewer service account and Secret

Configure auth/kubernetes/config

Policy and role for cert-manager

Verify with a Kubernetes login (optional)

Configure the (Cluster)Issuer for cert-manager

Issue a Certificate

Conclusion

Resources

The Guide to OpenBao - Authentication Methods - Part 7

Introduction

Authentication Workflow

Prerequisites for Authentication Methods

Using the CLI

Using the UI

Understanding Policies

Basic Policy Structure

Policy Capabilities

Default Policies

Creating Policies using the CLI

Kubernetes Authentication Method

Step 1: Enable Kubernetes Auth

Step 2: Configure Kubernetes Auth

Step 3: Create a Policy for the Application

Step 4: Create a Kubernetes Auth Role

Ensure the KV v2 secrets engine is mounted at secret/

Step 5: Store the database secret in OpenBao (one-time)

Step 6: Create the expense namespace and demo application

Step 7: Test Kubernetes authentication manually

Summary: Kubernetes auth method

LDAP Authentication Method

Step 1: Enable LDAP Auth

Step 2: Configure LDAP (Active Directory Example)

Step 3: Create the policies used by the group mapping

Step 4: Map LDAP groups to policies

Step 5: Test LDAP authentication

What jdoe can and cannot do (user-policy)

Common Practices for Authentication

Common Authentication Patterns

What is Coming Next?

Conclusion

Resources

The Guide to OpenBao - Initialisation, Unsealing, and Auto-Unseal - Part 6

What is happening?

Handling Initialisation and Unsealing

Approach 1: Manual Initialisation (simplest)

Approach 2: Kubernetes Job for initialisation

How the Secret is created

RBAC Configuration

Init Job that creates the Secret

Using the Secret in a Job (unseal)

Ensure the KV v2 secrets engine is mounted at `secret/`