Security on TechBlog about OpenShift/Ansible/Satellite and much more

The Guide to OpenBao - Authentication Methods - Part 7

Mon, 09 Mar 2026 00:00:00 +0000

With OpenBao deployed and running, the next critical step is configuring authentication. Ultimately, you want to limit access to only authorised people. This article covers two common authentication methods: Kubernetes for pods and LDAP for enterprise directories (in a simplified example). There are many more methods, but we cannot cover them all in this article.

Introduction

Authentication in OpenBao verifies the identity of clients before granting access to secrets. OpenBao supports multiple authentication methods, each suited for different use cases. Among others it supports:

Method	Best For	Key Features
Kubernetes	Pods running in K8s/OpenShift	Service account tokens, automatic
OIDC	Human users, SSO	OpenShift OAuth, Keycloak, Azure AD
LDAP	Enterprise directories	Active Directory, OpenLDAP
AppRole	CI/CD, automation	Role ID + Secret ID
Token	Direct access, bootstrap	Simple but less secure

Method

Best For

Key Features

Kubernetes

Pods running in K8s/OpenShift

Service account tokens, automatic

OIDC

Human users, SSO

OpenShift OAuth, Keycloak, Azure AD

LDAP

Enterprise directories

Active Directory, OpenLDAP

AppRole

CI/CD, automation

Role ID + Secret ID

Token

Direct access, bootstrap

Simple but less secure

The full documentation of the authentication methods and the list of available methods (for example Radius) can be found here: OpenBao Authentication Methods.

Authentication Workflow

The authentication workflow in OpenBao from a user perspective is as follows:

A client wants to authenticate to OpenBao and provides credentials.
OpenBao validates the credentials against the authentication method. For example: LDAP
If the authentication method (e.g. LDAP) is successful, it will return the required information about the client.
OpenBao maps this result to policies that are mapped to the authentication method.
OpenBao generates a token that is associated with the policies and returns it to the client.
The client can then use this token for further operations.

Prerequisites for Authentication Methods

Before configuring authentication:

OpenBao is deployed and unsealed (Parts 2-6)
You have admin access to OpenBao (root token)
Access to the required authentication backend:
- For Kubernetes auth: Access to the OpenShift cluster
- For LDAP: Access to the LDAP server
- etc.

Set up your environment:

You can set up your environment by using either the CLI or the UI.

Using the CLI

You can use the CLI to interact with OpenBao. In this example we will forward the port locally to the OpenBao server and set the environment variable BAO_ADDR to the local address. With boa login and the root token you can authenticate against OpenBao with full access.

# Port forward (if needed)
oc port-forward svc/openbao 8200:8200 -n openbao &
# Set environment
export BAO_ADDR='http://127.0.0.1:8200'
# You may need the SSL CA too.
# export BAO_CACERT="$PWD/openbao-ca.crt"
# Login with root token
bao login
# List all authentication methods
bao auth list

The last command returns a list of the available authentication methods. Currently there is only one authentication method enabled: token.

Path Type Accessor Description Version
---- ---- -------- ----------- -------
token/ token auth_token_1020fa7b token based credentials n/a

Using the UI

The UI, if enabled, is accessible via the OpenShift Route (if running on OpenShift and if it was created in Part 3). The password/token is the root token from the initialisation.

Figure 1. OpenBao UI Login Form

The UI has the advantage that it offers more visibility into the configuration and possibilities. For example, at the beginning there is one authentication method token enabled. This is required for the root token authentication.

Figure 2. OpenBao UI Authentication Methods

We can enable the other authentication methods by clicking on the Enable new method link to see what options are available.

Figure 3. OpenBao UI Configuring New Authentication Method

Understanding Policies

Before you can actually use an authentication method, it is important to understand what policies do. A policy is a way to declaratively define what (which paths) authenticated users can access or not. All paths are denied by default. A policy is mapped to an authentication method. For example, when we look at LDAP, we could configure something like this:

Every member of group "dev" is mapped to a policy named "dev-policy". The policy then allows the user to read the secrets under the path "secret/data/dev/*".

Basic Policy Structure

A policy is a set of path rules. Each rule says: “for this path (or path prefix), the bearer of this policy may use these capabilities.” Paths are tied to OpenBao’s internal API: for example, secrets in the KV v2 engine live under secret/data/<mount-path>/<key>, so a path like secret/data/myapp/* means “any key under the myapp prefix in the KV v2 store mounted at secret/.”

The example below does two things that are typical for an application policy:

Secrets access: It allows read and list on secret/data/myapp/*.
- read lets the client fetch the value of a secret at a path (e.g. secret/data/myapp/database).
- list is required to list keys under a path (e.g. to discover myapp/database, myapp/api-key); without it, the client would need to know every path in advance.
Token renewal: It allows update on auth/token/renew-self. Tokens often have a limited lifetime; this path lets the holder extend their own token without needing the root token or extra permissions. Including it avoids applications losing access when the token expires.

Policies can be written in JSON or HCL.

# Example policy: myapp-read-policy.hcl
# Allow reading secrets from specific path (KV v2 engine at mount "secret/")
path "secret/data/myapp/*" {
capabilities = ["read", "list"]
}
# Allow renewing own token so the app can extend its token before expiry
path "auth/token/renew-self" {
capabilities = ["update"]
}

The path secret/data/myapp/* assumes you have a KV v2 secrets engine mounted at secret/ and will store application secrets under keys like myapp/database, myapp/api-key, etc. Adjust the mount path and prefix to match your setup.

The use of globs (*) may result in surprising or unexpected behaviour. Use them with caution.

Policy Capabilities

Each path defined in a policy must have at least one capability. This provides control over the allowed or denied operations a user may perform. There are several capabilities available:

create - Create new data at the given path
read - Read data
update - Update existing data
patch - Patch existing data (Partial update)
delete - Delete data
list - List keys at a path
scan - Scan or browse the path for keys
sudo - Allows access to paths that are root-protected.
deny - Explicitly deny (overrides other grants)

Default Policies

When OpenBao is deployed the first time, there are no additional authentication methods enabled, except for token. This is required for the root access itself. To allow the administrator to access using the root token, there is one default policy called root. This policy is somewhat of a catch-all policy; it allows the administrator to access everything.

In addition to the root policy, there is one other default policy called default. This policy is used for all authenticated users. It allows them, for example, to look up their own properties and to renew or revoke their own token.

Let’s look at the default policies in the CLI:

A quick reminder of how to log in to OpenBao using the CLI when running it against Kubernetes: first open a port-forward, set the variable BAO_ADDR to the local address; if you use HTTPS you may need to set the SSL CA certificate, then log in with the root token.

You need to have the CA certificate in the current directory, which you might fetch with:

oc get secret openbao-ca-secret -n openbao -o jsonpath='{.data.ca\.crt}' | base64 -d > openbao-ca.crt

Port forward, set the environment variables and login with the root token:

# Port forward (if needed)
oc port-forward svc/openbao 8200:8200 -n openbao &
# Set environment
export BAO_ADDR='https://127.0.0.1:8200'
# You may need the SSL CA too.
export BAO_CACERT="$PWD/openbao-ca.crt"
# Login with root token
bao login

Now to list and fetch the current policies you can use the following commands:

# List policies
bao policy list
# Returns something like this:
#default
#root
# Read a policy
bao policy read default

The second command returns the policy content:

# Allow tokens to look up their own properties
path "auth/token/lookup-self" {
capabilities = ["read"]
}
# Allow tokens to renew themselves
path "auth/token/renew-self" {
capabilities = ["update"]
}
# Allow tokens to revoke themselves
path "auth/token/revoke-self" {
capabilities = ["update"]
}
# Allow a token to look up its own capabilities on a path
path "sys/capabilities-self" {
[...]

Again you will see the same policies in the UI. You can also manage them or create new ones using the UI.

Figure 4. OpenBao UI Listing Policies

Creating Policies using the CLI

Creating a policy, for example using the CLI, is straightforward. You can create a policy from a file or inline. The syntax is either JSON or HCL. The example below creates a policy called "myapp-read" from a file called "myapp-read-policy.hcl". The policy allows reading and listing secrets under the path "secret/data/myapp/*".

# Create a policy from file
bao policy write myapp-read myapp-read-policy.hcl
# Or inline
bao policy write myapp-read - <<EOF
path "secret/data/myapp/*" {
capabilities = ["read", "list"]
}
EOF
# List policies
bao policy list
# Read a policy
bao policy read myapp-read

Kubernetes Authentication Method

As we work a lot with OpenShift/Kubernetes, this method of authentication will be the first to try. The Kubernetes auth method can be used to authenticate against OpenBao using Kubernetes Service Account tokens. It is essential for pods to access secrets.

Let us try a simple example.

Step 1: Enable Kubernetes Auth

Assuming you are still logged into OpenBao, you can enable the kubernetes auth method with the following command:

bao auth enable --description="Authentication method for Kubernetes/OpenShift cluster" kubernetes

This simply enabled the authentication method and set a description. However, we need to configure it further.

Step 2: Configure Kubernetes Auth

To configure the kubernetes auth method, you need to get the Kubernetes API server address. You can get it with the following command (assuming you are logged into the OpenShift cluster):

The command oc can be replaced by kubectl if you are not using an OpenShift cluster.

# Get the Kubernetes API server address
KUBERNETES_HOST=$(oc whoami --show-server) (1)
# Configure the auth method
bao write auth/kubernetes/config \ (2)
kubernetes_host="$KUBERNETES_HOST"

1	Get the Kubernetes API server address. This will use something like https://api.cluster.example.com:6443
2	Configure the auth method with the Kubernetes API server address.

When running inside Kubernetes, OpenBao automatically uses the pod’s service account to communicate with the Kubernetes API.

Step 3: Create a Policy for the Application

Now we need to associate a policy to the authentication method. We could also assign the default policy but we want to provide access to the path secret/data/expense/database and secret/metadata/expense/config for the application.

# Create policy for an example application
bao policy write expense-app - <<EOF
# Read database credentials
path "secret/data/expense/database" {
capabilities = ["read"]
}
# Read application config
path "secret/data/expense/config" {
capabilities = ["read", "list"]
}
# Allow token renewal
path "auth/token/renew-self" {
capabilities = ["update"]
}
EOF

Step 4: Create a Kubernetes Auth Role

As the next step we map the policy to the authentication method. We will further limit the access to the namespace expense and the service account expense-app.

# Create a role that maps Kubernetes service accounts to OpenBao policies
bao write auth/kubernetes/role/expense-app \
bound_service_account_names=expense-app \ (1)
bound_service_account_namespaces=expense \ (2)
policies=expense-app \ (3)
ttl=1h \ (4)
max_ttl=24h (5)

1	`bound_service_account_names`: K8s service account name(s)
2	`bound_service_account_namespaces`: K8s namespace(s)
3	`policies`: OpenBao policies to attach
4	`ttl`: Token time-to-live
5	`max_ttl`: Maximum token lifetime

All of the configuration above, created using the CLI, can also be done using the UI of OpenBao.

Ensure the KV v2 secrets engine is mounted at `secret/`

We will discuss secret engines in more detail in the next part of the guide.

The policy and demo commands in this guide use the path secret/data/expense/database, which assumes a KV v2 secrets engine mounted at secret/. To check and enable it:

Check existing mounts: List enabled secrets engines and look for secret/ with type kv (version 2):
```
# List all secrets engine mounts (requires a token with read on sys/mounts)
bao secrets list
```
Look for an entry like secret/ with type kv or kv-v2. If you see secret/ and it is KV v2, you can skip to Step 5.
Enable KV v2 at secret/ if missing: If secret/ is not listed, or you want to use a fresh mount, enable the KV v2 engine at the path secret:
```
# Enable KV v2 secrets engine at path "secret" (requires root or policy with sys/mounts capability)
bao secrets enable -path=secret -version=2 kv
```
If secret/ already exists as KV v1, you cannot change it in place to v2; use a different path (e.g. secretv2/) and update the policy and demo paths accordingly (secretv2/data/expense/database, etc.).

Step 5: Store the database secret in OpenBao (one-time)

Before the application can read it, store the database password (and any other keys) at the path the policy allows. Run this once from a machine that has OpenBao access (e.g. bao CLI and a token).

You must use a token that has create and update capability on the KV path. The expense-app policy is read-only (for the application). Use the root token from OpenBao initialisation, or log in with bao login and use a token that has a policy granting create and update on secret/data/expense/*. If you use a read-only token (e.g. one that only has the expense-app policy), you will get a 403 "preflight capability check returned 403".

# Store the database secret at secret/data/expense/database (KV v2 path)
bao kv put secret/expense/database password="my-super-secret-db-password" username="expense_db_user"

This creates (or overwrites) the secret at secret/data/expense/database. The policy grants the expense-app role read on this path only; the application cannot create or update it.

Step 6: Create the expense namespace and demo application

Apply the following manifests to create the namespace, service account, optional long-lived token (Kubernetes 1.24+), and a demo deployment that authenticates to OpenBao and fetches the database secret. Adjust the OpenBao URL if your OpenBao is in another namespace or uses HTTPS (e.g. https://openbao.openbao.svc:8200).

Remember: We have created a policy that allows the service account with the name expense-app in the namespace expense to read the secret at secret/data/expense/database. If you want to use different paths or namespaces, you need to adjust the policy accordingly.

# full-demo-expense-app.yaml
# 1. Namespace
apiVersion: v1
kind: Namespace (1)
metadata:
name: expense
---
# 2. Service account used by the demo app to authenticate to OpenBao
apiVersion: v1
kind: ServiceAccount (2)
metadata:
name: expense-app
namespace: expense
---
# 3. Long-lived token for Kubernetes 1.24+ (optional; allows pods to use the SA token)
apiVersion: v1
kind: Secret (3)
metadata:
name: expense-app-token
namespace: expense
annotations:
kubernetes.io/service-account.name: expense-app
type: kubernetes.io/service-account-token
---
# 4. Demo deployment: authenticates to OpenBao and fetches secret/data/expense/database
apiVersion: apps/v1
kind: Deployment
metadata:
name: expense-demo
namespace: expense
labels:
app: expense-demo
spec:
replicas: 1
selector:
matchLabels:
app: expense-demo
template:
metadata:
labels:
app: expense-demo
spec:
serviceAccountName: expense-app
containers:
- name: demo
image: registry.access.redhat.com/ubi9/ubi-minimal:latest
command:
- /bin/sh
- -c
- |
echo "=== Installing curl ==="
microdnf install -y curl 2>/dev/null || true
echo "=== Fetching database secret from OpenBao ==="
JWT=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
OPENBAO_URL="${OPENBAO_URL:-http://openbao.openbao:8200}"
RESP=$(curl -s -k -X POST -H "Content-Type: application/json" \ (4)
-d "{\"jwt\": \"$JWT\", \"role\": \"expense-app\"}" \
"$OPENBAO_URL/v1/auth/kubernetes/login")
TOKEN=$(echo "$RESP" | sed -n 's/.*"client_token":"\([^"]*\)".*/\1/p')
if [ -z "$TOKEN" ]; then
echo "ERROR: Failed to get OpenBao token. Check OpenBao URL ($OPENBAO_URL), role expense-app, and network."
echo "Response: $RESP"
exit 1
fi
echo "OpenBao token obtained. Reading secret/data/expense/database ..."
SECRET=$(curl -s -k -H "X-Vault-Token: $TOKEN" "$OPENBAO_URL/v1/secret/data/expense/database") (5)
echo "$SECRET"
PASSWORD=$(echo "$SECRET" | sed -n 's/.*"password":"\([^"]*\)".*/\1/p')
echo "Database password (data.data.password): ${PASSWORD:-<not set>}"
echo "=== Demo complete. Pod stays running; exec in to run more curl commands. ==="
exec sleep infinity
env:
# Override if OpenBao is in another namespace or uses HTTPS
- name: OPENBAO_URL
value: "https://openbao.openbao:8200"
resources:
requests:
memory: "64Mi"
cpu: "10m"

1	Create the namespace
2	Create the service account
3	Create the secret (long-lived token for Kubernetes 1.24+)
4	Authenticate to OpenBao and get a client token (using the Kubernetes JWT)
5	Fetch the database secret

The examples above use curl -k to bypass certificate verification. For production, you should adjust this and mount the CA certificate in the pod (see the note earlier in this section on using the openbao-ca Secret).

Apply the manifests and wait for the pod to be ready:

oc apply -f full-demo-expense-app.yaml
# Wait for the demo pod to be running
oc -n expense rollout status deployment/expense-demo
# View the log: you should see the OpenBao token response and the fetched secret (including the password)
oc -n expense logs -l app=expense-demo -f

The log output shows the application authenticating with the Kubernetes JWT, receiving an OpenBao token, and reading the secret at secret/data/expense/database. The field data.data.password is the database password you stored in Step 5.

Step 7: Test Kubernetes authentication manually

From any pod that uses the same service account (e.g. the demo pod above), you can run the same steps by hand:

# From within a pod using the service account (e.g. oc exec -it deployment/expense-demo -n expense -- /bin/sh)
JWT=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
# 1. Authenticate to OpenBao and get a client token (sed works on UBI minimal; use jq -r .auth.client_token if jq is available)
LOGIN=$(curl -s -k --request POST \
--data "{\"jwt\": \"$JWT\", \"role\": \"expense-app\"}" \
https://openbao.openbao:8200/v1/auth/kubernetes/login)
TOKEN=$(echo "$LOGIN" | sed -n 's/.*"client_token":"\([^"]*\)".*/\1/p')
# 2. Fetch the database secret (same path the policy allows)
curl -s -k -H "X-Vault-Token: $TOKEN" https://openbao.openbao:8200/v1/secret/data/expense/database

Summary: Kubernetes auth method

A lot of steps were involved to configure the Kubernetes auth method. Let us summarize them:

In OpenBao: Enabled the Kubernetes auth method, configured it with the cluster API server address, created the expense-app policy (read on secret/data/expense/database and secret/data/expense/config, update on auth/token/renew-self), and created the expense-app role that binds the Kubernetes service account expense-app in namespace expense to that policy.
Secrets engine: Ensured the KV v2 engine is mounted at secret/ and stored the database secret at secret/expense/database (one-time, using the root token).
In the cluster: Created the expense namespace, the expense-app service account, an optional long-lived token Secret (for Kubernetes 1.24+), and the expense-demo deployment that uses that service account to authenticate to OpenBao and fetch the secret. For HTTPS, created the openbao-ca Secret in expense with the OpenBao server CA.
Verification: Applied the manifests, confirmed the demo pod starts and logs show a successful login and the fetched secret (including the database password). Optionally ran the same login and read steps manually via oc exec into the pod.

LDAP Authentication Method

LDAP auth integrates with enterprise directories like Active Directory.

This is a very simplified example. As a prerequisite you need to have an LDAP server and a service account with the necessary permissions to read the users and groups. This is not part of this guide.

Step 1: Enable LDAP Auth

To enable LDAP authentication, you need to use the following command:

bao auth enable ldap

Step 2: Configure LDAP (Active Directory Example)

In this step you need to configure the LDAP server address, the user and group attributes, the bind DN and the bind password. The example below configures the LDAP server address to ldaps://ad.example.com:636, the user attribute to sAMAccountName, the user DN to OU=Users,DC=example,DC=com, the group DN to OU=Groups,DC=example,DC=com, the group attribute to cn, the group filter to (&(objectClass=group)(member:1.2.840.113556.1.4.1941:={{.UserDN}})), the bind DN to CN=openbao-svc,OU=ServiceAccounts,DC=example,DC=com and the bind password to service-account-password.

bao write auth/ldap/config \
url="ldaps://ad.example.com:636" \
userattr="sAMAccountName" \
userdn="OU=Users,DC=example,DC=com" \
groupdn="OU=Groups,DC=example,DC=com" \
groupattr="cn" \
groupfilter="(&(objectClass=group)(member:1.2.840.113556.1.4.1941:={{.UserDN}}))" \
binddn="CN=openbao-svc,OU=ServiceAccounts,DC=example,DC=com" \
bindpass="service-account-password" \
certificate=@/path/to/ldap-ca.pem \
insecure_tls=false \
starttls=false

Step 3: Create the policies used by the group mapping

Before mapping LDAP groups to policy names, create those policies. Below are example definitions: admin-policy grants read/list/update on secrets (e.g. for operators), and user-policy grants read-only access to a limited path.

admin-policy — for members of openbao-admins (e.g. operators who may read and update secrets)

user-policy — for members of openbao-users (e.g. developers with read-only access to a subset of secrets)

Create them with the CLI (run before the group mapping in Step 4):

# Create admin-policy (e.g. from file admin-policy.hcl)
bao policy write admin-policy - <<EOF
path "secret/data/*" {
capabilities = ["read", "list", "update", "create", "delete"]
}
path "secret/metadata/*" {
capabilities = ["read", "list"]
}
path "auth/token/renew-self" {
capabilities = ["update"]
}
EOF
# Create user-policy
bao policy write user-policy - <<EOF
path "secret/data/myapp/*" {
capabilities = ["read", "list"]
}
path "auth/token/renew-self" {
capabilities = ["update"]
}
EOF

Adjust paths (e.g. secret/data/myapp/) to match your KV v2 mount and the paths you want each role to access. The default policy is built-in and grants token lookup/renew/revoke-self. Attaching it in addition to admin-policy or user-policy is a typical pattern.

Step 4: Map LDAP groups to policies

Now map the LDAP groups to the policy names. The example below maps the LDAP group openbao-admins to the policies admin-policy and default and the LDAP group openbao-users to the policies user-policy and default.

# Map an LDAP group to policies
bao write auth/ldap/groups/openbao-admins \
policies="admin-policy,default"
bao write auth/ldap/groups/openbao-users \
policies="user-policy,default"

Step 5: Test LDAP authentication

Assuming you have a user jdoe in the LDAP group openbao-users, you can test the LDAP authentication with the following commands:

# Login with LDAP credentials
bao login -method=ldap username=jdoe
# Or via API
curl --request POST \
--data '{"password": "user-password"}' \
$BAO_ADDR/v1/auth/ldap/login/jdoe

What jdoe can and cannot do (user-policy)

jdoe holds user-policy (and default), so their token has only the capabilities defined there.

What jdoe can do:

Get a secret under secret/data/myapp/*: after logging in, jdoe can read and list secrets in that path. For example:

# After bao login -method=ldap username=jdoe (and entering the password)
bao kv get secret/myapp/database
# Or via API (use the client_token from the login response)
curl -s -H "X-Vault-Token: $TOKEN" $BAO_ADDR/v1/secret/data/myapp/database

List keys under secret/data/myapp/ to discover available secrets.
Renew or revoke their own token (from the default policy) and look up their own token properties.

What jdoe cannot do:

Read secrets outside myapp — e.g. secret/data/expense/database or secret/data/other-app/* returns permission denied.
Create, update, or delete any secret — user-policy grants only read and list on secret/data/myapp/*, so write operations are denied.
Access admin or system paths — no access to sys/, other auth methods, or policies. Only the paths explicitly granted in *user-policy and default are allowed.

Common Practices for Authentication

Use the principle of least privilege - Do not grant more permissions than necessary.

# Bad: Too broad
path "secret/*" {
capabilities = ["read", "list"]
}
# Good: Specific paths
path "secret/data/myapp/database" {
capabilities = ["read"]
}

Set Appropriate Token TTLs - Set the appropriate TTLs for different use cases.

# Short-lived for automated systems
bao write auth/kubernetes/role/batch-job \
ttl=5m \
max_ttl=15m
# Longer for interactive users
bao write auth/oidc/role/human-user \
ttl=8h \
max_ttl=24h

Enable Audit Logging - Enable audit logging to track authentication attempts and other activities.

# Enable file audit
bao audit enable file file_path=/var/log/openbao/audit.log
# All authentication attempts are logged

Regularly Rotate Credentials - Rotate the credentials periodically to reduce the risk of compromise.

# Rotate AppRole Secret IDs periodically
bao write -f auth/approle/role/cicd-role/secret-id
# Revoke old tokens
bao token revoke <old-token>

Common Authentication Patterns

Pattern 1: Application Pods

Pod → Service Account Token → Kubernetes Auth → Policy → Secret

Pattern 2: Human Users

User → LDAP/OIDC Login → Group Mapping → Policy → Secret

Pattern 3: CI/CD Pipeline

Pipeline → AppRole (Role ID + Secret ID) → Policy → Secret

What is Coming Next?

In Part 8, we will explore secrets engines, like:

KV (Key-Value) for static secrets

In upcoming parts, I will try to cover common practices for OpenBao operation and maintenance. If there is time and an opportunity to test other authentication methods, I will do so.

Conclusion

Proper authentication configuration is crucial for OpenBao security. Key takeaways:

Use Kubernetes auth for pods
Use LDAP/OIDC for human users (SSO)
Use AppRole for automation
Create specific policies with least privilege
Set appropriate TTLs for different use cases
Enable audit logging for compliance

Resources

The Guide to OpenBao - Initialisation, Unsealing, and Auto-Unseal - Part 6

Thu, 05 Mar 2026 00:00:00 +0000

After deploying OpenBao via GitOps (Part 5), OpenBao must be initialised and then unsealed before it becomes functional. You usually do not want to do this unsealing manually, since this is not scalable especially in bigger, productive environments. This article explains how to handle initialisation and unsealing, and possible options to configure an auto-unseal process so that OpenBao unseals itself on every restart without manual key entry.

What is happening?

OpenBao starts in a sealed state. You must:

Initialise (once): run bao operator init to generate unseal keys (or recovery keys when using auto-unseal).
Unseal (after each restart): provide a threshold of unseal keys so each node can decrypt the root key. In the previous articles, we used a threshold of 3 (out of 5). This means that you need to run bao operator unseal three times with different keys on every node after every start before the service becomes available.

Unseal workflow (assuming a threshold of 3 and 5 nodes and initialisation already happened):

---
title: "Unseal Workflow"
config:
theme: 'dark'
---
flowchart LR
A["openbao-0 starts → 3× unseal using different keys"]
A --> B["openbao-1 starts → 3× unseal using different keys"]
B --> C["openbao-2 starts → 3× unseal using different keys"]
C --> D[OpenBao is ready]

This is a chicken-egg problem: you need to start somewhere, but you cannot start without the keys.

Manual unsealing does not scale very well and for production, auto-unseal is recommended.

This article covers:

Practical approaches to initialisation and unsealing (manual, Jobs, init containers).
Potential auto-unseal options: with AWS KMS, static key, and transit seal.

Handling Initialisation and Unsealing

The challenge with GitOps is that initialisation and unsealing are one-off or stateful operations. Below are several approaches. The problem is always the same: where do you start? You must have the unseal keys available to start the process. Do you keep them locally, do you keep them in a Kubernetes Secret, do you use an external Key Management System (KMS)? The most robust for production in my opinion is auto-unseal using a KMS (see [_approach_4_auto_unseal_with_kms_recommended_for_production] and the following sections). It uses an external Key Management System (KMS) to store the keys. This way, the keys are not stored in the cluster and are not exposed to the risk of being compromised.

Let’s have a look at the different options.

Approach 1: Manual Initialisation (simplest)

While manual initialisation and unsealing is the simplest approach, it is not recommended for production. Still, I would like to mention it here for completeness and at least provide a simple for loop to unseal the OpenBao service (on Kubernetes/OpenShift).

Manual unsealing is still a valid option when you have a small cluster and assume that it is not required often.

After Argo CD deploys OpenBao, the first pod, openbao-0, will not become ready until it is initialised and unsealed. Once done, the next pod, openbao-1, will try to start and wait until it is unsealed, and then the third pod, and so on.

The following, simplified script will go through the pods and unseal them one by one. It initialises the first pod and stores the unseal keys in the file openbao-init.json locally on your machine. From there it takes three different keys and unseals the OpenBao service. Since it takes a while until other pods are pulling the image and starting, we will use a sleep timer of 30 seconds between each pod. This may or may not be enough depending on your network and cluster speed. As said, this is a very simplified script:

The CLI tool oc can be replaced by kubectl in case you are using a different cluster. In addition, you will need the jq command line tool to parse the JSON file.

If your OpenBao was initialised previously, you can remove the first three commands and start with the unseal commands.

# Initialise the first pod, assuming the Pod openbao-0 is already running.
# You can skip this if OpenBao was initialised previously.
echo "Initialising OpenBao on openbao-0 (keys will be saved to openbao-init.json)..."
oc exec -it openbao-0 -n openbao -- bao operator init \
-key-shares=5 -key-threshold=3 -format=json > openbao-init.json (1)
echo "Init complete. Unseal keys and root token are in openbao-init.json."
# Unseal each pod (three times each with different keys)
echo "Unsealing pods openbao-0, openbao-1, openbao-2 (3 key steps each)..."
for i in 0 1 2; do
sleep_timer=30
SLEEPER_TMP=1
echo "Unsealing openbao-$i..."
for j in 1 2 3; do (2)
KEY=$(cat openbao-init.json | jq -r ".unseal_keys_b64[$((j-1))]")
oc exec -it openbao-$i -n openbao -- bao operator unseal $KEY
done
echo "openbao-$i unsealed. Waiting ${sleep_timer}s for next pod to be ready..."
while [[ $SLEEPER_TMP -le "$sleep_timer" ]]; do (3)
if (( SLEEPER_TMP % 10 == 0 )); then
echo -n "$SLEEPER_TMP"
else
echo -n "."
fi
sleep 1
SLEEPER_TMP=$((SLEEPER_TMP + 1))
done
echo ""
done
echo "All pods unsealed. Store init data securely (e.g. in a password manager)."
# Store init data securely (e.g. in a password manager)

1	Initialise the first pod, assuming the Pod openbao-0 is already running. This will store the keys (including the root token) in the file openbao-init.json.
2	Unseal each pod (three times each with different keys). The keys are taken from the openbao-init.json file.
3	Wait 30 seconds between each pod to give the other pods time to start and pull the image … super-fancy sleep timer.

The unseal keys as well as the root token are stored in the openbao-init.json file. This file is stored locally on your machine. Make sure to store it securely and do not share it with anyone.

Approach 2: Kubernetes Job for initialisation

A Job can run after OpenBao is deployed, initialise it if needed, and store the init output (unseal keys and root token) in a Kubernetes Secret. You can then unseal manually or with a second Job that reads from that Secret. This section assumes the init output is stored in a Kubernetes Secret and explains how to create that Secret from the Job and how to use it.

How the Secret is created

The init Job runs bao operator init and then creates the Secret using oc (or kubectl). The Secret name and key used here are openbao-init-data and init.json (the file contains the JSON output of bao operator init, including unseal_keys_b64 and root_token). One advantage of this approach is that you can integrate it into your GitOps pipeline. However, we can argue whether it is a good idea to store the unseal keys and the root token as a Secret in the same cluster where OpenBao is running.

Example structure of init.json:

{
"unseal_keys_b64": [
"key1",
"key2",
"key3",
"key4",
"key5"
],
"unseal_keys_hex": [
"hex1",
"hex2",
"hex3",
"hex4",
"hex5"
],
"unseal_shares": 5,
"unseal_threshold": 3,
"recovery_keys_b64": null,
"recovery_keys_hex": null,
"recovery_keys_shares": 0,
"recovery_keys_threshold": 0,
"root_token": "root.token"
}

You need at least unseal_threshold keys (e.g. 3) from unseal_keys_b64 to unseal each node. Store this file securely; anyone with access can unseal OpenBao and the root_token has full admin access.

RBAC Configuration

The Job’s service account (openbao-init) must be allowed to create (and optionally get/update) Secrets in the openbao namespace. Create a Role and RoleBinding:

apiVersion: v1
kind: ServiceAccount
metadata:
name: openbao-init (1)
namespace: openbao
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: openbao-init-secret-writer
namespace: openbao
rules: (2)
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create", "get", "update", "patch"]
- apiGroups: [""] (3)
resources: ["pods"]
verbs: ["get", "list"]
- apiGroups: [""] (4)
resources: ["pods/exec"]
verbs: ["create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding (5)
metadata:
name: openbao-init-secret-writer
namespace: openbao
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: openbao-init-secret-writer
subjects:
- kind: ServiceAccount
name: openbao-init
namespace: openbao

1	The ServiceAccount is used by the Job(s) to create the Secret and unseal the OpenBao pods.
2	The Role is used by the Job to create the Secret. The rules might be extended to include the permission to execute into the pods to unseal the OpenBao service (see below).
3	Permissions to get and list the Pods, required for the unseal process later
4	Permissions to execute into the Pods, required for the unseal process later
5	The RoleBinding is used by the Job to create the Secret.

The Job must also be able to reach the OpenBao API (e.g. openbao.openbao.svc:8200). This is usually possible when running in the same namespace as the OpenBao service.

The example below shows a working Job manifest that will perform the initialisation and the creation of the Secret in the openbao namespace. The manifest is already prepared for Argo CD with useful annotations. It runs an init container to initialise the OpenBao service and a main container to create the Secret. The init container writes (using the OpenBao CLI) the init.json file to a shared volume and the main container (using the kubectl command) creates the Secret from that file. In addition, the Secret containing the CA certificate is mounted as a volume and the BAO_CACERT environment variable is set to the path of the CA certificate.

Init Job that creates the Secret

apiVersion: batch/v1
kind: Job
metadata:
name: openbao-init
namespace: openbao
annotations: (1)
argocd.argoproj.io/sync-wave: "30"
argocd.argoproj.io/hook: PostSync
argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
template:
spec:
serviceAccountName: openbao-init (2)
initContainers: (3)
- name: openbao-init
image: ghcr.io/openbao/openbao:latest
command: (4)
- /bin/sh
- -c
- |
if [ -f /etc/openbao-ca/ca.crt ]; then
export BAO_CACERT=/etc/openbao-ca/ca.crt
export BAO_ADDR=https://openbao:8200
echo "Using TLS CA from /etc/openbao-ca/ca.crt"
else
export BAO_ADDR=http://openbao:8200
echo "Using plain HTTP"
fi
until bao status 2>&1 | grep -q "Initialized"; do
echo "Waiting for OpenBao..."
sleep 5
done
if bao status | grep -q "Initialized.*false"; then
echo "Initialising OpenBao..."
bao operator init -key-shares=5 -key-threshold=3 \
-format=json > /shared/init.json
echo "Init complete; init.json written to shared volume."
else
echo "OpenBao already initialised (skipping init)."
fi
volumeMounts:
- name: openbao-ca
mountPath: /etc/openbao-ca
readOnly: true
- name: shared
mountPath: /shared
containers: (5)
- name: create-secret
image: bitnami/kubectl:latest
command:
- /bin/sh
- -c
- |
if [ -f /shared/init.json ]; then
echo "Creating Secret openbao-init-data from init.json..."
kubectl create secret generic openbao-init-data \
--from-file=init.json=/shared/init.json \
-n openbao --dry-run=client -o yaml | kubectl apply -f -
echo "Initialisation complete!"
else
echo "No init.json (OpenBao already initialised or init skipped)."
fi
volumeMounts:
- name: shared
mountPath: /shared
readOnly: true
volumes:
- name: openbao-ca
secret:
secretName: openbao-ca-secret
optional: true
- name: shared (6)
emptyDir: {}
restartPolicy: Never
backoffLimit: 3

1	The annotations are used by Argo CD to trigger the Job after the OpenBao deployment.
2	The ServiceAccount is used by the Job to initialise the OpenBao service.
3	The init container runs the OpenBao CLI (wait for API, then operator init) and writes init.json to a shared volume.
4	The command is used to initialise the OpenBao service.
5	The main container runs kubectl to create the Secret from that file. This avoids needing a single image that bundles both `bao` and `kubectl`.
6	The shared volume is used to store the init.json file.

As a result the secret openbao-init-data is created in the openbao namespace. It contains the unseal keys and the root token.

Using the Secret in a Job (unseal)

A second Job can use the Secret openbao-init-data that was created by the init Job to unseal each OpenBao pod. A lot is happening here:

The init container extracts the unseal keys from the Secret using jq and writes them to a shared volume.
The main container is reading the unseal keys from the shared volume and unseals the first pod
Then the process is waiting for 30 seconds, to give the next pod time to pull the image and start
The process is repeated for the next pod and so on until all pods are unsealed.

First of all, the unseal Job needs permission to exec into the OpenBao pods. Add a Role that grants the Job’s service account create on pods/exec in the openbao namespace. To make it easier, we can extend the Role openbao-init-secret-writer that was created in the init Job.

These rules were already applied with the configuration for the init Job, but we will repeat them here for completeness.

rules: (1)
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"]

1	The role `openbao-init-secret-writer` must be extended to include the permission to execute into the pods to unseal the OpenBao service.

apiVersion: batch/v1
kind: Job
metadata:
name: openbao-unseal
namespace: openbao
annotations:
argocd.argoproj.io/sync-wave: "35"
argocd.argoproj.io/hook: PostSync
argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
template:
spec:
serviceAccountName: openbao-init (1)
initContainers:
- name: extract-keys
image: quay.io/codefreshplugins/curl-jq
command:
- /bin/sh
- -c
- |
if [ ! -f /secrets/init.json ]; then
echo "Secret not found; run init Job first."
exit 1
fi
for i in 0 1 2; do
jq -r ".unseal_keys_b64[$i]" /secrets/init.json | tr -d '\n' > "/shared/key$i" (2)
done
echo "Unseal keys extracted to shared volume."
volumeMounts:
- name: init-secret
mountPath: /secrets
readOnly: true
- name: shared
mountPath: /shared
containers: (3)
- name: unseal
image: bitnami/kubectl:latest
command: (4)
- /bin/sh
- -c
- |
if [ -f /etc/openbao-ca/ca.crt ]; then
export BAO_CACERT=/etc/openbao-ca/ca.crt
export BAO_ADDR=https://openbao:8200
echo "Using TLS CA from /etc/openbao-ca/ca.crt"
else
export BAO_ADDR=http://openbao:8200
echo "Using plain HTTP"
fi
sleep_timer=30
if [ ! -f /shared/key0 ]; then
echo "Keys not found; extract-keys init container may have failed."
exit 1
fi
echo "Unsealing pods openbao-0, openbao-1, openbao-2 (3 key steps each), ${sleep_timer}s delay between pods..."
for pod in openbao-0 openbao-1 openbao-2; do
echo "Unsealing $pod..."
for i in 0 1 2; do
key=$(cat "/shared/key$i" | tr -d '\n')
kubectl exec -n openbao "$pod" -- sh -c 'bao operator unseal "$1"' _ "$key" || true
done
echo "$pod unsealed."
if [ "$pod" != "openbao-2" ]; then
echo "Waiting ${sleep_timer}s for next pod to be ready..."
SLEEPER_TMP=1
while [ "$SLEEPER_TMP" -le "$sleep_timer" ]; do
if [ $(( SLEEPER_TMP % 10 )) -eq 0 ]; then
echo -n "$SLEEPER_TMP"
else
echo -n "."
fi
sleep 1
SLEEPER_TMP=$(( SLEEPER_TMP + 1 ))
done
echo ""
fi
done
echo "Unseal complete."
volumeMounts:
- name: shared
mountPath: /shared
readOnly: true
- name: openbao-ca
mountPath: /etc/openbao-ca
readOnly: true
volumes: (5)
- name: init-secret
secret:
secretName: openbao-init-data
- name: shared
emptyDir: {}
- name: openbao-ca
secret:
secretName: openbao-ca-secret
optional: true
restartPolicy: Never
backoffLimit: 2

1	The ServiceAccount is used by the Job to unseal the OpenBao pods. Here we are using the same ServiceAccount as the one used to initialise the OpenBao service.
2	The init container extracts unseal keys from init.json using jq and writes them to a shared volume.
3	The main container runs kubectl to exec into each OpenBao pod and run `bao operator unseal` (the OpenBao image is used inside the target pods).
4	The command is used to unseal the OpenBao service.
5	The volumes that are mounted: init-secret (contains the unseal keys), shared (contains the extracted unseal keys, shared between containers), openbao-ca (contains the CA certificate).

Both Jobs are using the same ServiceAccount and therefore the same RBAC rules. It is also possible to use a different ServiceAccount for the unseal Job.

Store init data securely. Keeping unseal keys in a cluster Secret is convenient but less secure than auto-unseal or an external secrets manager. Restrict access to the openbao-init-data Secret (e.g. RBAC and network policies) and consider rotating or moving keys to a more secure store after bootstrap.

Approach 3: Auto-unseal with Key Management Service KMS (recommended for production)

Configure a KMS (e.g. AWS KMS, Azure Key Vault, GCP Cloud KMS, Transit etc.) so that OpenBao unseals itself on every restart. You initialise once and thereafter no manual unseal step is needed.

The rest of this article will mention the different options. However, I can only demonstrate a real example using the AWS KMS option.

Auto-unseal options overview

Without auto-unseal, you must run bao operator unseal with a threshold of keys after each restart. With auto-unseal, OpenBao uses a seal backend to protect the root key and unseals itself on startup.

Supported options:

Option

Use case

Notes

1. AWS KMS

AWS (EKS, OpenShift on AWS)

IRSA or IAM user; no keys in cluster with IRSA.

2. Azure Key Vault

Azure (AKS, OpenShift on Azure)

Service principal or managed identity.

3. Google Cloud KMS

GCP (GKE)

Service account key or Workload Identity.

5. Transit

Existing Vault/OpenBao as root of trust; or a dedicated seal-only OpenBao that exists only to unseal production

External transit engine; token in a Secret.

Why use OpenBao when the seal is in KMS? Why not keep all secrets in KMS and skip OpenBao?

When using a cloud KMS (or Key Vault) for auto-unseal, a natural question is: why not store and retrieve all secrets from the KMS and run without OpenBao at all? OpenBao remains useful for operators and applications for several reasons:

KMS is for keys, not a full secrets platform. AWS KMS, Azure Key Vault keys, and GCP Cloud KMS are built to encrypt and decrypt small blobs (e.g. data encryption keys, or OpenBao’s seal blob). They are not a general-purpose secrets store with versioning, path-based access, dynamic credentials, and per-request audit. OpenBao provides that layer: applications request secrets by path, get short-lived tokens, and you get a single place to define who can read what.
Dynamic secrets. OpenBao can generate short-lived credentials on demand (e.g. database users, cloud IAM roles, PKI certificates). The KMS does not create or rotate such credentials; it only holds keys. If you need “give this pod a DB password that expires in 1 hour,” that is OpenBao’s domain, not the KMS.
Fine-grained, identity-aware access. OpenBao supports auth methods (Kubernetes, OIDC, AppRole, LDAP) and path-based policies. You can say “this service account may read only secret/data/myapp/*.” The KMS has IAM or Key Vault RBAC, but not the same model of “one API, many identities, many paths” that applications and operators use day to day.
Audit and compliance. OpenBao logs every secret read and write with identity and path. That gives a clear audit trail of who accessed which secret and when. KMS audit logs key usage, which is different from “which application read which secret at what time” in a unified way.
Abstraction and portability. Applications talk to OpenBao’s API. You can move the seal or the storage backend (or even the cloud) without changing how applications request secrets. If you put everything directly in a cloud KMS, you tie every app to that vendor’s API and limits.
Encryption as a service (Transit). OpenBao’s Transit secrets engine lets applications encrypt data with a key without ever seeing the key—useful for application-level encryption and key rotation. KMS can do something similar, but OpenBao integrates that with the same auth and audit as the rest of your secrets.

In short: the KMS is the root of trust for the seal (so OpenBao can unseal itself). OpenBao is the secrets management platform that applications and operators use for storing, retrieving, generating, and auditing secrets. Both layers complement each other.

General flow (same for all options):

Set up the seal backend (KMS key, Key Vault, static key, or transit server).
Add the appropriate seal "…" stanza to OpenBao configuration (e.g. in Helm server.ha.raft.config).
Deploy OpenBao; pods start and remain sealed until initialised.
Run one-time initialisation: bao operator init -recovery-shares=5 -recovery-threshold=3.
From then on, restarts are automatically unsealed.

Option: AWS KMS

Best for: EKS, OpenShift on AWS (ROSA), or any Kubernetes on AWS.

Create a KMS key

Create a symmetric KMS key used only for the OpenBao seal (do not use for application data).

AWS WebUI:
- search for: KMS → Create key → Symmetric, Encrypt/Decrypt
- Optionally set an alias (e.g. openbao-unseal).
- OPTIONAL: Define key administrative permissions
- OPTIONAL: Define key usage permissions
- Edit key policy

AWS CLI: (requires the AWS CLI to be installed and configured)

aws kms create-key \
--description "OpenBao auto-unseal key" \
--key-usage ENCRYPT_DECRYPT
aws kms create-alias \
--alias-name openbao-unseal \
--target-key-id <key-id-from-above>

Note the Key ID or Alias and the region (e.g. us-west-1) from the output of the command.

Key policy for OpenBao

The OpenBao process needs permission to use the key. Create an IAM policy. In the UI you can configure this directly when creating the key. When using the CLI, create a policy.json file with the following content:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": "arn:aws:kms:<REGION>:<ACCOUNT_ID>:key/<KEY_ID>" (1)
}
]
}

1	Replace `<REGION>`, `<ACCOUNT_ID>`, and `<KEY_ID>` with your values (or use the key ARN).

Create an IAM user for the seal

When using static credentials (e.g. on OpenShift or when IRSA is not available), create a dedicated IAM user that has permission only to use the KMS key above. Use that user’s access keys in the Kubernetes Secret.

AWS WebUI:
- search for: IAM → Users → Create user
- User name: e.g. openbao-seal
- DO NOT select "Provide user access to the AWS Management Console" (programmatic access only)
- Create user
- Open the created user → Permissions → Add permissions → Create inline policy (or Attach policies → Create policy)
- In the policy editor, choose JSON and paste the policy from Key policy for OpenBao, replacing <REGION>, <ACCOUNT_ID>, and <KEY_ID> with your KMS key ARN or alias
- Name the policy (e.g. OpenBaoSealKMS) and attach it to the user
- Security credentials → Create access key → Application running outside AWS (or "Third-party service") → Create access key
- Save the Access key ID and Secret access key; you will put them in the Kubernetes Secret openbao-aws-credentials.

AWS CLI: (requires the AWS CLI to be installed and configured)

Create a policy file (e.g. openbao-seal-policy.json) with the same JSON as in Key policy for OpenBao, then create the user, attach the policy, and create access keys:

# Replace REGION, ACCOUNT_ID, and KEY_ID in the policy (or use alias: arn:aws:kms:REGION:ACCOUNT_ID:alias/openbao-unseal)
aws iam create-policy \
--policy-name OpenBaoSealKMS \
--policy-document file://openbao-seal-policy.json
aws iam create-user --user-name openbao-seal
aws iam attach-user-policy \
--user-name openbao-seal \
--policy-arn arn:aws:iam::ACCOUNT_ID:policy/OpenBaoSealKMS
aws iam create-access-key --user-name openbao-seal

The last command returns AccessKeyId and SecretAccessKey; store them in the Secret openbao-aws-credentials with key AWS_REGION set to your region (e.g. us-west-1).

Create the Secret

Create the Secret openbao-aws-credentials in the openbao namespace. It contains the access key, secret access key, and region.

apiVersion: v1
kind: Secret
metadata:
name: openbao-aws-credentials
namespace: openbao
type: Opaque
stringData: (1)
AWS_ACCESS_KEY_ID: <your-access-key>
AWS_SECRET_ACCESS_KEY: <your-secret-key>
AWS_REGION: <your-region>

1	Replace `<your-access-key>`, `<your-secret-key>`, and `<your-region>` with the values from the previous steps.

Helm / Argo CD configuration (AWS)

Add the seal "awskms" block inside the same HCL config that contains listener, storage "raft". In Helm values this is typically under openbao.server.ha.raft.config.

Be sure to update the region and key ID.

The values file we used for Argo CD can be found at: OpenBao Argo CD values file

openbao:
server:
extraSecretEnvironmentVars: (1)
- envName: AWS_ACCESS_KEY_ID
secretName: openbao-aws-credentials
secretKey: AWS_ACCESS_KEY_ID
- envName: AWS_SECRET_ACCESS_KEY
secretName: openbao-aws-credentials
secretKey: AWS_SECRET_ACCESS_KEY
- envName: AWS_REGION
secretName: openbao-aws-credentials
secretKey: AWS_REGION
ha:
raft:
config: |
ui = true
seal "awskms" { (2)
region = "us-east-1"
kms_key_id = "alias/openbao-unseal"
}
listener "tcp" {
tls_disable = 0
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/openbao/tls/openbao-server-tls/tls.crt"
tls_key_file = "/openbao/tls/openbao-server-tls/tls.key"
tls_min_version = "tls12"
}
storage "raft" {
path = "/openbao/data"
}
service_registration "kubernetes" {}

1	If you use static credentials (e.g. for OpenShift), inject them via a Secret.
2	This must be added. Be sure to update your region and key ID.

The seal "awskms" block alone is not enough. OpenBao must have AWS credentials available at runtime. If you see an error such as NoCredentialProviders: no valid providers in chain or error fetching AWS KMS wrapping key information, the pod has no credentials. You must either inject static credentials from a Secret (see below) or use IRSA so the pod can assume an IAM role.

Approach 4: Init container with external secret store

Another option is to use an init container that retrieves secrets from an external source. Two use cases can be considered here:

Shamir unseal: The init container fetches the unseal keys from an external secret manager and then runs bao operator unseal (or writes keys to a file used by a sidecar/unseal script). This is the classic “external secret store” idea (e.g. using AWS Secrets Manager, HashiCorp Vault, another Kubernetes cluster).
Transit seal token: The init container fetches the token that production OpenBao uses to call the dedicated unseal service (see Dedicated unseal service (seal-only OpenBao)). That token is written to a file or shared volume so the main OpenBao process can use it in seal "transit". The token never lives in a Kubernetes Secret in Git or in plain YAML; it is fetched at pod start from the external store.

Below is a generic placeholder for the init container.

Option: init Container example

The following options are generic and more theoretical. Unlike the other options, I could not test them in a real environment yet. This may be covered in a future article.

Generic placeholder (implement according to your secret store):

# Additional values for the Helm chart
server:
extraInitContainers:
- name: fetch-secrets
image: registry.access.redhat.com/ubi9/ubi-minimal:latest
command:
- /bin/sh
- -c
- |
# Fetch unseal keys OR transit token from external secret manager
echo "Waiting for OpenBao to be ready..."
sleep 30
# Your fetch logic here (e.g. aws secretsmanager get-secret-value, vault kv get, curl to API) (1)
env:
- name: EXTERNAL_SECRET_ENDPOINT (2)
value: "https://your-external-secret-store.example.com"

1	Replace with your fetch logic.
2	Replace with your external secret store endpoint.

Option: Transit seal

When you already have a Vault or OpenBao cluster and want it to act as the root of trust (e.g. central Vault encrypts the seal key), you can use this option.

To prepare everything, you need to do the following:

Enable the Transit secrets engine on the external Vault/OpenBao and create a key (e.g. openbao-unseal).
Grant the token used by this OpenBao permission to encrypt/decrypt with that key.
Store the token in a Kubernetes Secret and inject via extraSecretEnvironmentVars

Do not put the token in the config file.

The following HCL configuration is used to configure the OpenBao cluster to use the transit seal.

seal "transit" {
address = "https://external-vault.example.com:8200"
token = "s.xxx"
disable_renewal = "false"
key_name = "openbao-unseal"
mount_path = "transit/"
}

See the OpenBao transit seal documentation for further details.

Dedicated unseal service (seal-only OpenBao)

A common and recommended pattern is to run a separate, standalone OpenBao instance whose only role is to provide the seal for your production OpenBao. In other words: the external OpenBao holds the unseal keys—via its Transit secrets engine—and the production OpenBao cluster uses the transit seal to auto-unseal by calling that external instance. No application secrets or other workloads run on the dedicated instance; it exists solely to unseal the production OpenBao.

Why use a dedicated unseal service?

Separation of concerns: The root of trust for unsealing lives outside the production cluster. If the production OpenBao is compromised or rebuilt, the seal keys remain in the dedicated instance.
Simpler operations: You unseal and manage only one small, locked-down OpenBao (the seal service). The production OpenBao unseals itself automatically via transit.
No cloud KMS required: On-premise or air-gapped environments can use this pattern instead of AWS KMS, Azure Key Vault, or GCP Cloud KMS.
Clear trust boundary: The dedicated instance can be hardened, network-isolated, and backed up independently.

Architecture (high level):

Dedicated unseal service: A standalone OpenBao (single node is often enough) with the Transit secrets engine enabled and a dedicated transit key (e.g. production-openbao-unseal). This instance is initialised and unsealed once; you keep its unseal keys and root token very secure. It does not store application secrets.
Production OpenBao: Your HA OpenBao cluster (e.g. in Kubernetes) is configured with seal "transit" pointing at the dedicated service URL and a token that has permission only to use that transit key. On startup, each production node asks the dedicated OpenBao to decrypt its seal blob and thus unseals automatically.

Migration from Shamir to auto-unseal

You might wonder if you can migrate from Shamir to auto-unseal. The answer is yes, you can.

If OpenBao was previously initialised with Shamir unseal keys and you want to switch to any auto-unseal backend, you can do the following:

Plan a short maintenance window. Raft HA will tolerate one node at a time.
Add the appropriate seal "…" block to the config and deploy (e.g. via Argo CD).

Do NOT re-run init.
Unseal the leader with the existing Shamir unseal keys.
Run seal migration on the leader:
```
bao operator unseal -migrate
```
Restart the leader. It should auto-unseal via the new seal.
Update and restart standby nodes. They should auto-unseal as well.

Resources

The Guide to OpenBao - GitOps Deployment with Argo CD - Part 5

Fri, 20 Feb 2026 00:00:00 +0000

Following the GitOps mantra "If it is not in Git, it does not exist", this article demonstrates how to deploy and manage OpenBao using Argo CD. This approach provides version control, audit trails, and declarative management for your secret management infrastructure.

Introduction

Deploying OpenBao via GitOps offers significant advantages:

Version Control: All configuration changes are tracked in Git
Audit Trail: Who changed what, when, and why
Declarative: Desired state is defined, not imperative commands
Reproducible: Same deployment process across environments
Self-healing: Argo CD ensures actual state matches desired state

However, there are challenges specific to secret management:

Initial unsealing requires manual intervention or automationxw
Root tokens and unseal keys must be handled carefully
Chicken-and-egg problem: How to store OpenBao secrets before OpenBao exists?

This article will focus on the deployment of the applicaiotns for the OpenBao deployment. The problem with the automatic unsealing will be addressed in the next article.

Prerequisites

Before you begin, ensure you have:

OpenShift GitOps (Argo CD) installed and configured
A Git repository for your cluster configuration
Understanding of the App-of-Apps pattern (see Configure App-of-Apps)
The official OpenBao Helm chart from Part 3

The (Wrapper) Helm Chart

The official OpenBao Helm Chart works well for deploying OpenBao itself. However, additional settings such as certificates are not covered there. Therefore, I created a (wrapper) Helm Chart that includes the official OpenBao Helm Chart, a Cert-Manager Helm Chart that I created a while ago, and allows you to add additional objects in the templates folder.

I will follow the same setup as discussed in Part 4. This means:

Create Issuer for Cert-Manager
Create CA Certificate for OpenBao
Create Certificate for OpenBao Server
Create Certificate for OpenBao Agent Injector
Install OpenBao Helm Chart

However, there is one significant issue with this approach: we need the certificate details (especially the CA certificate) before we can install the OpenBao Helm Chart, since the certificate must be referenced in the values file. This is a chicken-and-egg problem—particularly if you use self-signed CA certificates.

Since you typically create the CA certificate first and only once (or have it already), a separate Application for the CA certificate is a good approach. Once this is synchronised, the OpenBao Application can be updated and deployed.

This means we will need two Helm Charts: one for the CA certificate and one for OpenBao itself.

Helm Chart for the CA Certificate

Create a new Helm Chart for the CA certificate. This will use the Cert-Manager Helm Chart as a dependency and will create the:

Issuer for Cert-Manager
Request the CA certificate from the Issuer
Create the openbao namespace

The example I am using can be found at GitOps openbao-ca-certificate.

Please check the GitOps Repository Structure and subsequent articles for more information on how to structure your Git repository.

In the Chart.yaml file you can see two dependencies:

dependencies:
- name: tpl
version: ~1.0.0
repository: https://charts.stderr.at/
- name: cert-manager
version: ~2.0.3 (1)
repository: https://charts.stderr.at/
condition: cert-manager.enabled

1	The Cert-Manager Helm Chart version must be at least 2.0.3 to support the namespace parameter for the Issuer.

The tpl dependency is a library that contains templates for the namespace and other shared components (I keep this library in my Helm Charts repository). The cert-manager dependency is the Cert-Manager Helm Chart.

The values.yaml file below contains the configuration for the Cert-Manager:

All settings that are passed to a subchart (cert-manager) must be prefixed with the name of the subchart. In this case, cert-manager..

namespace:
create: true (1)
name: "openbao"
description: "OpenBao Namespace"
displayName: "OpenBao Namespace"
additionalLabels:
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/audit-version: latest
pod-security.kubernetes.io/warn: restricted
pod-security.kubernetes.io/warn-version: latest
cert-manager: (2)
enabled: true (3)
issuer:
# Name of issuer
- name: openbao-selfsigned (4)
# -- Syncwave to create this issuer
syncwave: 5 (5)
# -- Type can be either ClusterIssuer or Issuer
type: Issuer
# -- Enable this issuer.
# @default -- false
enabled: true
# -- Namespace for Issuer (ignored for ClusterIssuer). Defaults to the default namespace when not set.
namespace: openbao (6)
# -- Create a selfSigned issuer. The SelfSigned issuer doesn't represent a certificate authority as such, but instead denotes that certificates will "sign themselves" using a given private key.
selfSigned: true (7)
certificates: (8)
enabled: true
# List of certificates
certificate:
- name: openbao-ca
enabled: true
namespace: openbao
syncwave: "10"
secretName: openbao-ca-secret
duration: 87660h
dnsNames:
- openbao-ca
privateKey:
algorithm: ECDSA
size: 256
rotationPolicy: Always
isCA: true
# Reference to the issuer that shall be used.
issuerRef:
name: openbao-selfsigned
kind: Issuer

1	Create the openbao namespace with various settings.
2	Enable the Cert-Manager subchart. All settings below will be passed to the Cert-Manager subchart.
3	Enables Cert-Manager.
4	Name of the issuer.
5	Syncwave for this issuer; it must be lower than the syncwave of the certificate.
6	Namespace for the issuer; supported since version 2.0.3 of the Cert-Manager Helm Chart.
7	Creates a self-signed issuer.
8	The certificate section and all the settings. Verify the Part 4 for more information.

This chart creates the openbao namespace, which is required for the OpenBao deployment.

Helm Chart for the OpenBao Deployment

Create a new Helm Chart for the OpenBao deployment. This will use the official OpenBao Helm chart and the Cert-Manager Helm Chart as dependencies and will create the:

OpenBao deployment (including Route)
Issuer for the OpenBao server using the CA certificate
Required OpenBao certificates based on the CA certificate

I have added the full values.yaml file below. Please check the GitOps openbao/values.yaml to fetch the full chart.

Besides the two certificates added here in the values.yaml file, I made two further changes: fullnameOverride and nameOverride are both set to openbao. This is good practice for setting the name of the deployment when using Argo CD.

This values file contains the CA certificate in plain text. Whether that is acceptable is debatable—it is a public certificate, and here it is self-signed and used only in my demo environment.

# Full values.yaml file for the OpenBao deployment
########################################################
# Cert-Manager
########################################################
cert-manager:
enabled: true
issuer:
# Name of issuer
- name: openbao-ca-issuer (1)
# -- Syncwave to create this issuer
syncwave: '-1'
# -- Type can be either ClusterIssuer or Issuer
type: Issuer
# -- Enable this issuer.
# @default -- false
enabled: true
# -- Namespace for Issuer (ignored for ClusterIssuer). Defaults to the default namespace when not set.
namespace: openbao
ca:
secretName: openbao-ca-secret
certificates:
enabled: true
# List of certificates
certificate:
########################################################
# OpenBao Server TLS Certificate
########################################################
- name: openbao-server-tls (2)
enabled: true
namespace: openbao
syncwave: "0"
secretName: openbao-server-tls
duration: 8760h
renewBefore: 720h
dnsNames:
- openbao.openbao.svc
- openbao.apps.ocp.aws.ispworld.at # Route host (adjust to your domain)
- openbao
- openbao.openbao
- openbao.openbao.svc
- openbao.openbao.svc.cluster.local
- openbao-internal
- openbao-internal.openbao
- openbao-internal.openbao.svc
- openbao-internal.openbao.svc.cluster.local
- openbao-0.openbao-internal
- openbao-0.openbao-internal.openbao
- openbao-0.openbao-internal.openbao.svc
- openbao-0.openbao-internal.openbao.svc.cluster.local
- openbao-1.openbao-internal
- openbao-1.openbao-internal.openbao
- openbao-1.openbao-internal.openbao.svc
- openbao-2.openbao-internal
- openbao-2.openbao-internal.openbao
- openbao-2.openbao-internal.openbao.svc
ipAddresses:
- 127.0.0.1
- "::1"
# Reference to the issuer that shall be used.
issuerRef:
name: openbao-ca-issuer
kind: Issuer
########################################################
# Injector TLS Certificate
########################################################
- name: injector-certificate (3)
enabled: true
namespace: openbao
syncwave: "0"
secretName: injector-tls
duration: 24h
renewBefore: 144m
dnsNames:
- openbao-agent-injector-svc
- openbao-agent-injector-svc.openbao
- openbao-agent-injector-svc.openbao.svc
- openbao-agent-injector-svc.openbao.svc.cluster.local
# Reference to the issuer that shall be used.
issuerRef:
name: openbao-ca-issuer
kind: Issuer
########################################################
# OpenBao Deployment
########################################################
openbao: (4)
# Override the full name of the deployment via Argo CD
fullnameOverride: openbao
nameOverride: openbao
global:
# Enable OpenShift-specific settings
openshift: true
# -- The namespace to deploy to. Defaults to the `helm` installation namespace.
namespace: openbao
# Required when TLS is enabled: tells the chart to use HTTPS for readiness/liveness
# probes and for in-pod API_ADDR (127.0.0.1:8200). Otherwise you get "client sent
# an HTTP request to an HTTPS server" from the probes.
tlsDisable: false
server:
extraEnvironmentVars:
BAO_CACERT: /openbao/tls/openbao-server-tls/ca.crt
# High Availability configuration
ha:
enabled: true
replicas: 3
# Raft storage configuration
raft:
enabled: true
setNodeId: true
config: |
ui = true
listener "tcp" {
tls_disable = 0
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/openbao/tls/openbao-server-tls/tls.crt"
tls_key_file = "/openbao/tls/openbao-server-tls/tls.key"
tls_min_version = "tls12"
telemetry {
unauthenticated_metrics_access = "true"
}
}
storage "raft" {
path = "/openbao/data"
retry_join {
leader_api_addr = "https://openbao-0.openbao-internal:8200"
leader_tls_servername = "openbao-0.openbao-internal"
leader_ca_cert_file = "/openbao/tls/openbao-server-tls/ca.crt"
}
retry_join {
leader_api_addr = "https://openbao-1.openbao-internal:8200"
leader_tls_servername = "openbao-1.openbao-internal"
leader_ca_cert_file = "/openbao/tls/openbao-server-tls/ca.crt"
}
retry_join {
leader_api_addr = "https://openbao-2.openbao-internal:8200"
leader_tls_servername = "openbao-2.openbao-internal"
leader_ca_cert_file = "/openbao/tls/openbao-server-tls/ca.crt"
}
}
service_registration "kubernetes" {}
telemetry {
prometheus_retention_time = "30s"
disable_hostname = true
}
route:
enabled: true
host: openbao.apps.ocp.aws.ispworld.at
tls:
# Route terminates client TLS; backend can use reencrypt or passthrough
termination: reencrypt
insecureEdgeTerminationPolicy: Redirect
destinationCACertificate: |
-----BEGIN CERTIFICATE-----
MIIBWzCCAQCgAwIBAgIQNdbg4KIu9oi6dDClE8drmjAKBggqhkjOPQQDAjAAMB4X
DTI2MDIxODA5NDIxNFoXDTM2MDIxODIxNDIxNFowADBZMBMGByqGSM49AgEGCCqG
SM49AwEHA0IABIeYw35/kEHvyctLtOA5xMlyQNxUtXtfBbZMUfPh6AN5MFjIGuNS
cn07a3EpSpfY6/3DaPpu+4wYNFlc+/qDNYajXDBaMA4GA1UdDwEB/wQEAwICpDAP
BgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQgyRk5Vv9K0VULcCgga5mRg4O9kTAY
BgNVHREBAf8EDjAMggpvcGVuYmFvLWNhMAoGCCqGSM49BAMCA0kAMEYCIQCwQ7lZ
Q0jzUjJFzpTGkQjU2+OB159LIQMSbSQ7dz8nVQIhAIDa7f87tjQxDxbJio+/vJx2
awFaWnueGOOQpvwCcV/+
-----END CERTIFICATE-----
extraVolumes:
- type: secret
name: openbao-server-tls
path: /openbao/tls
readOnly: true
extraVolumeMounts:
- name: openbao-server-tls
mountPath: /openbao/tls
readOnly: true
# Resource requests and limits
resources:
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 1Gi
cpu: 1000m
# Persistent volume for data
dataStorage:
enabled: true
size: 10Gi
# storageClass: "gp3-csi"
# Injector configuration
injector:
enabled: true
replicas: 2 # HA for the injector too
certs:
secretName: injector-tls
# For a private CA: set caBundle to the CA cert (PEM) so the Kubernetes API server trusts the injector webhook. E.g. oc get secret openbao-ca-secret -n openbao -o jsonpath='{.data.ca\.crt}' | base64 -d
caBundle: "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJXekNDQVFDZ0F3SUJBZ0lRTmRiZzRLSXU5b2k2ZERDbEU4ZHJtakFLQmdncWhrak9QUVFEQWpBQU1CNFgKRFRJMk1ESXhPREE1TkRJeE5Gb1hEVE0yTURJeE9ESXhOREl4TkZvd0FEQlpNQk1HQnlxR1NNNDlBZ0VHQ0NxRwpTTTQ5QXdFSEEwSUFCSWVZdzM1L2tFSHZ5Y3RMdE9BNXhNbHlRTnhVdFh0ZkJiWk1VZlBoNkFONU1GaklHdU5TCmNuMDdhM0VwU3BmWTYvM0RhUHB1KzR3WU5GbGMrL3FETllhalhEQmFNQTRHQTFVZER3RUIvd1FFQXdJQ3BEQVAKQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlFneVJrNVZ2OUswVlVMY0NnZ2E1bVJnNE85a1RBWQpCZ05WSFJFQkFmOEVEakFNZ2dwdmNHVnVZbUZ2TFdOaE1Bb0dDQ3FHU000OUJBTUNBMGtBTUVZQ0lRQ3dRN2xaClEwanpVakpGenBUR2tRalUyK09CMTU5TElRTVNiU1E3ZHo4blZRSWhBSURhN2Y4N3RqUXhEeGJKaW8rL3ZKeDIKYXdGYVdudWVHT09RcHZ3Q2NWLysKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo="
certName: tls.crt
keyName: tls.key
# UI configuration
ui:
enabled: true

1	Issuer for OpenBao. The syncwave is set to '-1' to create the issuer before the certificate request and the OpenBao deployment.
2	Certificate for the OpenBao server, with all DNS names and IP addresses used to reach it. The syncwave is set to '0' so the certificate is created after the issuer.
3	Certificate for the OpenBao Agent Injector, with all DNS names used to reach it. The syncwave is set to '0' so the certificate is created after the issuer.
4	OpenBao deployment configuration; the two overrides fullnameOverride and nameOverride are both set to openbao. See Part 4 for more information.

Creating Argo CD Applications

Whilst the Helm Charts are ready, we need to create the Argo CD Applications for the CA certificate and the OpenBao deployment. If you read our blog carefully, you will notice that these Application resources are created automatically when I add something to the folder clusters/management-cluster :) This is done by leveraging ApplicationSet resources and is fully described in the GitOps Repository Structure and subsequent articles.

For the sake of simplicity, I will show the created Application resources below. The setup is the same for both; they simply target a different path in the Git repository.

# Full Application resource for the OpenBao CA Certificate
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: in-cluster-openbao-ca-certificate
namespace: openshift-gitops
spec:
destination:
name: in-cluster (1)
namespace: default (2)
info:
- name: Description
value: ApplicationSet that Deploys on Management Cluster Configuration (using Git Generator)
project: in-cluster (3)
source:
path: clusters/management-cluster/openbao-ca-certificate (4)
repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops' (5)
targetRevision: main
syncPolicy:
retry:
backoff:
duration: 5s
factor: 2
maxDuration: 3m
limit: 5
---
# Full Application resource for the OpenBao deployment
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: in-cluster-openbao
namespace: openshift-gitops
spec:
destination:
name: in-cluster (1)
namespace: openbao (2)
info:
- name: Description
value: ApplicationSet that Deploys on Management Cluster Configuration (using Git Generator)
project: in-cluster (3)
source:
path: clusters/management-cluster/openbao (6)
repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops' (5)
targetRevision: main
syncPolicy:
retry:
backoff:
duration: 5s
factor: 2
maxDuration: 3m
limit: 5

1	Target cluster; here, the local cluster
2	Namespace of the target cluster; here, the OpenBao deployment will be installed.
3	Argo CD Project (must exist)
4	Path to the Git repository for the CA certificate
5	URL of the Git repository
6	Path to the OpenBao deployment

This will create the following Applications: - in-cluster-openbao-ca-certificate - in-cluster-openbao

The ApplicationSet adds the prefix in-cluster- to each Application name so that they remain unique in Argo CD.

Figure 1. Argo CD: OpenBao Argo CD Applications

The first Application to synchronise is in-cluster-openbao-ca-certificate.

It creates the following:

Namespace
CA Issuer
CA Certificate

Then synchronise in-cluster-openbao. It creates:

OpenBao deployment
OpenBao Agent Injector
OpenBao Server TLS Certificate
OpenBao Agent Injector TLS Certificate

OpenBao is running—what next?

Deploying OpenBao via GitOps gives you version control and declarative management for your secret management infrastructure.

Whilst OpenBao is running and managed by Argo CD, the next step is to configure it: you will need to handle initialisation (for a new cluster) and unsealing (see Part 3 and Part 4). That manual approach does not scale. In the next article, I will discuss ways to automate initialisation and unsealing.

Key takeaways:

Use sync waves to control deployment order
Consider auto-unseal for production (Part 6)
Store initialisation data securely outside Git

Resources

The Guide to OpenBao - Enabling TLS on OpenShift - Part 4

Tue, 17 Feb 2026 00:00:00 +0000

In Part 3 we deployed OpenBao on OpenShift in HA mode with TLS disabled: the OpenShift Route terminates TLS at the edge, and traffic from the Route to the pods is plain HTTP. While this is ok for quick tests, for a production-ready deployment, you should consider TLS for the entire journey. This article explains why and how to enable TLS end-to-end using the cert-manager operator, what to consider, and the exact steps to achieve it.

Introduction

Part 3 uses tls_disable = 1 in the OpenBao listener and relies on the OpenShift Route for TLS. That gives encryption between the client and the Route, but not between the Route and the OpenBao pods or between pods (e.g. Raft). Enabling TLS on OpenBao itself adds encryption in transit everywhere and aligns with defense-in-depth and compliance requirements. After all, we are talking about a secrets management system here and should not compromise security.

This part assumes:

OpenBao is already deployed as in Part 3 (HA with Raft, Agent Injector enabled).
The cert-manager operator is installed and configured on your OpenShift cluster.

The article SSL Certificate Management for OpenShift describes the setup and usage of the cert-manager operator as an example.

Why Enable TLS?

Enabling TLS for OpenBao makes sense for several reasons:

Encryption in transit: Traffic between the Route and the pods, and between OpenBao peers (Raft), is encrypted. Secrets and tokens are never sent in plain text on the cluster network.
Defense in depth: Even if the Route or network is misconfigured, backend traffic remains protected.
Compliance: Many standards (e.g. PCI-DSS, SOC 2) require encryption in transit for sensitive data; TLS to the application (OpenBao) helps satisfy this.
Agent Injector: The injector webhook is called by the Kubernetes API server. Using TLS for the webhook (with a valid certificate) is required for production and when running multiple replicas.
Consistency: Using HTTPS everywhere simplifies client configuration and avoids mixing HTTP/HTTPS in the same environment.

What Must Be Considered?

Two TLS contexts: Each needs its own certificate and configuration.
1. OpenBao server (API and Raft)
2. Agent Injector (mutating webhook)
Certificate SANs: Server certificates must include all names used to reach OpenBao: Route host, internal service names (e.g. openbao.openbao.svc, openbao-0.openbao-internal.openbao.svc), 127.0.0.1 and ::1 for in-pod traffic, and any external DNS you use. The injector certificate must match the injector Service DNS name (e.g. openbao-agent-injector-svc.openbao.svc).
cert-manager: Using cert-manager gives automatic issuance and renewal. You need a ClusterIssuer (or an Issuer) in the OpenBao namespace.

In this example, we will use a self-signed CA for the OpenBao server and the Agent Injector. In a production environment, you should use a trusted CA.

OpenShift Route: With backend TLS enabled, you can keep the Route in reencrypt mode: the Route terminates TLS from the client and opens a new TLS connection to the pod. Alternatively, use passthrough if you want end-to-end TLS without re-encryption at the Route.
Raft join: After switching to TLS, retry_join and cluster addresses must use https:// and the correct hostnames. Existing unseal keys and root token are unchanged; only the transport is different.
Clients: CLI and applications must use https:// for BAO_ADDR and, if you use a private CA, BAO_CACERT (or the system trust store) so that the client trusts the server certificate.

Prerequisites

OpenBao HA deployment from Part 3 (namespace openbao, Helm release openbao).
cert-manager operator installed on OpenShift.
Sufficient rights to create Issuers, Certificates, and Secrets in the openbao namespace.

Overview of Steps

Create a Certificate Authority (CA) in the openbao namespace (or use an existing ClusterIssuer).
Issue a Certificate for the OpenBao server (API + Raft) and store it in a Secret.
Issue a Certificate for the Agent Injector and reference it in the Helm values.
Update Helm values to mount the server cert and CA, and configure the listener and Raft for TLS.
Upgrade the Helm release; initialize and unseal openbao-0 (new cluster) or re-unseal (existing).
Verify access via HTTPS and configure clients (BAO_ADDR, BAO_CACERT).

Step 1: Certificate Authority (CA) for OpenBao

If you already have a ClusterIssuer (e.g. Let’s Encrypt or an enterprise CA), you can use it for the server and injector certificates and skip this step. For a self-signed CA in the OpenBao namespace (typical for internal cluster TLS), create the following.

This self-signed CA is only for testing purposes. In a production environment, you should use a trusted CA, preferably your own.

Create a self-signed CA Issuer in the openbao namespace:

You might have your own CA or a ClusterIssuer already. You can use them, instead of creating a new one.
```
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: openbao-selfsigned
namespace: openbao (1)
spec:
selfSigned: {} (2)
```
1 The namespace where the OpenBao deployment is running.

2 The self-signed CA Issuer.

Create a self-signed CA Certificate in the openbao namespace:

This will now actually request the CA certificate from the self-signed CA Issuer. This process is fully automated by cert-manager because everything is self-signed. The requested certificate will be stored in the secret openbao-ca-secret and is available immediately.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: openbao-ca
namespace: openbao (1)
spec:
isCA: true (2)
commonName: OpenBao CA
secretName: openbao-ca-secret (3)
duration: 87660h (4)
privateKey: (5)
algorithm: ECDSA
size: 256 (6)
rotationPolicy: Always (6)
issuerRef:
name: openbao-selfsigned (7)
kind: Issuer (8)
group: cert-manager.io

1	The namespace where the OpenBao deployment is running.
2	The certificate is a CA certificate.
3	The name of the secret where the certificate and key will be stored.
4	The duration of the certificate. In this case 10 years.
5	The private key algorithm and size.
6	The rotation policy of the private key.
7	The issuer of the certificate.
8	The kind of the issuer. Can be Issuer or ClusterIssuer.

Step 2: Create an Issuer for the OpenBao Server and Agent Injector

Create an Issuer for the OpenBao Server and Agent Injector. This Issuer will reference the CA certificate and key stored in the secret openbao-ca-secret.

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: openbao-ca-issuer
namespace: openbao
spec:
ca:
secretName: openbao-ca-secret

The name of the secret where the CA certificate and key are stored.

Step 3: Certificate for the OpenBao Server

Now it is time to create the certificate for the OpenBao server. The server certificate must include every hostname used to reach OpenBao: the Route host, the headless service, and each Raft member. Adjust the dnsNames and optional uris to match your cluster and Route.

Create the following Certificate object:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: openbao-server-tls
namespace: openbao
spec:
secretName: openbao-server-tls (1)
duration: 8760h (2)
renewBefore: 720h (3)
commonName: openbao.openbao.svc (4)
dnsNames:
- openbao.apps.cluster.example.com # Route host (adjust to your domain) (5)
- openbao (6)
- openbao.openbao
- openbao.openbao.svc
- openbao.openbao.svc.cluster.local
- openbao-internal
- openbao-internal.openbao
- openbao-internal.openbao.svc
- openbao-internal.openbao.svc.cluster.local
- openbao-0.openbao-internal
- openbao-0.openbao-internal.openbao
- openbao-0.openbao-internal.openbao.svc
- openbao-0.openbao-internal.openbao.svc.cluster.local
- openbao-1.openbao-internal
- openbao-1.openbao-internal.openbao
- openbao-1.openbao-internal.openbao.svc
- openbao-2.openbao-internal
- openbao-2.openbao-internal.openbao
- openbao-2.openbao-internal.openbao.svc
ipAddresses: (7)
- 127.0.0.1
- "::1"
issuerRef:
name: openbao-ca-issuer (8)
kind: Issuer
group: cert-manager.io

1	The name of the secret where the certificate and key will be stored.
2	The duration of the certificate. In this case 1 year.
3	The duration before the certificate is renewed. In this case 30 days.
4	The common name of the certificate. This is the service name of the OpenBao server.
5	The Route host. Adjust to your domain.
6	The headless service names, all should be included.
7	Required for in-pod traffic: Readiness/liveness probes and local bao commands (e.g. raft join, operator unseal) connect to 127.0.0.1:8200. Without these IP SANs you get "tls: bad certificate" or "x509: cannot validate certificate for 127.0.0.1 because it doesn’t contain any IP SANs".
8	The issuer of the certificate, this time openbao-ca-issuer.

After a few moments the certificate will be ready and the secret will be created. The cert-manager will store the signed certificate and key in the Secret openbao-server-tls with keys tls.crt and tls.key. The Helm chart can mount this secret for the OpenBao listener.

Step 4: Certificate for the Agent Injector

The Agent Injector runs as a webhook; the Kubernetes API server calls it over TLS. The certificate must match the Service DNS name of the injector. Create a Certificate that references the same CA Issuer. In this example we use a short-lived certificate valid for 24 hours, renewed when 10% of the validity period remains.

Save as openbao-injector-cert.yaml:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: injector-certificate
namespace: openbao
spec:
secretName: injector-tls (1)
duration: 24h (2)
renewBefore: 144m
commonName: Agent Inject Cert (3)
dnsNames:
- openbao-agent-injector-svc (4)
- openbao-agent-injector-svc.openbao
- openbao-agent-injector-svc.openbao.svc
- openbao-agent-injector-svc.openbao.svc.cluster.local
issuerRef:
name: openbao-ca-issuer
kind: Issuer
group: cert-manager.io

1	The name of the secret where the certificate and key will be stored.
2	The duration of the certificate. In this case 24 hours. (renewal is done 10% before expiry)
3	The injector service name.
4	The injector service name is defined by the Helm chart. If you override the injector service name, adjust dnsNames accordingly.

Step 5: Helm Values for Server TLS

With all the certificates created, we can update the Helm values so that OpenBao uses the server certificate and listens with TLS. You need to:

Mount the Secret openbao-server-tls into the OpenBao pods.
Set the listener to use tls_cert_file and tls_key_file and disable tls_disable.
Switch Raft retry_join and cluster addresses to https://.
Add the environment variable BAO_CACERT to the OpenBao pods.

Before we start, we need to export the CA certificate from the secret openbao-ca-secret and save its value:

oc get secret openbao-ca-secret -n openbao -o jsonpath='{.data.ca\.crt}' | base64 -d

This will return the certificate like this:

-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

Save this, you will need it in the next step.

Now we need to create the Helm values file. This time we will enable TLS. Refer to Part 3 of this series to see what the initial values file looks like.

Create or update openbao-ha-values-tls.yaml (building on your Part 3 values):

global:
# Enable OpenShift-specific settings
openshift: true
# Required when TLS is enabled: tells the chart to use HTTPS for readiness/liveness
# probes and for in-pod API_ADDR (127.0.0.1:8200). Otherwise you get "client sent
# an HTTP request to an HTTPS server" from the probes.
tlsDisable: false (1)
server:
extraEnvironmentVars:
BAO_CACERT: /openbao/tls/openbao-server-tls/ca.crt (2)
# High Availability configuration
ha:
enabled: true
replicas: 3
# Raft storage configuration
raft:
enabled: true
setNodeId: true
config: |
ui = true
listener "tcp" {
tls_disable = 0 (3)
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/openbao/tls/openbao-server-tls/tls.crt" (4)
tls_key_file = "/openbao/tls/openbao-server-tls/tls.key" (5)
tls_min_version = "tls12" (6)
telemetry {
unauthenticated_metrics_access = "true"
}
}
storage "raft" {
path = "/openbao/data"
retry_join {
leader_api_addr = "https://openbao-0.openbao-internal:8200" (7)
leader_tls_servername = "openbao-0.openbao-internal"
leader_ca_cert_file = "/openbao/tls/openbao-server-tls/ca.crt" (8)
}
retry_join {
leader_api_addr = "https://openbao-1.openbao-internal:8200"
leader_tls_servername = "openbao-1.openbao-internal"
leader_ca_cert_file = "/openbao/tls/openbao-server-tls/ca.crt"
}
retry_join {
leader_api_addr = "https://openbao-2.openbao-internal:8200"
leader_tls_servername = "openbao-2.openbao-internal"
leader_ca_cert_file = "/openbao/tls/openbao-server-tls/ca.crt"
}
}
service_registration "kubernetes" {}
telemetry {
prometheus_retention_time = "30s"
disable_hostname = true
}
route:
enabled: true
host: openbao.apps.cluster.example.com
tls: (9)
# Route terminates client TLS; backend can use reencrypt or passthrough
termination: reencrypt
insecureEdgeTerminationPolicy: Redirect
destinationCACertificate: | (10)
-----BEGIN CERTIFICATE-----
# CA Certificate
-----END CERTIFICATE-----
extraVolumes: (11)
- type: secret
name: openbao-server-tls
path: /openbao/tls
readOnly: true
extraVolumeMounts: (12)
- name: openbao-server-tls
mountPath: /openbao/tls
readOnly: true
# Resource requests and limits
resources:
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 1Gi
cpu: 1000m
# Persistent volume for data
dataStorage:
enabled: true
size: 10Gi
# storageClass: "gp3-csi"
# Injector configuration
injector:
enabled: true
replicas: 2 # HA for the injector too
certs: (13)
secretName: injector-tls
# For a private CA: set caBundle to the CA cert (PEM) so the Kubernetes API server trusts the injector webhook. E.g. oc get secret openbao-ca-secret -n openbao -o jsonpath='{.data.ca\.crt}' | base64 -d
caBundle: "BASE64_ENCODED_CA_CERTIFICATE"
certName: tls.crt
keyName: tls.key
# UI configuration
ui:
enabled: true

1	global.tlsDisable: Set to false when the server listener uses TLS. This makes the chart use HTTPS for readiness/liveness probes and for the in-pod API_ADDR env var. If you leave it true (default), probes and local clients will use HTTP and you will see "client sent an HTTP request to an HTTPS server".
2	The environment variable BAO_CACERT is set to the CA certificate file path. This is helpful to execute the boa command inside the container.
3	The listener tls_disable is set to 0 to enable TLS.
4	The listener tls_cert_file is set to the certificate file path.
5	The listener tls_key_file is set to the key file path.
6	The listener tls_min_version is set to tls12.
7	The leader API address is set to the HTTPS address.
8	The leader CA certificate file is set to the CA certificate file path.
9	The Route tls termination is set to reencrypt.
10	The destination CA certificate is set to the CA certificate. Be sure not to add any extra lines or spaces.
11	The extra volumes are mounted at the /openbao/tls path.
12	The extra volume mounts are mounted at the /openbao/tls path.
13	The injector certs are set to the injector TLS secret name. The caBundle must be provided in base64 format.

The secret key names must match what cert-manager writes: tls.crt and tls.key. The OpenBao Helm chart mounts each entry in extraVolumes at path + name (e.g. with path: /openbao/tls and name: openbao-server-tls the secret is mounted at /openbao/tls/openbao-server-tls/). The listener tls_cert_file and tls_key_file must use that full path.

Be sure to change the Route host in the Helm values to the one you are using. In addition, make sure that the CA certificate is added correctly to the Route. Do not add any extra lines or spaces and do not forget the | after destinationCACertificate: and the -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- lines.

Step 6: Upgrade the Helm Release

Perform a Helm upgrade so the new volumes and configuration are applied. Pods will restart and pick up TLS; Raft will use HTTPS for join and replication.

helm upgrade openbao openbao/openbao \
--namespace openbao \
--values openbao-ha-values-tls.yaml

Watch the rollout. The first pod (openbao-0) will log "raft retry join initiated" and stay 0/1 Ready until it is initialized and unsealed (new cluster) or until it forms quorum (existing cluster).

New cluster: initialize and unseal openbao-0

openbao-0 will not become Ready until it is initialized and unsealed. Fetch the certificate, use port-forward and talk to OpenBao over HTTPS with the CA:

# 1. Save the CA cert (for BAO_CACERT)
oc get secret openbao-ca-secret -n openbao -o jsonpath='{.data.ca\.crt}' | base64 -d > openbao-ca.crt
# 2. Port-forward to openbao-0 (background)
oc port-forward openbao-0 8200:8200 -n openbao &
# 3. Use HTTPS and CA cert
export BAO_ADDR='https://127.0.0.1:8200'
export BAO_CACERT="$PWD/openbao-ca.crt"
# 4. Check status, then initialize (only once) and unseal 3 times
bao status
bao operator init -key-shares=5 -key-threshold=3 -format=json > openbao-init.json
bao operator unseal
bao operator unseal
bao operator unseal
bao status

After unsealing, openbao-0 becomes leader and goes 1/1 Ready. Then start openbao-1 and openbao-2 and join or unseal them as in Part 3 (use https://).

Perform the following steps for the other pods - join the raft cluster and unseal them (3 times):

oc exec -ti openbao-1 -- bao operator raft join https://openbao-0.openbao-internal:8200
# 3 times with 3 different unseal keys
oc exec -ti openbao-1 -- bao operator unseal
Unseal Key (will be hidden):

Repeat the same steps for openbao-2.

Existing cluster: re-unseal after restart

The existing cluster is already initialized. Re-unseal the pods (and re-join the raft cluster if necessary).

oc get pods -n openbao -w
oc exec -ti openbao-1 -- bao operator raft join https://openbao-0.openbao-internal:8200
oc exec -ti openbao-1 -- bao operator unseal # three times; same for openbao-2

Repeat the same steps for openbao-2.

Step 7: Verify and Use HTTPS from Clients

Verify server health over HTTPS (from inside the cluster):

curl -k https://openbao.apps.cluster.example.com/v1/sys/health | jq

Be sure to use the Route host in the Helm values.

From your workstation, use the Route URL with HTTPS. If the Route host is signed by your internal CA, set BAO_CACERT to the CA file:

export BAO_ADDR='https://openbao.apps.cluster.example.com'
export BAO_CACERT='/path/to/openbao-ca.crt'
bao status

Be sure to use the Route host in the Helm values.

Agent Injector: New pods that use the injector should start without webhook certificate errors. Check injector logs if you see TLS or certificate errors:

oc logs -n openbao -l app.kubernetes.io/name=openbao-agent-injector -f

Troubleshooting

MutatingWebhookConfiguration conflict (vault-k8s)

conflict occurred while applying object ... MutatingWebhookConfiguration: Apply failed with 1 conflict: conflict with "vault-k8s" using ... .webhooks[name="vault.hashicorp.com"].clientConfig.caBundle
this is a server-side apply (SSA) field ownership conflict. The OpenBao chart uses the same webhook name (vault.hashicorp.com) as the HashiCorp Vault agent injector for annotation compatibility. The clientConfig.caBundle field is still owned by a previous manager (e.g. a prior Vault Helm release or the Vault injector), so Helm cannot update it when you change the injector certificate.

Fix option 1 – Delete the webhook and re-upgrade (recommended)

Remove the MutatingWebhookConfiguration so Helm can recreate it and own all fields. There will be a short window where the injector webhook is missing (new pods requesting injection may fail until the upgrade completes).

# Replace RELEASE_NAME with your Helm release name (e.g. openbao)
RELEASE_NAME=openbao
oc delete mutatingwebhookconfiguration ${RELEASE_NAME}-agent-injector-cfg
# Re-run the upgrade
helm upgrade ${RELEASE_NAME} openbao/openbao \
--namespace openbao \
--values openbao-ha-values-tls.yaml

Fix option 2 – Force takeover of the conflicting field

If you cannot delete the webhook (e.g. in production), take over the caBundle field with server-side apply, then run the Helm upgrade again:

Export the MutatingWebhookConfiguration, set webhooks[0].clientConfig.caBundle to your injector CA (base64 PEM from openbao-ca-secret), then re-apply with --server-side --force-conflicts:

# Get the current object and the new CA bundle
oc get mutatingwebhookconfiguration openbao-agent-injector-cfg -o yaml > mwc.yaml
CA_BUNDLE=$(oc get secret openbao-ca-secret -n openbao -o jsonpath='{.data.ca\.crt}')
# Edit mwc.yaml: set .webhooks[0].clientConfig.caBundle to the value of CA_BUNDLE (no quotes in YAML).
# Then apply with force-conflicts so Helm can later manage it:
oc apply -f mwc.yaml --server-side --force-conflicts
# Re-run the Helm upgrade
helm upgrade openbao openbao/openbao --namespace openbao --values openbao-ha-values-tls.yaml

If you still have HashiCorp Vault’s agent injector installed on the same cluster, ensure only one injector is active for a given namespace (e.g. use namespace selectors) or uninstall the Vault injector to avoid two webhooks with the same name in different resources.

Route / UI – tls: bad record MAC

Pod logs show tls: bad record MAC from the router IP. The Route is likely using edge termination (HTTP to pod). Fix: Use reencrypt and set destinationCACertificate (Step 5).

oc get route openbao -n openbao -o jsonpath='{.spec.tls.termination}'

If the OpenBao UI does not load and the pod logs show:

http: TLS handshake error from 10.x.x.x:xxxxx: local error: tls: bad record MAC

the traffic is coming from the OpenShift router (the IP is typically a cluster pod IP).

Cause: The Route is likely using edge termination: the router terminates TLS at the edge and sends plain HTTP to the pod. The pod expects HTTPS, so the TLS layer receives non-TLS data and reports "bad record MAC".

Fix: Use reencrypt (or passthrough) and set destinationCACertificate so the router talks HTTPS to the pod. See Step 5 above. Quick check:

oc get route openbao -n openbao -o jsonpath='{.spec.tls.termination}'
# Must be "reencrypt" or "passthrough", not "edge"

If the output is edge, patch the Route to reencrypt and set spec.tls.destinationCACertificate to the CA PEM (Step 5).

TLS handshake / 127.0.0.1 certificate errors

If you see in the pod logs:

remote error: tls: bad certificate (from 127.0.0.1)

or when running:

oc exec -ti openbao-0 — bao operator raft join https://…;: x509: cannot validate certificate for 127.0.0.1 because it doesn’t contain any IP SANs

The server certificate does not include 127.0.0.1 (and optionally ::1) as Subject Alternative Names. Readiness/liveness probes and in-pod bao commands connect to the listener on 127.0.0.1, so the certificate must include these IP SANs.

Fix: Add ipAddresses to the OpenBao server Certificate and let cert-manager re-issue:

# Edit the Certificate (or re-apply the YAML from Step 3 with ipAddresses added)
oc edit certificate openbao-server-tls -n openbao
# Add under spec:
# ipAddresses:
# - 127.0.0.1
# - "::1"
# cert-manager will issue a new cert; wait until the secret is updated
oc get certificate openbao-server-tls -n openbao
# Restart OpenBao pods so they load the new cert
oc rollout restart statefulset/openbao -n openbao

Resources

The Guide to OpenBao - OpenShift Deployment with Helm - Part 3

Fri, 13 Feb 2026 00:00:00 +0000

After understanding standalone installation in Part 2, it is time to deploy OpenBao on OpenShift/Kubernetes using the official Helm chart. This approach provides high availability, Kubernetes-native management, and seamless integration with the OpenShift ecosystem.

Introduction

Deploying OpenBao on OpenShift/Kubernetes offers several advantages:

High Availability: Multiple replicas with automatic failover
Kubernetes-native: Managed by standard Kubernetes primitives
Persistent Storage: Data survives pod restarts via PVCs
Integration: Works with Kubernetes service accounts for authentication
Scalability: Easy to scale and manage

The official OpenBao Helm chart supports multiple deployment modes:

Dev: Single server, in-memory storage (testing only)
Standalone: Single server, persistent storage
HA: Multiple servers with Raft consensus (recommended)
External: Connect to external OpenBao cluster

Integrations

Currently, OpenBao supports the following integrations that help seamlessly load secrets into applications without the need to modify the application code:

Agent Injector: A mutating webhook that automatically injects a sidecar container that retrieves and renews secrets.
CSI Provider: A (vendor neutral) CSI driver that mounts secrets as volumes.

Prerequisites

Before deploying, ensure you have:

OpenShift 4.12+ or Kubernetes 1.30+
Helm 3.6+
oc or kubectl CLI configured
The OpenBao CLI (bao) for initialization and unsealing (see OpenBao installation)
Cluster-admin privileges (for initial setup)
A storage class that supports ReadWriteOnce PVCs

# Verify prerequisites
oc version
helm version
# Check available storage classes
oc get storageclass

Adding the Helm Repository

First, add the OpenBao Helm repository:

# Add the OpenBao Helm repository
helm repo add openbao https://openbao.github.io/openbao-helm
# Update repository cache
helm repo update
# Search for available charts
helm search repo openbao
# Expected output:
# NAME CHART VERSION APP VERSION DESCRIPTION
# openbao/openbao 0.x.x 2.x.x Official OpenBao Helm chart

According to the official documentation, the Helm chart is new and under significant development. It should always be run with --dry-run before any install or upgrade to verify changes.

Creating the Namespace

Create a dedicated namespace for OpenBao:

While I am using the oc CLI, you can also use the kubectl CLI as in-place replacement.

oc new-project openbao

Deployment Mode: High Availability (Recommended)

For production environments, deploy in HA mode with Raft. This is the recommended deployment mode for production and is straightforward to achieve on Kubernetes and OpenShift. The following values definitions will use OpenShift specific settings. I will mark them in the callouts.

We will first create a values file, deploy openbao and then discuss what must be done to activate all OpenBao pods.

Create HA Values File

Create openbao-ha-values.yaml:

The values file is based on the official values file from that chart, but only modified values or important changes are listed here.

global:
# Enable OpenShift-specific settings
openshift: true (1)
server:
# High Availability configuration
ha:
enabled: true (2)
replicas: 3 (3)
# Raft storage configuration
raft:
enabled: true (4)
setNodeId: true
config: | (5)
ui = true
listener "tcp" {
tls_disable = 1
address = "[::]:8200"
cluster_address = "[::]:8201"
telemetry {
unauthenticated_metrics_access = "true"
}
}
storage "raft" {
path = "/openbao/data"
retry_join {
leader_api_addr = "http://openbao-0.openbao-internal:8200"
}
retry_join {
leader_api_addr = "http://openbao-1.openbao-internal:8200"
}
retry_join {
leader_api_addr = "http://openbao-2.openbao-internal:8200"
}
}
service_registration "kubernetes" {}
telemetry {
prometheus_retention_time = "30s"
disable_hostname = true
}
route:
enabled: true (6)
host: openbao.apps.cluster.example.com (7)
tls:
termination: edge
# Resource requests and limits
resources:
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 1Gi
cpu: 1000m
# Persistent volume for data
dataStorage: (8)
enabled: true
size: 10Gi
# storageClass: "gp3-csi"
# Injector configuration
injector:
enabled: true
replicas: 2 # HA for the injector too (9)
# UI configuration
ui: (10)
enabled: true

1	OpenShift specific: Activate OpenShift Mode: Critical setting, if you install on OpenShift. It adjusts the Helm chart to use Routes instead of Ingress and modifies RoleBindings to work with OpenShift’s stricter authentication.
2	High Availability (HA): Deploys OpenBao as a StatefulSet rather than a Deployment.
3	Raft Consensus Quorum: Sets the cluster size to 3. Raft requires an odd number of nodes to handle leader elections and avoid split-brain scenarios. This cluster can survive the loss of 1 node.
4	Integrated Raft Storage: Enables the internal Raft storage backend, removing the need for external dependencies like Consul or Etcd.
5	Server Configuration (HCL): The actual OpenBao server configuration file. Note that tls_disable = 1 is used because the OpenShift Route handles TLS termination at the edge, passing unencrypted traffic to the pod.
6	OpenShift specific: Route: Tells Helm to create an OpenShift Route object automatically.
7	OpenShift specific: Host: The external DNS address where users and applications will access the OpenBao API and UI.
8	Persistent Storage: Allocates a 10Gi Persistent Volume Claim (PVC) for each of the 3 pods to store encrypted data and Raft logs.
9	Injector Redundancy: Runs 2 replicas of the sidecar injector. If the injector service is down, new application pods attempting to start with secrets will fail.
10	Web UI: Enables the graphical dashboard service.

Deploy HA Cluster

# Deploy with HA values
helm install openbao openbao/openbao \
--namespace openbao \
--values openbao-ha-values.yaml

Verify the Deployment

oc get pods -n openbao

This will result in the following output:

NAME READY STATUS RESTARTS AGE
openbao-0 0/1 Running 0 60s (1)
openbao-agent-injector-xxx 1/1 Running 0 60s
openbao-agent-injector-yyy 1/1 Running 0 60s

1	Only 1 openbao pod (instead of 3) is running, and the pod is not in "ready" state.

As you can see, only one OpenBao pod exists so far, and it is not in "ready" state. This is because OpenBao is not yet initialized and unsealed. Once that is done, the other pods will appear and join the cluster.

The OpenBao pods show 0/1 ready because they are sealed and need initialization.

This is also indicated in the logs of the openbao pod:

2026-02-13T14:44:52.587Z [ERROR] core: failed to get raft challenge: leader_addr=http://openbao-0.openbao-internal.openbao.svc:8200
error=
| error during raft bootstrap init call: Error making API request.
|
| URL: PUT http://openbao-0.openbao-internal.openbao.svc:8200/v1/sys/storage/raft/bootstrap/challenge
| Code: 503. Errors:
|
| * Vault is sealed (1)
2026-02-13T14:44:52.588Z [ERROR] core: failed to get raft challenge: leader_addr=http://openbao-2.openbao-internal.openbao.svc:8200 error="error during raft bootstrap init call: Put \"http://openbao-2.openbao-internal.openbao.svc:8200/v1/sys/storage/raft/bootstrap/challenge\": dial tcp: lookup openbao-2.openbao-internal.openbao.svc on 172.30.0.10:53: no such host" (2)
2026-02-13T14:44:52.589Z [ERROR] core: failed to get raft challenge: leader_addr=http://openbao-1.openbao-internal.openbao.svc:8200 error="error during raft bootstrap init call: Put \"http://openbao-1.openbao-internal.openbao.svc:8200/v1/sys/storage/raft/bootstrap/challenge\": dial tcp: lookup openbao-1.openbao-internal.openbao.svc on 172.30.0.10:53: no such host"
2026-02-13T14:44:52.589Z [ERROR] core: failed to retry join raft cluster: retry=2s err="failed to get raft challenge"

1	Vault is sealed: This means that the OpenBao service is not yet initialized and unsealed.
2	No such host: This means that the OpenBao service is not yet available.

Initializing and Unsealing OpenBao

After deployment, OpenBao needs to be initialized and unsealed. This is done on the first pod. Once this is done, the other pods will appear and can join the cluster. We will create a local portforwarding to the first pod to initialize it.

Initialize the Cluster

Create a local port forwarding to the first pod

oc port-forward openbao-0 8200:8200 -n openbao &

Set environment variable
```
export BAO_ADDR='http://127.0.0.1:8200'
```

Check status

bao status

This will result in the following output:

Key Value
--- -----
Seal Type shamir
Initialized false (1)
Sealed true (2)
Total Shares 0
Threshold 0
Unseal Progress 0/0
Unseal Nonce n/a
Version 2.5.0
Build Date 2026-02-04T16:19:33Z
Storage Type raft
HA Enabled true (3)

1	Not yet initialized
2	Vault is sealed
3	High Availability is enabled

Initialize the cluster with 5 key shares and 3 threshold
```
bao operator init -key-shares=5 -key-threshold=3 -format=json > openbao-init.json
```
This will create the file openbao-init.json with the unseal keys and root token.
```
cat openbao-init.json
```

Take care of the openbao-init.json file. It contains the unseal keys and root token!

Unseal Pod openbao-0

After initialization, we need to unseal the first pod. This is done by providing 3 different unseal keys. (threshold is 3)

# Unseal openbao-0 (3 times with different keys)
bao operator unseal # Enter first key
bao operator unseal # Enter second key
bao operator unseal # Enter third key

This unseals openbao-0, which can be verified with the command bao status.

Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false (1)
Total Shares 5
Threshold 3
Version 2.5.0
Build Date 2026-02-04T16:19:33Z
Storage Type raft
Cluster Name vault-cluster-80c01167
Cluster ID b81ecb85-9751-655a-95b7-69463dd13241
HA Enabled true
HA Cluster https://openbao-0.openbao-internal:8201
HA Mode active
Active Since 2026-02-13T14:46:13.292643992Z
Raft Committed Index 29
Raft Applied Index 29

1	Vault is unsealed

This makes openbao-0 ready and openbao-1 is trying to start:

oc get pods
NAME READY STATUS RESTARTS AGE
openbao-0 1/1 Running 0 2m10s (1)
openbao-1 0/1 Running 0 14s (2)
openbao-agent-injector-98769cf97-r4stk 1/1 Running 0 2m12s
openbao-agent-injector-98769cf97-xldgm 1/1 Running 0 2m12s

1	openbao-0 is ready
2	openbao-1 is trying to start and wants to join the Raft cluster

Activate Pod openbao-1

Since OpenBao is already initialized, we can skip the initialization step. However, we must join the Raft cluster.

oc exec -ti openbao-1 -- bao operator raft join http://openbao-0.openbao-internal:8200

Once done, we can unseal openbao-1. We need to provide 3 different unseal keys again. Execute the following command 3 times:

oc exec -ti openbao-1 -- bao operator unseal
Unseal Key (will be hidden):

Now openbao-1 is ready and openbao-2 is trying to start:

oc get pods
NAME READY STATUS RESTARTS AGE
openbao-0 1/1 Running 0 3m48s
openbao-1 1/1 Running 0 112s
openbao-2 0/1 Running 0 18s
openbao-agent-injector-98769cf97-r4stk 1/1 Running 0 3m50s
openbao-agent-injector-98769cf97-xldgm 1/1 Running 0 3m50s

Activate Pod openbao-2

Now we need to repeat the previous steps for openbao-2.

oc exec -ti openbao-2 -- bao operator raft join http://openbao-0.openbao-internal:8200
# Run 3 times (enter a different unseal key each time):
oc exec -ti openbao-2 -- bao operator unseal

Verify Raft Cluster

You can verify the Raft cluster by logging in with the root token and then checking the Raft peer list.

oc exec -ti openbao-0 -- bao login

Check the peer list:

# Check Raft peer list
oc exec -ti openbao-0 -- bao operator raft list-peers
Node Address State Voter
---- ------- ----- -----
openbao-0 openbao-0.openbao-internal:8201 leader true
openbao-1 openbao-1.openbao-internal:8201 follower true
openbao-2 openbao-2.openbao-internal:8201 follower true

Accessing the UI

Once unsealed, access the OpenBao UI:

Via Route (Production)

oc get route openbao -n openbao
# Open browser: https://openbao.apps.cluster.example.com

Figure 1. OpenBao Login Form

Upgrading OpenBao

Keep your secrets management up to date. To upgrade an existing deployment:

# Update Helm repository
helm repo update
# Check available versions
helm search repo openbao --versions
# Upgrade with your values file
helm upgrade openbao openbao/openbao \
--namespace openbao \
--values openbao-ha-values.yaml
# Watch the rolling update
oc get pods -n openbao -w

After upgrade, pods may need to be unsealed again if they restart.

Troubleshooting

Pods Not Starting

# Check pod status
oc describe pod openbao-0 -n openbao
# Check pod logs
oc logs openbao-0 -n openbao
# Check events
oc get events -n openbao --sort-by='.lastTimestamp'

PVC Issues

# Check PVC status
oc get pvc -n openbao
# If pending, check storage class
oc describe pvc data-openbao-0 -n openbao

Raft Join Failures

If pods cannot join the Raft cluster:

# Check internal DNS resolution
oc exec -it openbao-0 -n openbao -- nslookup openbao-internal
# Check connectivity between pods
oc exec -it openbao-0 -n openbao -- wget -O- http://openbao-1.openbao-internal:8200/v1/sys/health

What Should Be Considered Next?

Securely store the unseal keys and root token
Configure pod anti-affinity for true HA
Consider auto-unseal for operational ease (upcoming article)
Put everything into a GitOps pipeline

Conclusion

You now have OpenBao running on OpenShift in high-availability mode. This deployment:

Survives pod failures and restarts
Uses Raft for distributed consensus
Integrates with OpenShift security model
Is ready for production use (after unsealing automation)

Key points to remember:

Use HA mode for production
Store unseal keys securely
Configure pod anti-affinity for true HA
Consider auto-unseal for operational ease (upcoming article)

Resources

Hosted Control Planes behind a Proxy

Mon, 15 Dec 2025 00:00:00 +0000

Recently, I encountered a problem deploying a Hosted Control Plane (HCP) at a customer site. The installation started successfully—etcd came up fine—but then it just stopped. The virtual machines were created, but they never joined the cluster. No OVN or Multus pods ever started. The only meaningful message in the cluster-version-operator pod logs was:

I1204 08:22:35.473783 1 status.go:185] Synchronizing status errs=field.ErrorList(nil) status=&cvo.SyncWorkerStatus{Generation:1, Failure:(*payload.UpdateError)(0xc0006313b0), Done:575, Total:623,

This message did appear over and over again.

Problem Summary

etcd started successfully, but the installation stalled
API server was running
VMs started but never joined the cluster
No OVN or Multus pods ever started
The installation was stuck in a loop

Troubleshooting

Troubleshooting was not straightforward. The cluster-version-operator logs provided the only clue. However, the VMs were already running, so we could log in to them—and there it was, the reason for the stalled installation.

Connect to a VM

To connect to a VM, use the virtctl command. This connects you to the machine as the core user:

You can download virtctl from the OpenShift Downloads page in the OpenShift web console.

You need the SSH key for the core user.

virtctl ssh -n clusters-my-hosted-cluster core@vmi/my-node-pool-dsj7z-sss8w (1)

1	Replace my-hosted-cluster with the name of your hosted cluster and my-node-pool-dsj7z-sss8w with the name of your VM.

Verify the Problem

Once logged in, check the journalctl -xf output. In case of a proxy issue, you’ll see an error like this:

> Dec 10 06:23:09 my-node-pool2-gwvjh-rwtx8 sh[2143]: time="2025-12-10T06:23:09Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:500704de3ef374e61417cc14eda99585450c317d72f454dda0dadd5dda1ba57a: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 3.209.93.201:443: i/o timeout"

So we have a timeout when trying to pull the image from the container registry. This is a classic proxy issue. When you try curl or manual pulls you will get the same message.

Configuring the Proxy for the Hosted Control Plane

The big question is: How do we configure the proxy for the Hosted Control Plane?

This is not well documented yet—in fact, it’s barely documented at all.

The Hosted Control Plane is managed through a custom resource called HostedCluster, and that’s exactly where we configure the proxy. The upstream documentation at HyperShift Documentation explains that you can add a configuration section to the resource. Let’s do that:

Update the HostedCluster resource with the following configuration:

spec:
configuration:
proxy:
httpProxy: http://proxy.example.com:8080 (1)
httpsProxy: https://proxy.example.com:8080 (2)
noProxy: .cluster.local,.svc,10.128.0.0/14,127.0.0.1,localhost (3)
trustedCA:
name: user-ca-bundle (4)

1	HTTP proxy URL
2	HTTPS proxy URL
3	Comma-separated list of domains, IPs, or CIDRs to exclude from proxy routing
4	OPTIONAL: Name of a ConfigMap containing a custom CA certificate bundle

If you’re using a custom CA, create the ConfigMap beforehand or alongside the HostedCluster resource. The ConfigMap must contain a key named ca-bundle.crt with your CA certificate(s).

But wait there is more…

If you are using such a proxy, which is injecting it’s own certificate, then you probably saw the following: The Release Image is not available when you try to deploy a new Hosted Control Plane. The drop down in the UI is empty and you can’t select a release image.

This happens because the Pod that is trying to fetch the image is not able to connect to Github.

You can test this with the following commands:

oc rsh cluster-image-set-controller-XXXXX -n multicluster-engine (1)
curl -v https://github.com/stolostron/acm-hive-openshift-releases.git/info/refs?service=git-upload-pack (2)

1	Replace XXXXX with the name of the Pod cluster-image-set-controller in the multicluster-engine namespace
2	Try to execute the curl command to see if you can connect to Github

If this command fails with a certificate error, then you need to add the certificate to the Pod/Deployment. A Configmap called trusted-ca-bundle should already exist in the multicluster-engine namespace. If not, it must be created with the certificate chain of your Proxy (key: ca-bundle.crt)

The following command will add the ConfigMap to the deployment.

oc -n multicluster-engine set volume deployment/cluster-image-set-controller --add --type configmap --configmap-name trusted-ca-bundle --name trusted-ca-bundle --mount-path /etc/pki/tls/certs/ --overwrite

Wait a couple of minutes after the Pods has been restarted. It will try to download the release images from Github again.

You can check the logs of the Pod to see if the download was successful.

oc logs cluster-image-set-controller-XXXXX -n multicluster-engine (1)

1	Replace XXXXX with the name of the Pod cluster-image-set-controller in the multicluster-engine namespace

That should do it, you should now be able to select a release image and deploy a new Hosted Control Plane.

Figure 1. Drow Down with Release Images

Summary

That’s actually everything you need for proxy configuration with Hosted Control Planes. Hopefully, the official OpenShift documentation will be updated soon to include this information. red Hat created two tickets to track the progress of this:

The Guide to OpenBao - Standalone Installation - Part 2

Thu, 12 Feb 2026 00:00:00 +0000

In the previous article, we introduced OpenBao and its core concepts. Now it is time to get our hands dirty with a standalone installation. This approach is useful for testing, development environments, edge deployments, or scenarios where Kubernetes is not available.

Introduction

While OpenBao shines in Kubernetes environments, understanding the standalone installation helps you grasp the fundamentals. This knowledge is valuable whether you are:

Learning OpenBao before deploying to production
Running OpenBao outside of Kubernetes (edge, legacy systems)
Debugging issues in containerized deployments
Setting up a development environment

Installation Methods Overview

OpenBao can be installed through multiple methods:

Method

Best For

Complexity

Package managers (apt, dnf, brew)

Production Linux/macOS systems

Low

Container images (Podman/Docker)

Quick testing, isolated environments

Low

Binary download

Air-gapped environments

Low

Source compilation

Custom builds, development

Medium

For this article, we will focus on macOS for local testing with binary and container image (using Podman) and Red Hat Enterprise Linux to set up an example production-ready server. The deployment on the Kubernetes environment will be discussed in the next article.

Method 1: Package Manager Installation

RHEL/Fedora/CentOS

On RHEL or CentOS, before you can install OpenBao you need to install the EPEL repository. To do so, you first need to enable the Code Ready Repository. The following commands will do the trick. Be sure that your system is registered, in case of RHEL.

If you get the error "Repositories disabled by configuration." you need to tell the subscription manager that you want to manage the repositories. This can be done permanently or temporarily. You can use the command: sudo subscription-manager config --rhsm.manage_repos=1 to do so.

RHEL

Enable Code Ready Repository

sudo subscription-manager repos --enable codeready-builder-for-rhel-9-$(arch)-rpms

Install EPEL Repository

sudo dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm

Install OpenBao
```
sudo dnf install -y openbao
```

CentOS

Enable Code Ready Repository on CentOS

sudo dnf config-manager --set-enabled crb

Install EPEL Repository

sudo dnf install epel-release epel-next-release

Install OpenBao
```
sudo dnf install -y openbao
```

macOS (Homebrew)

Install OpenBao
```
brew install openbao
```

Verify Installation

After installation, verify OpenBao is available:

bao version
# Output:
OpenBao v2.5.0 (bcbb6036ec2b747bceb98c7706ce9b974faa1b23), built 2026-02-04T15:57:17Z (cgo)

The OpenBao CLI command is bao, not vault. This distinguishes it from HashiCorp Vault.

Start Development Server

To start a development environment, you can use the following command.

This is NOT suitable for production, it is just a test to evaluate the basic concepts of OpenBao.

bao server -dev -dev-root-token-id="dev-only-token"

This will start the server. The UI is accessible at http://localhost:8200 where you can login using the root token dev-only-token.

Method 2: Container Image

For quick testing or isolated environments, container images are ideal. Luckily, OpenBao offers several types of containers suitable for any environment. We will use the image hosted on quay.io, which is based on RHEL UBI and can be found at: quay.io/openbao/openbao-ubi

Using Podman

The following command will fetch the image and start the container.

This is NOT suitable for production, it is just a test to evaluate the basic concepts of OpenBao.

# Run in dev mode (for testing only!)
podman run --rm -d \
--name openbao-dev \
-p 8200:8200 \
-e 'BAO_DEV_ROOT_TOKEN_ID=dev-only-token' \
-e 'BAO_DEV_LISTEN_ADDRESS=0.0.0.0:8200' \
quay.io/openbao/openbao-ubi:latest server -dev

# Verify it is running
podman logs -f openbao-dev

This will start the server. The UI is accessible at http://localhost:8200 where you can login using the root token dev-only-token.

If you prefer to use Docker, simply replace podman with docker in the commands.

Dev Mode: Quick Start for Testing

Dev mode is the fastest way to start using OpenBao for learning and testing.

The characteristics in this mode are:

In-memory storage (data lost on restart)
Automatically initialized and unsealed
Root token printed to stdout
TLS disabled
Single server (no HA)

Let’s create an example secret and try to retrieve it again.

Authenticate against OpenBao

export VAULT_TOKEN="dev-only-token"
export BAO_ADDR='http://127.0.0.1:8200'

Create Secret

curl \
--header "X-Vault-Token: $VAULT_TOKEN" \
--header "Content-Type: application/json" \
--request POST \
--data '{"data": {"password": "OpenBao123"}}' \
$BAO_ADDR/v1/secret/data/my-secret-password &&
echo "Secret written successfully."

Retrieve Secret

curl --header "X-Vault-Token: $VAULT_TOKEN" \
$BAO_ADDR/v1/secret/data/my-secret-password | jq '.data.data'

You should see the password that was created before:

{
"password": "OpenBao123"
}

Check Status of OpenBao Server

bao status

This will give you the status of your running OpenBao instance.

Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 2.5.0
Build Date 2026-02-04T15:57:17Z
Storage Type inmem
Cluster Name vault-cluster-421b2431
Cluster ID 6d42dcd2-e399-211e-999a-49b1874cc8ce
HA Enabled false

Hardening your System

OpenBao does a good job to secure your secrets, however, memory paging (or swap) can undermine the protection. Your OS should either have swap disabled completely or encrypt the swap space.

As I am testing on macOS, the swap space is encrypted out of the box.

However, OpenBao has documented what must be done for various operating systems at Post-installation hardening

Production Standalone Setup

For a proper standalone installation, follow these steps:

Step 1: Create Configuration File

Create /etc/openbao.d/openbao.hcl:

# Full configuration for standalone OpenBao server
# Cluster name for identification
cluster_name = "openbao-standalone"
# Storage backend using integrated Raft
storage "raft" {
path = "/var/lib/openbao/data"
node_id = "node1"
}
# HTTP listener (for internal communication)
listener "tcp" {
address = "0.0.0.0:8200"
cluster_address = "0.0.0.0:8201"
tls_disable = false
tls_cert_file = "/etc/openbao.d/tls/tls.crt"
tls_key_file = "/etc/openbao.d/tls/tls.key"
}
# API address for clients
api_addr = "https://openbao.example.com:8200"
# Cluster address for raft communication
cluster_addr = "https://openbao.example.com:8201"
# UI enabled
ui = true
# Logging
log_level = "info"
log_file = "/var/log/openbao/openbao.log"
# Disable memory locking (enable in production if possible)
disable_mlock = true
# Telemetry (optional)
telemetry {
prometheus_retention_time = "30s"
disable_hostname = true
}

Step 2: Generate TLS Certificates

For production, you should use proper certificates. For testing, create self-signed ones:

This is only for testing purposes. In production, you should use proper certificates.

First, prepare the TLS directory if it does not exist
```
sudo mkdir -p /etc/openbao.d/tls
```

Then create a configuration file for the TLS certificates:

# Create TLS config
cat <<EOF > openbao.cnf
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no
[req_distinguished_name]
CN = openbao.example.com
[v3_req]
subjectAltName = @alt_names
[alt_names]
DNS.1 = openbao.example.com
IP.1 = 127.0.0.1 (1)
EOF

1	The IP address of the server. In this case we are using localhost, but you can add your IPs here.

Then generate the certificate:

# Generate private key
sudo openssl genrsa -out /etc/openbao.d/tls/tls.key 4096
# Generate certificate signing request
sudo openssl req -new -key /etc/openbao.d/tls/tls.key -out /etc/openbao.d/tls/tls.csr -subj "/CN=openbao.example.com"
# Generate self-signed certificate
sudo openssl x509 -req -days 365 -in /etc/openbao.d/tls/tls.csr -signkey /etc/openbao.d/tls/tls.key -out /etc/openbao.d/tls/tls.crt -extfile openbao.cnf -extensions v3_req
# Set permissions
sudo chown -R openbao:openbao /etc/openbao.d/tls
sudo chmod 600 /etc/openbao.d/tls/tls.key
sudo chmod 755 /etc/openbao.d/tls

Step 3: Create Systemd Service

Create /etc/systemd/system/openbao.service:

[Unit]
Description=OpenBao Secret Management
Documentation=https://openbao.org/docs
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/openbao.d/openbao.hcl
[Service]
User=openbao
Group=openbao
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/bin/bao server -config=/etc/openbao.d/openbao.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
LimitNOFILE=65536
LimitMEMLOCK=infinity
[Install]
WantedBy=multi-user.target

Step 4: Start the Service

# Reload systemd
sudo systemctl daemon-reload
# Enable and start OpenBao
sudo systemctl enable openbao
sudo systemctl start openbao
# Check status
sudo systemctl status openbao
# View logs
sudo journalctl -u openbao -f

Initialize and Unseal OpenBao

After starting OpenBao for the first time, it needs to be initialized and unsealed.

Set Environment Variables

We will do the following commands as root user.

Be sure that the hostname is resolvable. In this case I am using the test domain: openbao.example.com.

# Set the OpenBao address
export BAO_ADDR='https://openbao.example.com:8200'
# If using self-signed certificates
export BAO_CACERT='/etc/openbao.d/tls/tls.crt'

Check Status

You will see that OpenBao is running but not yet initialized. This will be the next step.

bao status
Key Value
--- -----
Seal Type shamir
Initialized false (1)
Sealed true
Total Shares 0
Threshold 0
Unseal Progress 0/0
Unseal Nonce n/a
Version 2.4.4-1.el9
Build Date 2025-11-24
Storage Type file
HA Enabled false

1	Not yet initialized

Initialize OpenBao

Before we use OpenBao, we need to initialize it. This will create the unseal keys and the root token. The unseal keys are used to … well, unseal the OpenBao service. The root token is used as a master key to authenticate against the OpenBao service. This token has access to ALL secrets. Treat this root token with care.

We will initialize OpenBao with default options of 5 key shares and a threshold of 3. This means that we need 3 different unseal keys to unseal the OpenBao service.

bao operator init -key-shares=5 -key-threshold=3 -format=json > /root/openbao-init.json

Store the unseal keys and root token securely! Anyone with these can access all secrets.

The output looks like:

{
"unseal_keys_b64": [
"key1...",
"key2...",
"key3...",
"key4...",
"key5..."
],
"unseal_keys_hex": [...],
"unseal_shares": 5,
"unseal_threshold": 3,
"recovery_keys_b64": [],
"recovery_keys_hex": [],
"root_token": "sbr.xxxxxxxxxxxx"
}

Unseal OpenBao

Now everything is set up and we can use the unseal keys to unseal the OpenBao service.

You need to provide 3 (threshold) different unseal keys:

# First key
bao operator unseal
# Enter first unseal key
# Second key
bao operator unseal
# Enter second unseal key
# Third key
bao operator unseal
# Enter third unseal key

After providing enough keys:

bao status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false (1)
Total Shares 5
Threshold 3
Version 2.4.4-1.el9
Build Date 2025-11-24
Storage Type file
Cluster Name openbao-standalone
Cluster ID d07874ef-df52-e45f-1723-ade333c6d609
HA Enabled false

1	Now unsealed and ready

Now we can login with the root token to the OpenBao service.

Login with Root Token

Since we have not created any users yet, we will use the root token to authenticate.

# Enter the root token from initialization
bao login

Basic Verification

Let us verify the installation works:

List enabled secrets engines

bao secrets list

This lists the enabled secrets engines.

Path Type Accessor Description
---- ---- -------- -----------
cubbyhole/ cubbyhole cubbyhole_84c3d2a5 per-token private secret storage
identity/ identity identity_61809171 identity store
sys/ system system_034ae772 system endpoints used for control, policy and debugging

List enabled auth methods

bao auth list

This lists the enabled authentication methods.

Path Type Accessor Description Version
---- ---- -------- ----------- -------
token/ token auth_token_c7feca17 token based credentials n/a

Create a test secret
```
bao secrets enable -path=secret kv-v2
```
This enables the KV (key/value) secrets engine at the path secret.
```
bao kv put secret/test message="Hello from OpenBao"
```
This creates a test secret at the path secret/test.
```
bao kv get secret/test
```
This retrieves the test secret from the path secret/test.

That’s it for the basic verification. You can now start to use OpenBao in your production environment. Let’s see what we can do in the UI.

Visiting the UI

If you watched the previous commands closely, you will have noticed that the UI has been enabled in the configuration file:

# UI enabled
ui = true

The UI is accessible at http://openbao.example.com:8200. You can login with the root token from the initialization.

Figure 1. OpenBao Login Form

Here you will see several options to explore. For now we are interested in the Secrets Engine section.

Figure 2. OpenBao Secrets Engine

We have created the path "secret" and inside it a secret called "test". We can retrieve it now:

Figure 3. OpenBao Secret Retrieval

Security Hardening Checklist

Before using OpenBao in production you should always consider the following checklist:

Item

Recommendation

TLS

Always enable TLS with valid certificates

Root Token

Revoke root token after initial setup and create admin users instead

Unseal Keys

Distribute to different people/locations, consider auto-unseal

Network

Restrict access with firewall rules

Audit

Enable audit logging

Backups

Regular Raft snapshots

Updates

Keep OpenBao updated

What is Coming Next?

In Part 3, we will deploy OpenBao on OpenShift/Kubernetes using the official Helm chart. This provides:

High availability out of the box
Kubernetes-native management
Integration with OpenShift security features
Persistent storage via PVCs

Conclusion

You now have a working standalone OpenBao installation. This forms the foundation for understanding how OpenBao operates. While standalone mode is useful for testing and edge cases, most production deployments will use Kubernetes, which we will cover next.

Key takeaways:

OpenBao can run standalone or in containers
Dev mode is for testing only
Production requires proper TLS, initialization, and security hardening
Unseal keys must be stored securely

Resources

The Guide to OpenBao - Introduction - Part 1

Wed, 11 Feb 2026 00:00:00 +0000

I finally had some time to dig into Secret Management. For my demo environments, SealedSecrets is usually enough to quickly test something. But if you want to deploy a real application with Secret Management, you need to think of a more permanent solution.

This article is the first of a series of articles about OpenBao, a HashiCorp Vault fork. Today, we will explore what OpenBao is, why it was created, and when you should consider using it for your secret management needs. If you are familiar with HashiCorp Vault, you will find many similarities, but also some important differences that we will discuss.

Introduction

In general, sensitive information in any system should be stored in a secure way. When it comes to OpenShift or Kubernetes, this is especially true, since the secrets are stored in the etcd database. Even if etcd is encrypted at rest, anybody can decode a given base64 string which is stored in the Secret.

Base64 is not an encryption format. It is an encoding format.

For example, the string Thomas encoded as base64 is VGhvbWFzCg==. This is simply masked plain text and it is not secure to share these values, especially not on Git. To make your CI/CD pipelines or GitOps process secure, you need to think of a secure way to manage your Secrets.

This is where OpenBao comes in.

What is OpenBao?

OpenBao is an identity-based secrets and encryption management system. It is an open-source, community driven fork of HashiCorp Vault and provides secure storage, fine-grained access control, and lifecycle management for secrets such as API or SSH keys, passwords, certificates, and encryption keys.

The core OpenBao workflow consists of four stages:

Authentication: Verifying the identity of the client to determine if they are who they say they are. Once authenticated, a token is created and associated with a so-called policy.
Validation: Checking if the client has the required permissions
Authorization: Granting access based on policies that provide or deny access to certain paths and operations in OpenBao.
Access: Limiting what the client can do with the secret

At its core, OpenBao:

Stores and encrypts sensitive information at rest and in transit
Provides fine-grained access controls (ACL)
Supports dynamic secret generation
Offers a comprehensive audit trail (must be enabled)
Enables encryption as a service

The History: Why OpenBao Exists

OpenBao is a community-driven, open-source fork of HashiCorp Vault. But why was it created?

In August 2023, HashiCorp announced a significant change to the licensing of their products, including Vault. They moved from the Mozilla Public License 2.0 (MPL 2.0) to the Business Source License 1.1 (BSL 1.1).

The BSL 1.1 license includes restrictions on competitive use:

Competitors could no longer use Vault’s code to offer competing services
Cloud providers and managed service providers faced restrictions
The open-source community lost the freedom to fork and commercialize

The Fork

In response to this license change, the Linux Foundation announced the OpenBao project in December 2023. OpenBao is:

A fork of HashiCorp Vault (from version 1.14.x)
Licensed under the OSI-approved Mozilla Public License 2.0
Governed by a community-driven model
Maintained independently of HashiCorp
Continues to enable innovation without license restrictions

OpenBao vs. HashiCorp Vault: Key Differences

While OpenBao shares its heritage with Vault, there are several important differences:

Licensing

Aspect

OpenBao

HashiCorp Vault

License

MPL 2.0 (OSI-approved open source)

BSL 1.1 (with staged conversion)

Governance

Community-driven (Linux Foundation)

HashiCorp/IBM controlled

Commercial restrictions

None

Restrictions on competitive use

Enterprise Features

Community-driven additions

Paid Enterprise tier

Support

Community support

Paid support available

Branding

bao CLI, OpenBao naming

vault CLI, Vault naming

Technical Differences

Token Format - OpenBao uses shorter tokens in the format sbr.[random], while Vault uses longer tokens (hvs., hvb., hvr. prefixes followed by long random strings). Old Vault tokens are still accepted according to their TTLs, but newly issued tokens follow the OpenBao format.
Plugin Ecosystem - OpenBao comes with fewer built-in plugins by default, focusing on OSI-licensed integrations. Proprietary cloud vendor plugins have been moved to external repositories.
Storage Backend - OpenBao has simplified its storage options, primarily supporting Raft as the recommended backend.
API Compatibility - OpenBao’s API is designed to be compatible with Vault, meaning existing clients and integrations should work without modification. However, some edge cases may require updates.

Core Concepts

Before diving into installation and configuration in the next articles, it is essential to understand the fundamental concepts of OpenBao:

Secrets Engines

Secrets engines are components that store, generate, or encrypt data. OpenBao supports multiple types:

KV (Key-Value): Simple static secret storage with optional versioning
PKI: Certificate Authority for generating TLS certificates
Database: Dynamic credential generation for databases
Transit: Encryption as a Service (EaaS)
SSH: SSH key signing and OTP generation

Authentication Methods

Authentication methods verify client identity before granting access:

Kubernetes: Uses Kubernetes service account tokens. Essential for Kubernetes/OpenShift deployments.
OIDC: OpenID Connect for user authentication, like Keycloak, Okta, or Azure AD.
LDAP: Directory service authentication, like Active Directory, OpenLDAP, or Microsoft AD.
AppRole: Machine-oriented authentication for applications. Ideal for CI/CD pipelines.
Token: Direct token-based authentication. The root token is created during initialization.

Storage Backend

OpenBao encrypts all data before writing to storage. Supported backends include:

Integrated Raft (recommended): Built-in distributed storage
Consul: HashiCorp’s service discovery and KV store (less common with OpenBao)
File: Single-node deployments only
PostgreSQL/MySQL: Database-backed storage

Policies

Policies define what a client can do after authentication. They use path-based access control (deny-by-default mode):

path "secret/data/myapp/*" {
capabilities = ["read", "list"]
}
path "pki/issue/my-role" {
capabilities = ["create", "update"]
}
path "database/creds/myapp-role" {
capabilities = ["read"]
}

Tokens

Tokens are the primary authentication credential in OpenBao. After successful authentication, clients receive a token that:

Has a TTL (Time To Live)
Is associated with one or more policies
Can be renewed (if renewable)
Can create child tokens

Seal/Unseal

OpenBao starts in a sealed state where it cannot access encrypted data. The unseal process requires multiple key shares (using Shamir’s Secret Sharing) or auto-unseal mechanisms to decrypt the master key.

Leases

Most secrets in OpenBao have an associated lease - a duration after which the secret expires. This enables:

Automatic secret rotation
Revocation of compromised credentials
Audit trail of secret usage

Use Cases

Dynamic Database Credentials

Instead of storing static database passwords:

Application authenticates to OpenBao
Requests database credentials
OpenBao creates a temporary database user
Returns credentials with a short TTL
Credentials automatically expire

Benefits: No long-lived credentials, automatic rotation, per-application isolation.

Certificate Management with PKI

Instead of manually managing TLS certificates:

Configure OpenBao as an intermediate CA
Applications request certificates on demand
Short-lived certificates (hours/days instead of years)
Automatic renewal before expiration

Benefits: No certificate sprawl, automated rotation, reduced attack surface.

Encryption as a Service

Instead of implementing encryption in each application:

Applications send plaintext to OpenBao’s Transit engine
OpenBao encrypts with managed keys
Applications store ciphertext
Decryption requests go through OpenBao

Benefits: Centralized key management, separation of duties, key rotation without re-encryption.

Kubernetes Secret Injection

Instead of storing secrets in Kubernetes Secrets:

Deploy OpenBao with Kubernetes auth
Configure injector or External Secrets Operator
Pods automatically receive secrets at startup
Secrets never stored in etcd

Benefits: Secrets not in cluster, dynamic injection, centralized management.

When to Use OpenBao

OpenBao is an excellent choice when you need:

Centralized Secret Management - If you have secrets scattered across configuration files, environment variables, and various secret stores, OpenBao provides a single source of truth.
Dynamic Secrets - For use cases where you need short-lived, automatically rotated credentials (e.g., database passwords), OpenBao can generate them on-demand.
Encryption as a Service - If applications need encryption capabilities without managing encryption keys, OpenBao’s Transit engine provides this functionality.
Certificate Management - OpenBao’s PKI engine can act as a Certificate Authority, issuing and managing TLS certificates.
Compliance Requirements - For environments with strict audit requirements, OpenBao provides comprehensive audit logging of all secret access.

When NOT to Use OpenBao

OpenBao might be overkill for:

Simple, Static Secrets - If you only have a few static secrets that rarely change, simpler solutions like Sealed Secrets or External Secrets Operator with a basic backend might suffice.
Small Teams with Limited Resources - OpenBao requires operational expertise to maintain. If you do not have the resources to operate it properly, consider managed alternatives.
Single-Application Deployments - If you have a single application with minimal secret requirements, the complexity of OpenBao may not be justified.

Architecture Overview

A typical OpenBao deployment consists of:

Figure 1. Architecture Overview

Multiple Nodes: For high availability (HA), OpenBao runs as a cluster
Raft Consensus: Leader election and data replication
Persistent Storage: Encrypted data stored on persistent volumes
Load Balancer: Distributes client requests

Conclusion

OpenBao provides a powerful, open-source solution for secret management that addresses the challenges of modern cloud-native environments. Its fork from HashiCorp Vault means it benefits from years of development while remaining truly open source.

In the next article, we will start with a standalone installation to understand the fundamentals before moving to Kubernetes deployments.

Resources

Cert-Manager Policy Approver in OpenShift

Tue, 03 Jun 2025 00:00:00 +0000

One of the most commonly deployed operators in OpenShift environments is the Cert-Manager Operator. It automates the management of TLS certificates for applications running within the cluster, including their issuance and renewal.

The tool supports a variety of certificate issuers by default, including ACME, Vault, and self-signed certificates. Whenever a certificate is needed, Cert-Manager will automatically create a CertificateRequest resource that contains the details of the certificate. This resource is then processed by the appropriate issuer to generate the actual TLS certificate. The approval process in this case is usually fully automated, meaning that the certificate is issued without any manual intervention.

But what if you want to have more control? What if certificate issuance must follow strict organizational policies, such as requiring a specifc country code or organization name? This is where the CertificateRequestPolicy resource, a resource provided by the Approver Policy, comes into play.

This article walks through configuring the Cert-Manager Approver Policy in OpenShift to enforce granular policies on certificate requests.

Prerequisites

Before you begin, ensure you have the following:

OpenShift 4.16 or higher with cluster-admin access
Cert-Manager Operator installed

The installation of Cert-Manager itself is discussed in the article: Managing Certificates using GitOps approach.

The Cert-Manager Operator does not currently support the Approver Policy by default. You need to install the Approver Policy manually using a Helm Chart. There is a feature request to include the Approver Policy in the Cert-Manager Operator in the future.

Adding Approver Policy Chart as a dependency

Source: Cert-Manager Deployment

The above Chart contains the necessary resources to deploy the Cert-Manager itself and Cert-Manager Approver Policy in OpenShift. To deploy the Approver Policy alongside Cert-Manager, add it as a dependency in your Chart.yaml:

[...]
- name: cert-manager-approver-policy
version: v0.19.0
repository: https://charts.jetstack.io
[...]

This is the official Helm Chart for the Cert-Manager Approver Policy tool provided by Jetstack. The version used in this example is v0.19.0, but you can use a newer version if available.

Configuration of the Helm Chart

The initial values.yaml file was extended to:

include the configuration for the Approver Policy Chart
the configuration for a CertificateRequestPolicy object
with some specific modifications to the CertManager resource itself

Disabling Cert-Manager’s Auto-Approver

The first step we need to do is to disable the auto-approver of the Cert-Manager. If this is not done, there will be a race condition between the auto-approver and the Approver Policy, which will lead to unexpected results. These changes are done by:

cert-manager:
certManager:
enable_patch: true (1)
unsupportedConfigOverrides:
controller:
args:
- '--controllers=*,-certificaterequests-approver' (2)

1	This enables the patching of the CertManager resource, which tells the chart to overwrite (patch) the automatically generated CertManager resource with the custom configuration.
2	This disables the auto-approver Controller of the Cert-Manager.

As the name suggests, this is currently an unsupported configuration, but it is necessary to disable the auto-approver for the Cert-Manager. In the future versions of the Cert-Manager, this might change, and the auto-approver might be supported out of the box.

In my case, the full CertManager resource looks like this:

apiVersion: operator.openshift.io/v1alpha1
kind: CertManager
metadata:
annotations:
name: cluster
spec:
controllerConfig:
overrideArgs: (1)
- '--dns01-recursive-nameservers-only'
- '--dns01-recursive-nameservers=ns-362.awsdns-45.com:53,ns-930.awsdns-52.net:53'
logLevel: Normal
managementState: Managed
operatorLogLevel: Normal
unsupportedConfigOverrides: (2)
controller:
args:
- '--controllers=*,-certificaterequests-approver'

1	Settings to support AWS Nameservers for DNS01 challenges.
2	This disables the auto-approver Controller of the Cert-Manager.

Configuration of the Approver Policy

The second step is to configure the Approver Policy chart. This chart will deploy the necessary resources, most importantly a Deployment that will start the Pods which will process the CertificateRequestPolicy resources later on.

My configuration for that chart looks like this (some default values are omitted for brevity):

cert-manager-approver-policy:
crds: (1)
# This option decides if the CRDs should be installed
# as part of the Helm installation.
enabled: true
# This option makes it so that the "helm.sh/resource-policy": keep
# annotation is added to the CRD. This will prevent Helm from uninstalling
# the CRD when the Helm release is uninstalled.
# WARNING: when the CRDs are removed, all cert-manager-approver-policy custom resources
# (CertificateRequestPolicy) will be removed too by the garbage collector.
keep: true
# Number of replicas of approver-policy to run.
replicaCount: 1 (2)
image: (3)
# Target image repository.
repository: quay.io/jetstack/cert-manager-approver-policy
# Kubernetes imagePullPolicy on Deployment.
pullPolicy: IfNotPresent
tag: v0.19.0
app:
# List of signer names that approver-policy will be given permission to
# approve and deny. CertificateRequests referencing these signer names can be
# processed by approver-policy. Defaults to an empty array, allowing approval
# for all signers.
approveSignerNames: (4)
- 'issuers.cert-manager.io/*'
- 'clusterissuers.cert-manager.io/*'

1	This enables the installation of the CRDs that are required for the Approver Policy.
2	The number of replicas of the Approver Policy Deployment. In most cases, one replica is enough.
3	The image configuration for the Approver Policy. The image is pulled from the Jetstack Quay.io repository.
4	The list of signer names that the Approver Policy will be allowed to approve. In this case, it is configured to allow all issuers and clusterissuers.

The approveSignerNames are, if configured, an important setting, especially if you want to add custom (cluster)issuers. In such a case, you need to add the name of the custom issuer to this list. Otherwise the Approver Policy will not be able to approve the CertificateRequests for that issuer.

Creating a CertificateRequestPolicy and Role(Binding)

The final step in our configuration is to define a CertificateRequestPolicy resource that will define the policy for the certificate requests. This resource will be processed by the Approver Policy and will determine if a certificate request is approved or denied based on the defined criteria.

The following example shows a CertificateRequestPolicy that will:

Allow certificate requests with any common name, DNS names, IP addresses, URIs, and email addresses.
Require DNS names to be set.
Require the subject to contain a specific organization (MyOrganization) and country code (AT).
Allow usages for server auth and client auth.
Set constraints for the certificate duration (1h-24h) and private key algorithm (RSA) and size (2048-4096).
Allow all issuers by using an empty selector.

role: cert-manager-policy:global-approver (1)
serviceAccount: cert-manager (2)
cert_manager_Namespace: cert-manager (3)
policies:
- name: my-approver-policy
enabled: true
allowed:
commonName:
required: false
value: "*"
validations: []
dnsNames: (4)
required: true
values:
- "*"
validations: []
ipAddresses:
required: false
values: ["*"]
validations: []
uris:
required: false
values:
- "*"
validations: []
emailAddresses:
required: false
values:
- "*"
validations: []
# isCA: false
subject:
organizations: (5)
required: true
values:
- "MyOrganization"
validations:
- rule: self.matches("MyOrganization")
message: Organization must be MyOrganization
countries:
required: true
values:
- AT
validations:
- rule: self.matches("AT")
message: Country code must be AT
usages:
- "server auth"
- "client auth"
constraints: (6)
minDuration: 1h
maxDuration: 24h
privateKey:
algorithm: RSA
minSize: 2048
maxSize: 4096
selector:
issuerRef: {} (7)

1	The role that is used to approve the certificate requests. This role must be created in the OpenShift cluster and must have the necessary permissions to approve certificate requests.
2	The service account that is used by the Approver Policy to process the certificate requests. This service account must have the necessary permissions to access the CertificateRequest resources.
3	The namespace where the Cert-Manager is deployed. This is usually the `cert-manager` namespace, but you can change it if you have a different namespace.
4	The DNS names are required to be set for the certificate request.
5	The subject must contain the organization MyOrganization and the country code AT.
6	The constraints for the certificate request, such as the minimum and maximum duration, private key algorithm, and size.
7	The selector is empty, which means that the policy applies to all issuers. If you want to limit the policy to specific issuers, you can specify the issuerRef here.

Rendered CertificateRequestPolicy and Role(Binding)

The above configuration will create a CertificateRequestPolicy resource that looks like this:

Expand me...

---
# Source: cert-manager/templates/ClusterRole-Approver-Policy-approving.yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: "cert-manager-policy:global-approver"
labels:
helm.sh/chart: cert-manager-2.0.0
app.kubernetes.io/name: cert-manager
app.kubernetes.io/instance: release-name
app.kubernetes.io/managed-by: Helm
rules:
- verbs:
- use
apiGroups:
- policy.cert-manager.io
resources:
- certificaterequestpolicies
resourceNames:
- my-approver-policy
---
# Source: cert-manager/charts/cert-manager-approver-policy/templates/clusterrolebinding.yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
labels:
app.kubernetes.io/name: cert-manager-approver-policy
helm.sh/chart: cert-manager-approver-policy-v0.19.0
app.kubernetes.io/instance: release-name
app.kubernetes.io/version: "v0.19.0"
app.kubernetes.io/managed-by: Helm
name: cert-manager-approver-policy
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cert-manager-approver-policy
subjects:
- kind: ServiceAccount
name: cert-manager-approver-policy
namespace: default
---
# Source: cert-manager/templates/CertificateRequestPolicy.yaml
apiVersion: policy.cert-manager.io/v1alpha1
kind: CertificateRequestPolicy
metadata:
name: my-approver-policy
annotations:
argocd.argoproj.io/sync-wave: "10"
labels:
helm.sh/chart: cert-manager-2.0.0
app.kubernetes.io/name: cert-manager
app.kubernetes.io/instance: release-name
app.kubernetes.io/managed-by: Helm
spec:
allowed:
commonName:
required: false
value: "*"
validations: []
dnsNames:
required: true
values:
- "*"
validations: []
emailAddresses:
required: false
values:
- "*"
validations: []
ipAddresses:
required: false
values:
- "*"
validations: []
uris:
required: false
values:
- "*"
validations: []
isCA: false
subject:
organizations:
required: true
values:
- "MyOrganization"
validations:
- rule: "self.matches(\"MyOrganization\")"
message: "Organization must be MyOrganization"
countries:
required: true
values:
- "AT"
validations:
- rule: "self.matches(\"AT\")"
message: "Country code must be AT"
organizationalUnits:
required: false
values: ["*"]
validations: []
localities:
required: false
values: ["*"]
validations: []
provinces:
required: false
values: ["*"]
validations: []
streetAddresses:
required: false
values: ["*"]
validations: []
postalCodes:
required: false
values: ["*"]
validations: []
serialNumber:
required: false
value: "*"
validations: []
usages:
- "server auth"
- "client auth"
constraints:
minDuration: 1h
maxDuration: 24h
privateKey:
algorithm: RSA
minSize: 2048
maxSize: 4096
selector:
issuerRef: {}

One important note is about the ClusterRole and ClusterRoleBinding that are created by the Helm Chart. The role looks like the following and is required to allow the Approver Policy to approve certificate requests. This small bit, puzzled me for a while:

rules:
- verbs:
- use
apiGroups:
- policy.cert-manager.io
resources:
- certificaterequestpolicies
resourceNames:
- my-approver-policy

With the above configuration we are good to go. The Helm Chart can be deployed to the OpenShift cluster (for example, using Argo CD), and the CertificateRequestPolicy will be created automatically.

A new Pod is running in the cert-manager namespace:

❯ oc get pods -n cert-manager | grep approver
NAME READY STATUS RESTARTS AGE
cert-manager-approver-policy-xxxxx 1/1 Running 0 XXm (1)

1	The Pod `cert-manager-approver-policy-xxxxx` is the Pod that is responsible for processing the CertificateRequestPolicy resources.

Testing the Policy

Test 1 - Valid Certificate Request

Now it is time to test the policy. We need to create a Certificate and monitor the output of our approval pod.

As a reminder, the policy we created requires the following:

The subject must contain the organization MyOrganization
The subject must contain the country code AT.
The keysize must be at least 2048 bits. (max 4096 bits)
The duration must be between 1 hour and 24 hours.
The usage must be server auth or client auth.

Let’s create this example Certificate in the myproject namespace:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: test-certificate1
namespace: myproject
spec:
dnsNames:
- test1.apps.ocp.aws.ispworld.at
duration: 24h
issuerRef:
kind: ClusterIssuer
name: letsencrypt-prod
privateKey:
algorithm: RSA
encoding: PKCS1
rotationPolicy: Always
secretName: test1
subject:
organizations:
- MyOrganization
countries:
- AT
usages:
- server auth

In the log of the Approver Policy Pod, we should see the following output:

time=2025-06-03T16:07:58.656Z level=DEBUG+3 msg="Approved by CertificateRequestPolicy: \"my-approver-policy\"" logger=controller-manager/events type=Normal object="{Kind:CertificateRequest Namespace:myproject Name:test-certificate1-1 [...]}" reason=Approved

This indicates that the CertificateRequest was approved by the Approver Policy. The policy was able to validate the subject, keysize, duration, and usage of the certificate request and approved it accordingly. The certificate has been created successfully in the myproject namespace, and the secret test1 contains the TLS certificate and private key.

❯ oc get secret test1 -n myproject -o yaml
apiVersion: v1
data:
tls.crt: ...
tls.key: ...
kind: Secret
metadata:
labels:
controller.cert-manager.io/fao: "true"
name: test1
namespace: myproject
type: kubernetes.io/tls

Test 2 - Invalid Certificate Request

That was easy, but what happens if we create a CertificateRequest that does not meet the policy requirements? Let’s try to create a Certificate without the required organization or a wrong country code:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: test-certificate2
namespace: myproject
spec:
dnsNames:
- test2.apps.ocp.aws.ispworld.at
duration: 24h
issuerRef:
kind: ClusterIssuer
name: letsencrypt-prod
privateKey:
algorithm: RSA
encoding: PKCS1
rotationPolicy: Always
secretName: test2
subject: (1)
countries:
- XX
usages:
- server auth

1	The subject does not contain the required organization MyOrganization and the country code is set to XX, which is not allowed by the policy.

This will lead to the following error in the log of the Approver Policy Pod:

time=2025-06-03T16:16:02.233Z level=DEBUG+3 msg="No policy approved this request: [my-approver-policy: [spec.allowed.subject.organizations.required: Required value: true, spec.allowed.subject.countries.values: Invalid value: []string{\"XX\"}: AT, spec.allowed.subject.countries.validations[0]: Invalid value: \"XX\": Country code must be AT]]" logger=controller-manager/events type=Warning object="{Kind:CertificateRequest Namespace:myproject Name:test-certificate2-1 ..." reason=Denied

It complains that the subject does not meet the policy requirements and therefore the CertificateRequest was denied.

Ok we have a policy, but whats next?

The above example shows how the Cert-Manager Approver Policy can be configured and deployed, even if it is not yet supported by the Cert-Manager Operator. However, we only scratched the surface of what is possible with the Approver Policy. You can create more complex policies that include additional validations, such as checking the validity of the DNS names, IP addresses, or URIs. You can also create policies that require specific email addresses or organizational units in the subject.

You can even create fine-grained policies that apply to specific issuers or namespaces by using the selector field in the CertificateRequestPolicy resource. This allows you to create policies that are tailored to your specific requirements and use cases.

The best references can be found here:

Official documentation: Cert-Manager Approver Policy
Example Policies: Cert-Manager Approver Policy Examples

Conclusion

The Cert-Manager Approver Policy is a powerful tool that allows you to implement custom policies for certificate requests in OpenShift. It provides a way to control the issuance of TLS certificates based on specific criteria, such as the subject, key size, duration, and usage of the certificate. While not yet officially supported by the Cert-Manager Operator, it can be easily integrated into your OpenShift environment using a Helm Chart. Future support is currently being discussed, and it is expected that the Approver Policy will be included in the Cert-Manager Operator in the future.

Single log out from Keycloak and OpenShift

Thu, 22 May 2025 00:00:00 +0000

The following 1-minute article is a follow-up to my previous article about how to use Keycloak as an authentication provider for OpenShift. In this article, I will show you how to configure Keycloak and OpenShift for Single Log Out (SLO). This means that when you log out from Keycloak, you will also be logged out from OpenShift automatically. This requires some additional configuration in Keycloak and OpenShift, but it is not too complicated.

Prerequisites

The following prerequisites are required to follow this article:

OpenShift 4.16 or higher with cluster-admin privileges
Keycloak installed and configured, as described in the Step by Step - Using Keycloak Authentication in OpenShift

Keycloak Configuration

Configure the logout URL in Keycloak. This is done by adding a new Valid post logout redirect URIs to the Keycloak client configuration. In this case, we want to call the OpenShift logout URL. Which is: https://oauth-openshift.apps.<your-cluster-name>/logout

This is done in the client settings:

OpenShift Configuration

Now we need to configure OpenShift to use the logout URL. This is done by adding a new Post Logout Redirect URI to the OpenShift Console configuration.

Modify the existing Console resource named cluster and add the logoutRedirect parameter to the authentication section. The important pieces of the logout URL are the client_id and the post_logout_redirect_uri. The client ID must be the same as the Keycloak client ID, and the post logout redirect URI must be the OpenShift logout URL. (Also change the <keycloakURL> and <realm> to your Keycloak URL and realm name.)

Actually, instead of client_id, the id_token_hint should be used. But OpenShift does not store the token, so we are using the client ID instead.

apiVersion: config.openshift.io/v1
kind: Console
metadata:
name: cluster
spec:
authentication:
logoutRedirect: 'https://<keycloakURL>/realms/<realm>/protocol/openid-connect/logout?client_id=<realm>&post_logout_redirect_uri=https://console-openshift-console.apps.ocp.aws.ispworld.at' (1)

1	The logout URL for OpenShift. This is the URL that will be called when you log out from Keycloak. The URL must contain the client ID and the post logout redirect URI (console of OpenShift).

Wait a few moments until the OpenShift Console Operator applies the new configuration.

Testing the configuration

Now that the configuration is done, we can test the Single Log Out functionality.

We are logged into OpenShift already as user testuser
Now we log out from OpenShift using the Log out button in the upper right corner.
This will redirect us to the Keycloak logout page where we need to confirm the logout.
This will log us out from Keycloak and redirect us back to the OpenShift logout page.

This is it. You are now logged out from both Keycloak and OpenShift. You can also check the Sessions in Keycloak to see that the session for the user is terminated.

Conclusion

As promised, this was a short article about how to configure Keycloak and OpenShift for Single Log Out. This is a beneficial feature if you want to ensure that users are logged out from all applications when they log out from Keycloak. It is also a good security practice to ensure that users are logged out from all applications when they are done using them.

Step by Step - Using Keycloak Authentication in OpenShift

Sat, 17 May 2025 00:00:00 +0000

I was recently asked about how to use Keycloak as an authentication provider for OpenShift. How to install Keycloak using the Operator and how to configure Keycloak and OpenShift so that users can log in to OpenShift using OpenID. I have to admit that the exact steps are not easy to find, so I decided to write a blog post about it, describing each step in detail. This time I will not use GitOps, but the OpenShift and Keycloak Web Console to show the steps, because before we put it into GitOps, we need to understand what is actually happening.

This article tries to explain every step required so that a user can authenticate to OpenShift using Keycloak as an Identity Provider (IDP) and that Groups from Keycloak are imported into OpenShift. This article does not cover a production grade installation of Keycloak, but only a test installation, so you can see how it works. For production, you might want to consider a proper database (maybe external, but at least with a backup), high availability, etc.).

Prerequisites

The following prerequisites are required to follow this article:

OpenShift 4.16 or higher with cluster-admin privileges

Installing Keycloak Operator

The very first step is to install the Keycloak Operator. This can be done using the OpenShift Web Console or the CLI or GitOps. I will show the steps using the OpenShift Web Console.

Installing the Keycloak Operator via GitOps can be seen in this GitHub repository. By the time you read this article, this repository will install and configure a Keycloak instance automatically using a local EXAMPLE Postgres database. It does not yet import any Realm or additional configuration, but I will try to add this in the future when time allows. Oh, and it is just a demo and not production ready, so please do not use it in production.

Log in to the OpenShift Web Console as a user with cluster-admin privileges.
In the OpenShift Web Console, navigate to the Operators → OperatorHub.
In the OperatorHub, search for Red Hat build of Keycloak and select the Operator from the list.
Install the latest version of the Keycloak Operator.

You can keep the default settings for the installation, but I recommend using a specific namespace. In this example, I will use the namespace keycloak for the installation.
Press the Install button to install the Keycloak Operator. After a few minutes, the Keycloak Operator should be installed and running in the OpenShift cluster. You can check the status of the Operator in the Installed Operators view.

Create a local example Postgres Database

Demo only! This is not production ready and should not be used in production.

As a next step, we need to create a local Postgres database for Keycloak. Secrets are used to store the database credentials, and a simple StatefulSet is used to create the database. The database will be created in the same namespace as the Keycloak Operator.

First, let’s create a Secret for the database credentials. In OpenShift select the keycloak namespace and navigate to the Secrets view. Create a new Secret with the following test configuration:
```
kind: Secret
apiVersion: v1
metadata:
name: keycloak-db-secret
namespace: keycloak
stringData:
password: thisisonly4testingNOT4prod (1)
username: testuser
type: Opaque
```
1 The password for the database. This is just a test password and should not be used in production.

Next, we need to create a StatefulSet for the Postgres database, again in the keycloak namespace. Again, as I cannot mention that enough, this is just a test configuration and should not be used in production.

apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgresql-db
namespace: keycloak
spec:
serviceName: postgresql-db-service
selector:
matchLabels:
app: postgresql-db
replicas: 1 (1)
template:
metadata:
labels:
app: postgresql-db
spec:
containers:
- name: postgresql-db
image: postgres:15 (2)
volumeMounts:
- mountPath: /data
name: psql
env: (3)
- name: POSTGRES_USER
value: testuser
- name: POSTGRES_PASSWORD
value: thisisonly4testingNOT4prod
- name: PGDATA
value: /data/pgdata
- name: POSTGRES_DB
value: keycloak
volumeClaimTemplates: (4)
- metadata:
name: psql
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "gp3-csi"
resources:
requests:
storage: 10Gi

1	The number of replicas for the database. In this example, we are using a single replica.
2	The image for the database, postgres version 15 in this example.
3	The environment variables for the database. The username and password are the same as in the Secret we created before, and clear text
4	The volume for the database. In this example, the StatefulSet uses a volume claim template to create a volume with the size of 10 GB for the database. The volume is created using the `gp3-csi` storage class. You can use any other storage class that is available in your OpenShift cluster or even remove this line and use the default class instead.

Finally, we need to create a Service for the database so that the Keycloak Operator can access the database. Again, in the keycloak namespace.
```
apiVersion: v1
kind: Service
metadata:
name: postgres-db
namespace: keycloak
spec:
selector:
app: postgresql-db (1)
type: LoadBalancer
ports:
- port: 5432
targetPort: 5432
```
1 The selector for the Service. This must match the label of the StatefulSet we created before.

Creating a Keycloak Instance

Now that the Keycloak Operator is installed and our example database is running, we can create a Keycloak instance.

In the OpenShift Web Console, navigate to the Installed Operators view and select the Keycloak Operator. (Maybe you need to select the keycloak namespace first.)
In the Keycloak Operator view, create a new instance of Keycloak and switch to the YAML view.

The fun part here is that the YAML example the Operator provides is actually wrong and does not work. Something that kept me busy for a while.

Replace the YAML with the following configuration:

apiVersion: k8s.keycloak.org/v2alpha1
kind: Keycloak
metadata:
name: keycloak (1)
namespace: keycloak
labels:
app: sso
spec:
db: (2)
host: postgres-db
passwordSecret:
key: password
name: keycloak-db-secret
usernameSecret:
key: username
name: keycloak-db-secret
vendor: postgres
hostname: (3)
hostname: sso.apps.ocp.aws.ispworld.at
http: (4)
tlsSecret: keycloak-certificate
instances: 1 (5)

1	The name and the namespace of the Keycloak instance.
2	The database configuration. In this example, we are using a local Postgres database. You can also use an external database, but you need to configure the connection string accordingly.
3	Hostname of our Keycloak instance.
4	The TLS secret for the Keycloak instance. You need to create a TLS secret with the certificate and key for the hostname. This is where the example YAML is wrong. It tries to put tlsSecret under spec, but it should be under http.
5	The number of instances of Keycloak. In this example, we are using a single instance.

What about the SSL Certificate?

The Keycloak Operator does not create a certificate for the Keycloak instance. You need to create a certificate manually and store it in a secret. The Operator will use this secret to create the TLS certificate for the Keycloak instance. In the example above we are referencing a secret called keycloak-certificate in the keycloak namespace. This secret was created using the Cert Manager Operator. For example, you can use the following configuration to create a certificate for the Keycloak instance.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: keycloak-certificate
namespace: keycloak
spec:
dnsNames:
- sso.apps.ocp.aws.ispworld.at (1)
duration: 2160h0m0s
issuerRef:
kind: ClusterIssuer
name: letsencrypt-prod (2)
privateKey:
algorithm: RSA
encoding: PKCS1
rotationPolicy: Always
secretName: keycloak-certificate (3)

1	The DNS name the Certificate is valid for. This should be the same as the hostname in the Keycloak instance.
2	The issuer for the certificate. In this example, we are using the LetsEncrypt ClusterIssuer.
3	The name of the secret where the certificate is stored. This should be the same as the TLS secret in the Keycloak instance.

I strongly recommend using the Cert Manager Operator to automatically request and approve the certificate. However, if you do not have this automation in place, you can use a self-signed certificate. This certificate must be created manually and stored as a secret. For example, you can use the following command to create a self-signed certificate and store it in a secret:

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=test.keycloak.org/O=Test Keycloak./C=US"
oc create secret -n keycloak tls keycloak-certificate2 --cert=tls.crt --key=tls.key

Login in to Keycloak

Once the Keycloak instance is created and all Pods (1) are running, you can log in to the Keycloak Admin Console using the following URL: https://sso.apps.ocp.aws.ispworld.at This is the hostname we configured in the keycloak instance.

To authenticate, you need to fetch the initial password for the admin user. This password is stored in a secret called keycloak-initial-admin

You can use the following command to fetch the password:

oc extract secret/keycloak-initial-admin -n keycloak --to=-

or you can use the OpenShift Web Console to view the secret.

Once authenticated, you should see the Keycloak Admin Console:

The first thing you should do is to change the password for the admin user. I trust you know how to do this :)

Configure Keycloak to be used by OpenShift

The next steps are to configure Keycloak to be used as an Identity Provider (IDP) for OpenShift. This is done by creating a new Realm and a new Client in Keycloak. The following steps will show you the minimum configuration required to use Keycloak as an IDP for OpenShift. It does not cover all the options and features of Keycloak (and there are a lot), but it should be enough to get you started.

The full documentation for Keycloak can be found at Keycloak Documentation.

Create a new Realm and Client

In the Realm Dropdown (upper left corner) select Create new Realm
Create a new Realm called openshift (Enabled, of course) and press Create.
Now, inside the Realm openshift, select Clients and press the Create client button.
Create a new Client with the following configuration. Name it, for example, openshift.
- Be sure the Client type is set to OpenID Connect
Enable Client authentication. The rest can be left as default.
Add the following redirect URL and Web origin and press Save.:
- Redirect URL: https://oauth-openshift.apps.<your-cluster-name>/oauth2callback/* … redirecting everything under oauth2callback
- Web origin: https://oauth-openshift.apps.<your-cluster-name>;

Create a new User and a Group

In the openshift Realm, select Groups, press the Create group button and create a group called, for example, openshift-users.
In the openshift Realm, select Users and press the Add user button. Be sure to join the group openshift-users.

No more configuration is needed for the (test) user at this point.
Set the password for the user. Select the user we have just created, select the Credentials tab and press Set password. Set the password to Temporary to force the user to change the password on the first login.

Configure a Group Mapper

The above configuration is enough to log in to OpenShift using Keycloak as an IDP (except that we need to configure OpenShift itself). However, we also want to import the groups from Keycloak into OpenShift. This configuration was not easy to find, and is done by creating a Group Mapper in Keycloak.

In the openshift Realm, select Clients scopes and select the profile scope:
Select the Mappers tab and Add a mapper By configuration:
Select the Group Membership mapper.

Configure the mapper with the following settings:

Mapper Type: Group Membership
Name: openshift-groups
Token Claim Name: groups → This is the name of the claim that will be used to map the groups from Keycloak to OpenShift.
Full group path: OFF
Add to ID token: ON
Add to access token: ON

Add to userinfo: ON

Disable the Full groupo path option, otherwise the group name will be prefixed with a /. Moreover, be sure that you set the Token Claim Name correctly to the claim we will configure in OpenShift (groups).

Configure OpenShift to use Keycloak as an IDP

Now that Keycloak is configured, we need to configure OpenShift to use Keycloak as an IDP. This is done by creating a new Identity Provider in OpenShift. Before we do this, we need to create a new OAuth client secret for OpenShift in the Namespace openshift-config The secret will be used to authenticate OpenShift with Keycloak. When we created the keycloak client, we enabled the Client authentication option. This created a client secret we need to use in OpenShift.

In Keycloak, select the openshift client and select the Credentials tab and copy the Client secret.
Back in OpenShift, navigate to the openshift-config namespace and select the Secrets view. Create a new secret with the following configuration:
```
kind: Secret
apiVersion: v1
metadata:
name: openid-client-secret
namespace: openshift-config
stringData:
clientSecret: <you client secret from Keycloak> (1)
type: Opaque
```
1 The client secret we copied from Keycloak. This is the secret we will use to authenticate OpenShift with Keycloak.
Now we need to create a new Identity Provider in OpenShift. In the OpenShift Web Console, navigate to the Administration → Cluster Settings → Configuration search for OAuth and select the YAML view. Here the following must be created or added to an existing OAuth configuration:

[...]
spec:
identityProviders:
- mappingMethod: claim
name: rhsso (1)
openID:
claims: (2)
email:
- email
groups:
- groups
name:
- name
preferredUsername:
- preferred_username
clientID: openshift (3)
clientSecret:
name: openid-client-secret (4)
extraScopes: []
issuer: 'https://sso.apps.ocp.aws.ispworld.at/realms/openshift' (5)
type: OpenID (6)

1	The name of the Identity Provider. This is the name that will be displayed in the OpenShift login screen.
2	The claims that will be used to map the user to OpenShift. In this example, we are using the email, groups, name and preferred_username claims from Keycloak.
3	The client ID we created in Keycloak. This is the client ID that will be used to authenticate OpenShift with Keycloak.
4	The name of the secret we created in the openshift-config namespace.
5	The issuer URL for Keycloak. It is <hostname of keycloak>/realms/<realm name>.
6	The type of the Identity Provider. In this example, we are using OpenID Connect.

The above configuration will trigger a restart of the authentication Pods in OpenShift. Wait until all Pods have been restarted and the Operator is running again.

Test the configuration

Now it is time to test the configuration. Open a new browser window (or incognito window), navigate to the OpenShift login page and try to log in using Keycloak as an IDP rhsso.

If you have multiple IDPs configured, it is important to select the correct IDP. rhsso in this example.

By selecting the rhsso Identity Provider, you should be redirected to the Keycloak login page.

If you selected Temporary for the password, you will now be asked to change the password on the first login.
After a successful login, you should see the OpenShift Web Console, and you should be logged in as the user you created in Keycloak.
In OpenShift you will see the user created. In the Identities column you will see that it starts with rhsso, indicating that the user was authenticated using the rhsso Identity Provider.
And finally, if you navigate to User Management → Groups, you should see the group openshift-users that was created in Keycloak.

Conclusion

In this article, I have shown how to install and configure Keycloak in OpenShift for authentication. I have also shown how to configure Keycloak to be used as an Identity Provider for OpenShift and how to import groups from Keycloak into OpenShift. The biggest two challenges were to find the correct callback URL and to configure the Group Mapper in Keycloak. The rest was pretty straightforward.

What’s next? With the groups now mapped into OpenShift, you can now create RoleBindings and ClusterRoleBindings to assign the appropriate roles to the users in Keycloak. This is quite nice, as I do not need to create Users manually in OpenShift anymore (previously used HTPasswd) but instead use Keycloak as the single source of truth for users and groups. All I need to configure is the RoleBindings and ClusterRoleBindings in OpenShift.

I hope this article was helpful and you learned something new. Remember, this is just a test configuration. In production you should use a proper database and keycloak setup (high Availability, backup, etc.). If you have any questions or comments, please feel free to reach out to me on LinkedIn, Email or via GitHub issues.

References

Introducing AdminNetworkPolicies

Wed, 06 Nov 2024 00:00:00 +0000

Classic Kubernetes/OpenShift offer a feature called NetworkPolicy that allows users to control the traffic to and from their assigned Namespace. NetworkPolicies are designed to give project owners or tenants the ability to protect their own namespace. Sometimes, however, I worked with customers where the cluster administrators or a dedicated (network) team need to enforce these policies.

Since the NetworkPolicy API is namespace-scoped, it is not possible to enforce policies across namespaces. The only solution was to create custom (project) admin and edit roles, and remove the ability of creating, modifying or deleting NetworkPolicy objects. Technically, this is possible and easily done. But shifts the whole network security to cluster administrators.

Luckily, this is where AdminNetworkPolicy (ANP) and BaselineAdminNetworkPolicy (BANP) comes into play.

AdminNetworkPolicy (ANP) and BaselineAdminNetworkPolicy (BANP)

This article demonstrates the configuration of the new AdminNetworkPolicy and BaselineAdminNetworkPolicy objects using the Helm Chart admin-networkpolicies. The NetworkPolicy object is not covered in this article.

ANP and BANP are designed for cluster administrators to protect the entire cluster by creating cluster-scoped policies. They are not replacing NetworkPolicies, but instead create a tier model and can be used together. Administrators can use ANPs to enforce non-overridable policies that take precedence over NetworkPolicy objects. Administrators can use BANP to set up and enforce optional cluster-scoped network policy rules that are overridable by users using NetworkPolicy objects when necessary. When used together, ANP, BANP, and network policy can achieve full multi-tenant isolation that administrators can use to secure their cluster.

The three resources create a 3-Tier Access Control List (ACL) that is evaluated in descending order:

Tier 1 - AdminNetworkPolicy (ANP): If the traffic matches an allow or deny rule, then any existing NetworkPolicy and BaselineAdminNetworkPolicy (BANP) objects in the cluster are skipped from evaluation. If a pass rule is matched, then the evaluation is handed over to the next tier (NetworkPolicy). This means, that Cluster Administrators can enforce policies that cannot be overwritten by users (allow/deny rules) or pass the evaluation to the Network Policy, where the project owners can decide further.
Tier 2 - NetworkPolicy (NP): If the traffic passed the ANP then the NetworkPolicy is evaluating the traffic. The NetworkPolicy resources are controlled by the project owners by default.
Tier 3 - BaselineAdminNetworkPolicy (BANP): If the traffic passed the ANP and the NetworkPolicy, then the BANP is evaluating the traffic. These objects are controlled by the cluster administrators again and are cluster scoped. There can only be one BANP (named "default") configured on the cluster.

AdminNetworkPolicy

An AdminNetworkPolicy (ANP) is a cluster-scoped resource, that allow cluster administrators to secure the network traffic before NetworkPolicies in the namespaces are evaluated. These rules cannot be overwritten by project owners or developers and allow the administrators to enforce the security. Use cases could be, for example:

You want to enforce only specific egress endpoints (e.g. only allow traffic to the specific database servers)
You want to be sure that traffic from OpenShift monitoring is always allowed
You want to allow the management of NetworkPolicies to project owners and do not want to take care for them or during the onboarding.

The ANP allows cluster administrators to define:

A priority value that determines the order of its evaluation. The lower the value the higher the precedence.
A set of pods that consists of a set of namespaces or namespace on which the policy is applied.
A list of ingress rules to be applied for all ingress traffic towards the subject.
A list of egress rules to be applied for all egress traffic from the subject.

AdminNetworkPolicy Actions for Rules

The AdminNetworkPolicy allows three actions for the rules:

Allow: The traffic is allowed, and no further rules are evaluated.
Deny: The traffic is denied, and no further rules are evaluated.
Pass: The traffic is passed to the next tier (NetworkPolicy).

Subject of a Policy

In any ANP (or BANP) a subject can be defined and they specify the pods to which this AdminNetworkPolicy applies. (Note that host-networked pods are not included in subject.selection.) There are two ways to define the subject:

namespaces: The namespaces block is used to select pods via namespace selectors. Here, matchLabels or matchExpressions can be used to limit the namespaces.
pods: The Pods-Array is used to select pods via namespace AND pod selectors. Here namespaceSelector and podSelector can be set to limit the Pods.

If subject is not defined, the policy applies to all pods and namespaces in the cluster.

In my Helm chart the options are supported like the following snippets show:

Select Namespaces with matchExpressions

Values in the Helm Chart:

anp:
- name: sample-anp-rule-1
enabled: true
priority: 50
subject:
matchNamespaces: (1)
matchExpressions: (2)
- key: kubernetes.io/metadata.name
operator: NotIn
values:
- kube-system
- openshift*
- default
- kubde-info

1	matchNamespaces is used to select namespaces
2	matchExpressions is used to select namespaces with matchExpressions. In this example all namespaces that do not match (operator == NotIn) the values, so all namespaces except "kube-system, kube-info, default and openshift*" are selected.

This will result in the following AdminNetworkPolicy snippet:

 subject:
namespaces:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: NotIn
values:
- "kube-system"
- "openshift*"
- "default"
- "kubde-info"

Select Namespaces with matchLabels

Values in the Helm Chart:

anp:
- name: sample-anp-rule-1
enabled: true
priority: 5
subject:
matchNamespaces: (1)
matchLabels: (2)
apps: my-apps
tenant: my-tenant

1	matchNamespaces is used to select namespaces
2	matchLabels is used to select namespaces based on labels. In this example, all namespaces that have the labels "apps: my-apps" and "tenant: my-tenant" are selected.

This will result in the following AdminNetworkPolicy snippet:

spec:
priority: 5
subject:
namespaces:
matchLabels:
apps: "my-apps"
tenant: "my-tenant"

Select Pods with podSelectors and namespaceSelectors

Values in the Helm Chart:

anp:
- name: sample-anp-rule-1
enabled: true
priority: 5
subject:
matchPods:
- pods: (1)
namespaceSelector: (2)
labels:
kubernetes.io/metadata.name: openshift-dns
podSelector: (3)
labels:
app: dns

1	matchPods is used to select pods. Here a list of pods can be defined.
2	namespaceSelector is used to select namespaces based on labels. In this example all namespaces that have the label "kubernetes.io/metadata.name: openshift-dns" are selected.
3	podSelector is used to select pods based on labels. In this example all pods that have the label "app: dns" are selected.

This will result in the following AdminNetworkPolicy snippet:

 subject:
- pods:
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: openshift-dns
podSelector:
matchLabels:
app: dns

BaselineAdminNetworkPolicy

BaselineAdminNetworkPolicy (BANP) is a cluster-scoped resource, that allow cluster administrators to secure the network traffic after NetworkPolicies in the namespaces have been evaluated. These rules can be overwritten by project owners or developers using NetworkPolicies.

BANP is a singleton resource, meaning it can be defined only one time. Therefore, its name must be default. Moreover, the priority field is not required here.

Use cases could be, for example:

Creating default rules, such as blocking any intra-cluster traffic by default. Users will need to explicitly use NetworkPolicy objects to allow known traffic.

A BANP allows administrators to specify:

A subject that consists of a set of namespaces or namespace.
A list of ingress rules to be applied for all ingress traffic towards the subject.
A list of egress rules to be applied for all egress traffic from the subject.

BaselineAdminNetworkPolicy Actions for Rules

The BaselineAdminNetworkPolicy allows two actions for the rules. They are like the AdminNetworkPolicy, except for the pass action, which does not make sense here as BANP is the last tier (nowhere to pass).

Allow: The traffic is allowed, and no further rules are evaluated.
Deny: The traffic is denied, and no further rules are evaluated.

Examples Examples Examples

The following examples are taken directly from Kubernetes Blog: Getting started with the AdminNetworkPolicy API and Official OpenShift Documentation. Verify the values-file of the Helm Chart for the further examples.

I will show, how to configure them using the Helm Chart admin-networkpolicies and the actual result. The chart is already configured with these examples and prepared to be used with GitOps/Argo CD.

Example 1: Allow all traffic from the OpenShift monitoring namespace

Typically, it makes sense to allow the traffic from OpenShift Monitoring to all namespaces. After all, monitoring is useful :) The following example shows the possible configuration for the Helm Chart, which will render a valid ANP resource for us. It will allow ALL (including OpenShift internal Namespaces) traffic from the OpenShift monitoring namespace (labeled as kubernetes.io/metadata.name: monitoring).

---
anp:
- name: sample-anp-rule-1 (1)
enabled: true (2)
syncwave: 10
priority: 5 (3)
subject: {} (4)
ingress: (5)
- name: allow-ingress-from-monitoring (6)
enabled: true (7)
action: Allow (8)
peers: (9)
- type: namespaces
labels:
kubernetes.io/metadata.name: monitoring

1	Name of the ANP
2	Enable or disable the ANP. If disabled, the ANP will not be created. (Default is `false`)
3	Priority of the ANP. The lower the value the higher the precedence. (Default is `50`)
4	Subject of the ANP. In this case, it is empty, which means all namespaces including OpenShift internal namespaces.
5	Ingress rules of the ANP. Here a list of ingress rules for this ANP can be defined
6	Name of the ingress rule
7	Enable or disable the ingress rule. If disabled, the particular ingress rule will not be created. (Default is `false`)
8	Action of the ingress rule. In this case, it is `Allow`, which means all traffic from the OpenShift monitoring namespace will be allowed. Other options are described at AdminNetworkPolicy Actions for Rules.
9	Peers of the ingress rule. In this case, all namespaces labeled as `kubernetes.io/metadata.name: monitoring` are allowed to access all namespaces.

The ANP that will be created is the following. It is a valid ANP resource and can be applied to the cluster. (Typically applied by Argo CD) As described above it will allow incoming access from the OpenShift monitoring namespace to all namespaces.

---
apiVersion: policy.networking.k8s.io/v1alpha1
kind: AdminNetworkPolicy
metadata:
name: "sample-anp-rule-1"
labels:
helm.sh/chart: admin-networkpolicies-1.0.2
app.kubernetes.io/name: admin-networkpolicies
app.kubernetes.io/instance: release-name
app.kubernetes.io/managed-by: Helm
annotations:
argocd.argoproj.io/sync-wave: "10"
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
priority: 5
subject:
namespaces: {}
ingress:
- name: "allow-ingress-from-monitoring"
action: "Allow"
from:
- namespaces:
matchLabels:
kubernetes.io/metadata.name: "monitoring"

Example 2: Allow all traffic from labeled namespaces

As a second example, we want to allow all traffic from namespaces that are labeled with tenant: restricted to all namespaces that are labeled with anp: cluster-control-anp. This is useful, if you want to restrict access to certain namespaces. However, the rule action is configured as Pass which means that the traffic will be allowed but might be further restricted by a NetworkPolicy in the tenant namespace.

---
anp:
- name: sample-anp-rule-2
enabled: true
priority: 5
subject:
matchNamespaces: (1)
matchLabels:
anp: cluster-control-anp (2)
ingress:
- name: pass-from-restricted-tenants
enabled: true
action: Pass (3)
peers:
- type: namespaces (4)
labels:
tenant: restricted

1	Subject of the ANP. In this case, we select based on labels.
2	Label selector for the namespaces. In this case, all namespaces that are labeled with `anp: cluster-control-anp` are subject of this ANP.
3	Action of the ingress rule. In this case, it is `Pass`, which means the traffic is allowed, but might be restricted by NetworkPolicies in the tenant namespace. Other options are described at AdminNetworkPolicy Actions for Rules.
4	Peers of the ingress rule. In this case, all namespaces labeled as `tenant: restricted` are allowed to access all namespaces.

---
apiVersion: policy.networking.k8s.io/v1alpha1
kind: AdminNetworkPolicy
metadata:
name: "sample-anp-rule-2"
labels:
helm.sh/chart: admin-networkpolicies-1.0.2
app.kubernetes.io/name: admin-networkpolicies
app.kubernetes.io/instance: release-name
app.kubernetes.io/managed-by: Helm
annotations:
argocd.argoproj.io/sync-wave: "10"
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
priority: 5
subject:
namespaces:
matchLabels:
anp: "cluster-control-anp"
ingress:
- name: "pass-from-restricted-tenants"
action: "Pass"
from:
- namespaces:
matchLabels:
tenant: "restricted"

Example 3: Show possible peers settings

The most important settings for the rules are the peers settings. The following examples show the snippets of possible peers. For further information, please refer to the example in the values file: Helm Chart Values with further examples

The following rules are examples of EGRESS rules.

Allow egress traffic to namespaces labeled splunk on ports 80 and 443:

 peers:
- type: namespaces
labels:
tenant: splunk
ports:
- protocol: TCP
portNumber: 80
- portName: https

Allow egress traffic to nodes where the key "node-role.kubernetes.io/control-plane" exists on the port 6443:

 peers:
- type: nodes
expr:
- key: node-role.kubernetes.io/control-plane
operator: Exists
ports:
- protocol: TCP
portNumber: 6443

Allow egress traffic to pods labeled "app: dns" in the namespace openshift-dns on the port 53 and 5353:

 peers:
- type: pods
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: openshift-dns
podSelector:
matchLabels:
app: dns
ports:
- protocol: TCP
port: 5353
- protocol: TCP
port: 53
- protocol: UDP
port: 53
- protocol: UDP
port: 5353

Allow egress traffic to a list of IPs:

 peers:
- type: networks
ips:
- 172.29.0.0/30
- 10.0.54.0/19
- 10.0.56.38/32
- 10.0.69.0/24

Allows egress traffic to a list of domain names (*.kubernetes.io and kubernetes.io)

 peers:
- type: domainNames
domains:
- '*.kubernetes.io'
- kubernetes.io

Deny all egress traffic. This should be the last rule when full egress traffic shall be disabled. This might also be put into the BANP.

 - name: default-deny
enabled: true
action: Deny
peers:
- type: networks
ips:
- 0.0.0.0/0

Example 4: BaselineAdminNetworkPolicy

The BANP is more or less identical to ANP, except that you cannot define a "name" and a "priority". The following example creates a BANP that allows incoming and outgoing traffic to namespaces labeled "tenant-1".

banp: (1)
- enabled: true (2)
syncwave: 10
subject:
matchNamespaces:
matchLabels:
kubernetes.io/metadata.name: example.name
ingress:
- name: "deny-all-ingress-from-tenant-1"
enabled: true
action: Deny
peers:
- type: pods
namespaceSelector:
matchLabels:
custom-banp: tenant-1
podSelector:
matchLabels:
custom-banp: tenant-1
egress:
- name: allow-all-egress-to-tenant-1
enabled: true
action: Allow
peers:
- type: pods
namespaceSelector:
matchLabels:
custom-banp: tenant-1
podSelector:
matchLabels:
custom-banp: tenant-1

1	Using the key banp (instead of anp)
2	No name or priority are defined here.

Further Information

[Ep.6] Setup & Configure Advanced Cluster Security

Sun, 28 Apr 2024 00:00:00 +0000

Today I want to demonstrate the deployment and configuration of Advanced Cluster Security (ACS) using a GitOps approach. The required operator shall be installed, verified if it is running and then ACS shall be initialized. This initialization contains the deployment of several components:

Central - as UI and as a main component of ACS
SecuredClusters - installs a Scanner, Controller pods etc.
Console link into OpenShift UI - to directly access the ACS Central UI
Job to create an initialization bundle to install the Secured Cluster
Job to configure authentication using OpenShift

Let’s start …

Prerequisites

Argo CD (OpenShift GitOps) deployed
App-Of-Apps deployed

Introduction

The main components of ACS are the two custom resources: Central and SecuredCluster. The recommended way to deploy ACS is to use the Operator (alternatives would be the command line or Helm Chart). When I first came across ACS I thought about how to automate the full deployment. I did not want to install the Operator, then the Central, then manually create a so-called init-bundle, then deploy the Secured Cluster, then find the route that has been used, then find the secret that stores the initial administrator credentials and then, finally, log into ACS and activate OpenShift authentication.

As you can see there are a lot of tasks to do before I can start a customer demo.

At the same time, I started to dig into GitOps and I thought this would be a good option to create my very first Helm Chart.

Long story short, I have now a Helm Chart (actually three, because I outsourced some of the features into sub-charts) that automatically does all these things above. Once I am synchronizing my Argo CD Application everything will happen automatically.

The Configure App-of-Apps installed an Argo CD Application called in-cluster-setup-acs:

Figure 1. Argo CD Application: setup-acs

This Argo CD Application used the following path to find the Helm Chart: setup-acs

This Helm chart is a wrapper chart that uses sub-charts as dependencies to install and configure the operator as well as to do some OpenShift Jobs on top, for example, creating a ConsoleLink or creating an init-bundle.

Please check out the article Setup Compliance Operator on why I am using a wrapper chart.

Installing Advanced Cluster Security

Analysing Chart.yaml

Let’s examine the Chart.yaml file to see which dependencies are used:

The file looks like the following. Three sub-charts are defined as required to deploy and configure the ACS. This is pretty much the same as it was for the Setup Compliance Operator.

apiVersion: v2
name: setup-acs
description: Deploys Advanced Cluster Security (ACS) on target cluster. If enabled Central will be deployed too.
version: 1.0.0
dependencies:
- name: rhacs-setup (1)
version: ~1.0.0 (2)
repository: https://charts.stderr.at/
- name: helper-operator (3)
version: ~1.0.23
repository: https://charts.stderr.at/
- name: helper-status-checker (4)
version: ~4.0.0
repository: https://charts.stderr.at/
condition: helper-status-checker.enabled (5)

1	Dependency: RHACS Setup
2	Version that will be used. The "~" means that the latest version of 1.0.X will be used.
3	Dependency: Helper Operator
4	Dependency: Helper Status Checker
5	Only use this dependency when "enabled" is set

Verify the READMEs of the different Charts for detailed information on how to configure them.

Configuration of the Chart

To configure Advanced Cluster Security the values file of the wrapper Chart must be prepared accordingly.

The important thing here is, that any value that should be bypassed to a sub-chart is defined under the name of the sub-chart. For example, everything under helper-operator: will be sent to the helper-operator Chart and is used there for its configuration.

Check out the example values file I use to configure ACS and the README to find further information about the possible settings that can be done.

Let’s check the main example, to quickly start:

Installing and verifying the Operator

The first thing to do is to deploy the Operator and to verify if the Operator installation finished successfully.

The two Helm Charts helper-operator and helper-status-checker are responsible to do so.

They are configured as follows:

helper-operator:
operators:
rhacs-operator: (1)
enabled: true (2)
syncwave: '0' (3)
namespace:
name: rhacs-operator (4)
create: true
subscription:
channel: stable (5)
approval: Automatic
operatorName: rhacs-operator
source: redhat-operators
sourceNamespace: openshift-marketplace
operatorgroup: (6)
create: true
# rhacs does not support to monitor own namespace,
# therefore the spec in the OperatorGroup must be empty
notownnamespace: true
# Subchart helper-status-checker
# checks if ACS operator is ready
helper-status-checker:
enabled: true (7)
checks: (8)
- operatorName: rhacs-operator (9)
namespace:
name: rhacs-operator (10)
syncwave: 3
serviceAccount:
name: "status-checker-acs" (11)

1	Key that can be freely defined. Theoretically, you can deploy multiple operators at once.
2	Is this Operator enabled yes/no.
3	Syncwave for the Operator deployment. (Subscription and OperatorGroup etc.)
4	The Namespace where the Operator shall be deployed and if this namespace shall be created.
5	Configuration of the Subscription resource.
6	Configuration of the OperatorGroup
7	Enable status checker or not. Default: false
8	List of operators to check. Typically, only one is checked, but there could be more.
9	Name of the Operator to check (same as for helper-operator)
10	Namespace where the Operator has been installed (same as for helper-operator)
11	Name of the ServiceAccount that will be created to check the status.

Verify the READMEs at Helper Operator and Helper Operator Status Checker to find additional possible configurations.

Also verify the separate article Operator Installation with Argo CD to understand why I am verifying the status of the Operator installation.

Configuring Advanced Cluster Security

Besides the deployment of the Operator, the configuration of ACS is the most important part here. The ACS Operator provides two custom resources: Central and SecuredCluster. On the Central cluster both CRDs are required. On any other (spoke) cluster, the SecuredCluster resource is enough.

In the following example, I am going to configure both Central and SecuredCluster. Since the values file is quite huge I removed most of the additional comments, to keep this article short and readable. You can read the example values file or the README at Advanced Cluster Security Chart to find additional possible configurations. Especially, if you like to add tolerations or set resource limits.

#########################################
# Settings for Advanced Cluster Security
#########################################
rhacs-setup:
rhacs:
namespace: (1)
name: stackrox
syncwave: '0'
descr: 'Red Hat Advanced Cluster Security'
################
# CENTRAL of ACS
################
# Settings for the Central of ACS
central: (2)
enabled: true
syncwave: '3'
egress:
connectivityPolicy: Online
###############
# CENTRAL DB
###############
# Settings for Central DB, which is responsible for data persistence.
db: (3)
# -- Set Central DB resources.requests for a DEMO environment to save resources.
resources:
requests:
cpu: '1'
memory: '1Gi'
# -- If you want this component to only run on specific nodes, you can
# configure tolerations of tainted nodes.
tolerations: {}
# - effect: NoSchedule
# key: infra
# operator: Equal
# value: reserved
# - effect: NoSchedule
# key: infra
# operator: Equal
# value: reserved
###############
# SCANNER
###############
scanner: (4)
enabled: true
analyzer:
# The following settings are NOT suitable for a production environment
autoscaling:
status: "Disabled"
max: 1
min: 1
# When autoscaling is disabled, the number of replicas will always be
# configured to match this value.
replicas: 1
tolerations: {}
###############
# SCANNER DB
###############
db:
tolerations: {}
#################
# SECURED CLUSTER
#################
secured_cluster: (5)
enabled: true
syncwave: '4'
clustername: local-cluster
sensor:
tolerations: {}
admissioncontrol:
listenOn:
creates: true
events: true
updates: true
tolerations: {}
# -- Basic settings for ACS authentication
# This configuration is done by a Job, that will configure the OpenShift oauth for ACS.
basic_acs_settings: (6)
auth_provider: 'OpenShift'
auth_provider_type: 'openshift'
min_access_role: 'None'
syncwave: 5
####################################################
# Additional settings for Central and possible Jobs
####################################################
job_vars: (7)
max_attempts: 20
job_init_bundle: (8)
enabled: true
syncwave: '3'
consolelink: (9)
enabled: true
syncwave: '3'
location: ApplicationMenu
text: Advanced Cluster Security
section: Observability

1	Create the Namespace stackrox and install the ACS resources there.
2	Enable the Central during Syncwave 3 and set the connectivityPolicy to Online
3	The Central DB and its configuration. Here the resource requests are modified to allow a small installation on the DEMO environment. Also, tolerations might be set here, as well a PVCs etc.
4	Settings for the Scanner and its databases. Again, tolerations might be configurated here, but also, not shown in this example, resource limits and requests and other settings. Since I am configuring for a DEMO environment, I disabled the autoscaler and set the replica to 1.
5	The SecuredCluster is the 2nd CRD that is provided by ACS Operator. It is installed after the Central (thus a higher Syncwave). The most important setting here is the clustername. In our "local" example, the name is set to local-cluster.
6	Some basic settings, that will configure the OpenShift authentication and the minimum role for authenticated users (None)
7	Some default settings for Jobs that are started by this Helm chart.
8	The Job that initializes the creation of the init-bundle
9	The Job and its configuration to generate a direct link to ACS in the OpenShift UI.

With this ACS is about to be installed on the cluster. Let’s see what will happen during the synchronization.

Deploying Advanced Cluster Security (ACS)

Let’s hit the sync button inside OpenShift GitOps. This will start the whole process, walking through the syncwaves and the hooks that have been defined.

Figure 2. Syncing Argo CD

Since hooks are used, you must sync the whole Argo CD Application. As you can see inside Argo CD, the hooks are not shown, because they will only appear when their time has come (and will disappear afterward again). This means, that if you perform a selective sync, Argo CD does not know when it should start such a hook and they are never triggered.

The Operator installation is now started. At the moment the Operator has the status Installing. Currently, no CRDs (Central or SecuredCluster) are available yet. If we would just let Argo CD continue, it would try to create the Central configuration, based on a CRD which does not yet exist. Thus, the syncing process will fail and therefore the status-checker is going to verify if the installation was truly successful.

Figure 3. Operator is installed

The Status Checker is a simple Pod that is triggered by a Kubernetes Job. If waits until the status of the Operator is Succeeded. Until this is the case, Argo CD waits before it continues with the synchronization. (It waits until the hook ends the Job)

Figure 4. Status Checker

In the logs file of the Pod, we can see that the Operator is ready.

Figure 5. Status Checker Logs

And indeed, the status of the Operator is now Succeeded. Now it is time for Argo CD to continue the synchronization.

Figure 6. Operator Ready

The next step is to create the Central CRD. This will deploy the UI of ACS and the local Scanner. You can also see two other Jobs that have been created by the Helm Charts create_cluster_link and create_cluster_init_bundle. They will finish when the Central becomes ready.

Until the Central becomes ready, these two additional Jobs may show errors. Do not worry, OpenShift will reschedule them.

Figure 7. Installing Central

You can also see two other Jobs that have been created by the Helm Charts create_cluster_link and create_cluster_init_bundle. They will finish when the Central becomes ready.

Until the Central becomes ready, these two additional Jobs may show errors. Do not worry, OpenShift will reschedule them.

Figure 8. Init Job is waiting for Central

Once the Central has been deployed, the second CRD SecuredCluster will be added. This will trigger the installation of the Collectors and other components.

Figure 9. Installing SecuredCluster

Eventually, all Pods are running at the end. The additional Jobs are completed and ACS is ready to take care of the cluster security.

Figure 10. All Pods running

We can now use the console link that was created.

Figure 11. ACS ConsoleLink

If you disabled the creation of the ConsoleLink you would need to find the Route that ACS Operator created. Honestly, I do not know why the Operator does not create such ConsoleLink out of the box.

Since I am lazy I created another Job that automatically configures authentication via OpenShift for ACS. This way, we can simply use our OpenShift credentials to login.

Figure 12. ACS Login

And that’s it, we can now use ACS, which was deployed fully automatically.

Figure 13. Advanced Cluster Security

Conclusion

As you can see Advanced Cluster Security was completely installed. This included the Operator, the Central, the creation and installation of an init-bundle, the creation of a ConsoleLink, the configuration of the SecuredCluster CRD and the initial configuration of the auth provider inside ACS. You can now start using ACS or add additional clusters.

Speaking of additional clusters: The create-cluster-init-bundle created three certificates: collector-tls, sensor-tls and admission-control-tls. They are required so that the SecuredClusters can communicate with the Central. You could now create a separate init-bundle for each SecuredCluster, which is not really easy to automate, or you simply take these created secrets and put them into your GitOps and re-use them for any other SecuredCluster.

All these steps and configurations seem quite complicated but honestly, it is straightforward. I install any Operator using Helper Operator and in most cases. I also use Helper Operator Status Checker to find additional possible configurations. Both require simple configuration only, which you would need to know anyway when you create the Subscription object manually. Once done, you can repeat this for any other Operator.

The real magic happens when the Operator is configured at the same time because this is very individual to the Operator.

[Ep.5] Setup & Configure Compliance Operator

Thu, 25 Apr 2024 00:00:00 +0000

In the previous articles, we have discussed the Git repository folder structure and the configuration of the App-Of-Apps. Now it is time to deploy our first configuration. One of the first things I usually deploy is the Compliance Operator. This Operator is recommended for any cluster and can be deployed without any addition to the Subscription.

In this article, I will describe how it is installed and how the Helm Chart is configured.

Prerequisites

Argo CD (OpenShift GitOps) deployed
App-Of-Apps deployed

Introduction

As a reminder, at Git repository folder structure I described my preferred folder structure. I would like to deploy the Compliance Operator in the Management Cluster now. All my examples can be found at GitHub repository OpenShift Clusterconfig GitOps. The folder clusters/management-cluster/setup-compliance-operator is the one I am interested in.

Inside this folder, you will find another Helm Chart. The Helm Chart has no local templates, instead, it uses dependencies to call other (sub-) charts. However, the values.yaml is the main part to configure everything.

In case you want to have any local template, that you do NOT want to integrate into one of the sub-charts, you can easily do so, by storing them in the templates folder.

Why "empty" Helm Charts?

Actually, it would be possible to use the Helm Chart of the Chart repository directly, without creating a separate chart, that does nothing else than using dependency charts.

The reasons why I am using such an "empty" Chart are the following (in no particular order):

With that way it is possible to add templates (i.e. SealedSecrets) and modify the values-file without packaging and releasing a new Chart version every time you change a small thing.
The Multi-Source Option, which allows you to use a Helm Chart from repository A and a values file from repository B is still a TechPreview feature (Argo CD 2.10). I am using this for the App-of-Apps already, but I did not do this for all charts. This feature is on the list for Argo CD version 2.11 to become globally available.

As an alternative, it is also possible to mix Kustomize and Helm. That way you only need a kustomization.yaml file and reference to a Helm Chart. In the folder clusters/management-cluster/ingresscontroller I have such an example.

Installing Compliance Operator

Analysing Chart.yaml

As any Helm Chart a Chart.yaml file exists, that stores the basic information. The most important ones for now are the dependencies.

The file looks like the following. Three sub-charts are defined as required to deploy and configure the Compliance Operator.

apiVersion: v2
name: setup-compliance-operator
description: Deploy and configure the Compliance Operator
version: 1.0.1
dependencies:
- name: compliance-operator-full-stack (1)
version: ~1.0.0 (2)
repository: https://charts.stderr.at/
- name: helper-operator (3)
version: ~1.0.21
repository: https://charts.stderr.at/
- name: helper-status-checker (4)
version: ~4.0.0
repository: https://charts.stderr.at/
condition: helper-status-checker.enabled (5)

1	Dependency: Compliance Operator Full Stack
2	Version that will be used. The "~" means that the latest version of 1.0.X will be used.
3	Dependency: Helper Operator
4	Dependency: Helper Status Checker
5	Only use this dependency when "enabled" is set

Verify the READMEs of the different Charts for detailed information on how to configure them.

As you can see three other Helm Charts are used to actually deploy and configure the Operator.

Configuration of the Chart

To configure the Compliance Operator, the values files must be prepared accordingly.

The following is a full example of the values I typically use.

# Install Operator Compliance Operator
# Deploys Operator --> Subscription and Operatorgroup
helper-operator:
operators:
compliance-operator:
enabled: true
syncwave: '0'
namespace:
name: openshift-compliance
create: true
subscription:
channel: stable
approval: Automatic
operatorName: compliance-operator
source: redhat-operators
sourceNamespace: openshift-marketplace
operatorgroup:
create: true
notownnamespace: true
# Verify if the Operator has been successfully deployed
helper-status-checker:
enabled: true
checks:
- operatorName: compliance-operator
namespace:
name: openshift-compliance
serviceAccount:
name: "status-checker-compliance"
# Setting for the Compliance Operator
compliance-operator-full-stack:
compliance:
namespace:
name: openshift-compliance
syncwave: '0'
descr: 'Red Hat Compliance'
scansettingbinding:
enabled: true
syncwave: '3'
profiles:
- name: ocp4-cis-node
kind: Profile # Could be Profile or TailedProfile
- name: ocp4-cis
kind: Profile
scansetting: default

Let us walk through the settings in more detail.

Installing the Operator

The first thing to do is to deploy the Operator. Two resources are relevant to install an Operator:

Subscription
OperatorGroup

Both objects should be deployed at the very beginning of Argo CD synchronisation. This is done by setting the Syncwave to 0.

The main settings are the operatorName, the channel (which is the version of the operator) and the approval (which defines if the Operator is updated automatically or manually).

In addition, a Namespace object is deployed, because this Operator should run in its very own namespace.

This will start the Operator installation process.

helper-operator:
operators:
compliance-operator: (1)
enabled: true (2)
syncwave: '0' (3)
namespace:
name: openshift-compliance (4)
create: true
subscription: (5)
channel: stable # Version of the Operator
approval: Automatic # Automatic or Manual
operatorName: compliance-operator # Name of the Operator
source: redhat-operators
sourceNamespace: openshift-marketplace
operatorgroup: (6)
create: true
notownnamespace: true

1	Key that can be freely defined. Theoretically, you can deploy multiple operators at once.
2	Is this Operator enabled yes/no.
3	Syncwave for the Operator deployment. (Subscription and OperatorGroup etc.)
4	The Namespace where the Operator shall be deployed and if this namespace shall be created.
5	Configuration of the Subscription resource.
6	Configuration of the OperatorGroup

Verify the README at Helper Operator to find additional possible configurations.

Verify the Status of the Operator

After Argo CD creates the subscription and operatorgroup resources (and namespace), OpenShift will start the installation of the Operator. This installation will take a while but Argo CD does not see this. All it sees is that the Subscription resource is available and it tries to continue with the configuration of the Operator. Here it will fail because the CRDs are not available yet.

Therefore, I created a mechanism to verify if an Operator is ready or not.

Also verify the separate article Operator Installation with Argo CD that addresses the problem in more detail.

All it does is to start a small Job inside OpenShift and to verify the status of the Operator installation. If everything is fine, the Job will end successfully and Argo CD will continue with the next syncwave. Argo CD Hook and syncwaves are required here. The Job should be started after the Subscription/OperatorGroup resources have been created, which means any syncwave after "0".

The following annotations will be used by the Job:

 argocd.argoproj.io/hook: Sync (1)
argocd.argoproj.io/hook-delete-policy: HookSucceeded (2)
argocd.argoproj.io/sync-wave: {{ .syncwave | default 1 | quote }} (3)

1	Hooks are ways to run scripts before, during, and after a Sync operation.
2	Deletes the OpenShift Job again. The hook resource is deleted after the hook succeeded (e.g. Job/Workflow completed successfully).
3	Syncwave: can be configured. Must be after helper-operator (default 0) and before the Operator is configured further. Default value is 1.

The configuration for hepler_status_checker will look like the following:

# Verify if the Operator has been successfully deployed
helper-status-checker:
enabled: true (1)
checks: (2)
- operatorName: compliance-operator (3)
namespace:
name: openshift-compliance (4)
serviceAccount:
name: "status-checker-compliance" (5)

1	Enable status checker or not. Default: false
2	List of operators to check. Typically, only one is checked, but there could be more.
3	Name of the Operator to check (same as for helper-operator)
4	Namespace where the Operator has been installed (same as for helper-operator)
5	Name of the ServiceAccount that will be created to check the status.

Verify the README at Helper Operator Status Checker to find additional possible configurations.

Configuring Compliance Operator

Finally, the Operator has been deployed and has been verified. Now the time is right to configure the Operator with any configuration we would like. This means, using CRDs to do whatever the Operator offers.

This is reflected in the following part of the values file. All these settings are handed over to the sub-chart compliance-operator-full-stack.

Verify the README at Compliance Operator Chart to find additional possible configurations. Especially, if you like to do Tailored Profiles.

The compliance operator requires a so-called ScanSettingBinding that uses Profiles which are used to check the cluster compliance once a day. In this case, I am using CIS Benchmarks. There are two profiles:

ocp4-cis-node: will check the node operating system for missing but suggested configuration.
ocp4-cis: will check the OpenShift cluster for missing but suggested configuration.

# Setting for the Compliance Operator
compliance-operator-full-stack: (1)
compliance:
namespace:
name: openshift-compliance (2)
syncwave: '0'
descr: 'Red Hat Compliance'
scansettingbinding: (3)
enabled: true
syncwave: '3'
profiles: (4)
- name: ocp4-cis-node
kind: Profile # Could be Profile or TailedProfile
- name: ocp4-cis
kind: Profile
scansetting: default

1	Handing everything that comes below to the sub-chart compliance-operator-full-stack
2	Namespace where the configuration should be deployed. The Syncwave at this point could be omitted.
3	The configuration for the ScanSettingBinding. It is enabled (default = false) and has a Syncwave AFTER the helper-status-checker.
4	The list of profiles that shall be used. These must exist. The Compliance Operator offers several profiles. I usually use these two for full CIS compliance check.

Conclusion

With this configuration, the Compliance Operator will not only be installed but also configured with the same Argo CD Application. All you need to do is to synchronize Argo CD and let the magic happen. After a few minutes, everything should be in sync.

Figure 1. Sync Compliance Operator

Inside OpenShift the Operator is configured and starts doing its job:

Figure 2. Configured Compliance Operator

This concludes the deployment of the Compliance Operator. For further information about the Operator itself, please read the documentation or articles:

Also, be sure to check out the READMEs of the different Charts:

If you have any questions or problems, feel free to create a GitHub issue at any time.

Secrets Management - Vault on OpenShift

Tue, 16 Aug 2022 00:00:00 +0000

Sensitive information in OpenShift or Kubernetes is stored as a so-called Secret. The management of these Secrets is one of the most important questions, when it comes to OpenShift. Secrets in Kubernetes are encoded in base64. This is not an encryption format. Even if etcd is encrypted at rest, everybody can decode a given base64 string which is stored in the Secret.

For example: The string Thomas encoded as base64 is VGhvbWFzCg==. This is simply a masked plain text and it is not secure to share these values, especially not on Git. To make your CI/CD pipelines or Gitops process secure, you need to think of a secure way to manage your Secrets. Thus, your Secret objects must be encrypted somehow. HashiCorp Vault is one option to achieve this requirement.

HashiCorp Vault

HashiCorp Vault is a solution to solve our Secret management problem. It stores and encrypts our sensitive information at rest and in transit and enables you to create fine grained access controls (ACL). This defines who has access to a specific secret. Application A should only get access to secret A and so on. However, that is just the tip of the iceberg. Vault can do much more. If you are interested in further details, check out the introduction video by Armon, the Co-Founder of Hashicorp Vault at: https://learn.hashicorp.com/tutorials/vault/getting-started-intro?in=vault/getting-started

For this article we will keep it simple and cover:

Installing HashiCorp Vault to OpenShift
Integrating the plugin "Kubernetes Authentication" to access the Secrets
Accessing a static secret stored in HashiCorp Vault by an example application

Installing Vault

The easiest way to install Vault is by using the supported Helm chart.

Add the Helm repository to your repo:

helm repo add hashicorp https://helm.releases.hashicorp.com

Update the repository to get the latest updates.

helm repo update

Before we install Vault, we need to create a values file to set certain variables. Let’s create a file called "overwrite-values.yaml" with the following content

global:
openshift: true
server:
ha:
enabled: true
replicas: 3
raft:
enabled: true

This will tell HashiCorp Vault the environment we are going to install is (OpenShift), enables high availability with 3 replicas and enables RAFT (see below for some introduction).

Finally, let us deploy HashiCorp Vault into the namespace vault using the values file we have created in the previous step:

helm upgrade --install vault hashicorp/vault --values bootstrap/vault/overwrite-values.yaml --namespace=vault --create-namespace

This will start the agent-injector and 3 vault-pods:

oc get pods -n vault
NAME READY STATUS RESTARTS AGE
vault-0 0/1 Running 0 36s
vault-1 0/1 Running 0 36s
vault-2 0/1 Running 0 36s
vault-agent-injector-74c848f67b-sq4dq 1/1 Running 0 37s

vault agent injector: detecting applications with annotations that require vault agent which will get injected
vault-0: vault server

The vault servers remain in not-ready state until Vault has been unsealed by you.

When you check the logs of one of the pods you will see:

oc logs vault-0 -n vault
...
2022-08-11T12:31:22.497Z [INFO] core: security barrier not initialized
2022-08-11T12:31:22.497Z [INFO] core: seal configuration missing, not initialized

Vault always starts uninitialized and sealed, to protect the secrets. You need to give at least three different unseal-keys in order to be able to use Vault.

Vault’s seal mechanism uses Shamir’s secret sharing. This is a manual process to secure the cluster if it restarts. You can use auto-unseal for specific cloud providers to bypass the manual requirement.

Raft Storage

During the deployment we have enabled Raft which is the integrated HA storage for Vault. Vault can use several different styles of storage backends, but when it comes to HA and Kubernetes, Raft is easiest one since it has no other dependencies, and it is well known inside OpenShift. Etcd is using Raft as well. It is a distributed Consensus Algorithm where multiple members form a cluster and elect one leader. The leader has the responsibility to replicate everything to the followers. Since this leader election requires a majority an odd number of cluster members must be available to ensure there is a minimum number left if a member is failing.

Initialize and Unseal Vault

As mentioned, the Vault Pods are not fully available yet, since Vault it currently sealed and cannot be used. The first thing to do is to initialize and unseal it.

The following commands will login into one Pod and execute commands from there.

Let’s get the status of our Vault fist:

oc -n vault exec -it vault-0 -- vault operator init -status

This will return the message: Vault is not initialized

The following command will initialize Vault for further usage:

oc exec -ti -n vault vault-0 -- vault operator init -format=json > unseal.json

This will create the file unseal.json locally on your machine. Keep this file secure It contains by default 5 key shards and the root token you will need to unseal Vault and authenticate as root.

Keep this file secure, it contains keys to unseal Vault and the root_token to authenticate against.

Never share this file. It will look like this:

{
"unseal_keys_b64": [
"key_1",
"key_2",
"key_3",
"key_4",
"key_5"
],
"unseal_keys_hex": [
"key_hex_1",
"key_hex_2",
"key_hex_3",
"key_hex_4",
"key_hex_5"
],
"unseal_shares": 5,
"unseal_threshold": 3,
"recovery_keys_b64": [],
"recovery_keys_hex": [],
"recovery_keys_shares": 5,
"recovery_keys_threshold": 3,
"root_token": "root.token"
}

With the initialization in place Vault is put into a sealed mode. This means Vault cannot decrypt secrets at this moment. To unseal Vault you need the unseal key, which is split into multiple shards using Shamir’s secret sharing. A certain number of individual shards (default 3) must be provided to reconstruct the unseal key.

To unseal Vault lets login to our Pod "vault-0" and unseal it. Use the following command and provide one of the keys:

oc exec -ti -n vault vault-0 -- vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true (1)
Sealed true (2)
Total Shares 5
Threshold 3
Unseal Progress 1/3 (3)
Unseal Nonce 08b01535-be15-e865-251c-f948ed0661c9
Version 1.11.2
Build Date 2022-07-29T09:48:47Z
Storage Type raft
HA Enabled true

1	Vault is initialized
2	Vault is still sealed
3	The unseal progress: Currently 1 out of 3 keys have been provided

Use the same command another 2 times using different keys to complete the unseal process:

At the end the following output should be shown:

Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false (1)
Total Shares 5
Threshold 3
Version 1.11.2
Build Date 2022-07-29T09:48:47Z
Storage Type raft
Cluster Name vault-cluster-f7402e5b (2)
Cluster ID aff648f0-b3a2-1fdd-12f6-492842b08b2b
HA Enabled true
HA Cluster https://vault-0.vault-internal:8201
HA Mode active (3)
Active Since 2022-08-16T07:20:45.828215961Z
Raft Committed Index 36
Raft Applied Index 36

1	Vault is now unsealed
2	Name of our cluster
3	High availability is enabled

Vault-0 is now initialized, but there are 2 other members in our HA cluster which must be added. Let vault-1 and vault-2 join the cluster and perform the same unseal process as previously: use 3 different keys:

oc exec -ti vault-1 -n vault -- vault operator raft join http://vault-0.vault-internal:8200
# 3 times....
oc exec -ti vault-1 -n vault -- vault operator unseal
oc exec -ti vault-2 -n vault -- vault operator raft join http://vault-0.vault-internal:8200
# 3 times...
oc exec -ti vault-2 -n vault -- vault operator unseal

Verify Vault Cluster

To verify if the Raft cluster has successfully been initialized, run the following.

First, login using the root_token, that was created above, on the vault-0 pod.

oc exec -ti vault-0 -n vault -- vault login
Token (will be hidden): <root_token>

oc exec -ti vault-0 -n vault -- vault operator raft list-peers

This should return:

Node Address State Voter
---- ------- ----- -----
16ec7490-f621-42ea-976d-5f054cfaeecc vault-0.vault-internal:8201 leader true
60ba2885-432a-c7d3-d280-a824f0acce42 vault-1.vault-internal:8201 follower true
bcc4f551-79bc-47e5-d01b-97fc12d1afa5 vault-2.vault-internal:8201 follower true

As you can see Vault-0 is the leader while the other two members are followers.

Configure Kubernetes Authentication

There are multiple ways how an application can interact with Vault. One example is to use Tokens. This is quite easy but has the disadvantage that it does require additional steps of managing the life cycle of such token, moreover they might be shared, which is not what we want.

HashiCorp Vault supports different authentication methods. One of which is the Kubernetes Auth Method that must be enabled before we can use. The Kubernetes Auth Method makes use of Jason Web Tokens (JWT)s that are bound to a Service Account. When we tell Vault that a Service Account is fine to authenticate, then a Deployment using this account is able to authenticate and request Secrets.

Vault has a plugin ecosystem, which allows to enable certain plugins. To enable Kubernetes Auth Method use the following process:

Login vault-0 pods oc exec -it vault-0 -n vault — /bin/sh
execute the command: vault auth enable kubernetes which returns: Success! Enabled kubernetes auth method at: kubernetes/
Set up the Kubernetes configuration to use Vault’s service account JWT.

the address to the OpenShift API (KUBERNETES_PORT_443_TCP_ADDR) is automatically available via an environment variable.

vault write auth/kubernetes/config issuer="" \
token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Success! Data written to: auth/kubernetes/config

With this step authentication against OpenShift is enabled.

Configure a Secret

With the Kubernetes Auth Method in place we can configure a secret to test our setup. We will use an example application called expenses that has a MySQL database. The static password to bootstrap this database shall be stored in Vault. A plugin called key-value secrets engine will be used to achieve this.

There are other plugins that are specifically designed to automatically rotate secrets. For example, it is possible to dynamically create user credentials für MySQL.

You can list available engines by using the command: oc -n vault exec -it vault-0 — vault secrets list

There are currently two versions of this key/value engine:

KV Version 1: does not versionize the key/values, thus updates will overwrite the old values.
KV Version 2: does versionize the key/value pairs

In our example we will use version 2.

Like the authentication method, we need to enable the secrets engine:

oc -n vault exec -it vault-0 -- vault secrets enable \
-path=expense/static \ (1)
-version=2 \ (2)
kv (3)

1	API path where our secrets are stored
2	Version 2
3	name of our engine

You can list the enabled engines with the following command:

oc -n vault exec -it vault-0 -- vault secrets list
Path Type Accessor Description
---- ---- -------- -----------
cubbyhole/ cubbyhole cubbyhole_f1e955f9 per-token private secret storage
expense/static/ kv kv_6db09e5d n/a
identity/ identity identity_ca05e6ab identity store (1)
sys/ system system_e34a76c3 system endpoints used for control, policy and debugging

1	Enabled KV secrets engine using the path expense/static/

Now lets put a secret into our store. We will store our super-secure MySQL password into expense/static/mysql

MYSQL_DB_PASSWORD=mysuperpassword$
oc -n vault exec -it vault-0 -- vault kv put expense/static/mysql db_login_password=${MYSQL_DB_PASSWORD}

This command will store the key db_login_password with the database as value. We can get the secret by calling:

oc -n vault exec -it vault-0 -- vault kv get expense/static/mysql
====== Secret Path ======
expense/static/data/mysql (1)
======= Metadata =======
Key Value
--- -----
created_time 2022-08-17T06:08:05.839663508Z
custom_metadata <nil>
deletion_time n/a
destroyed false
version 1
========== Data ==========
Key Value
--- -----
db_login_password mysuperpassword$ (2)

1	The data path of our secret
2	our password

Configuring policies

The Secret is now stored at expense/static/mysql but there is no policy in place. Everybody who is authenticated and is calling this path will get to see the secrets. Luckily, one or more policies can be assigned to the authentication method. A policy defines capabilities that allow you to perform certain actions.

The following capabilities are known:

create - to create new data
read - to read data
delete - to delete data
list - to list data

Policies can be written either in JSON or HCL (HashiCorp Configuration Language). Let’s create a file with the following content:

path "expense/static/data/mysql" {
capabilities = ["read", "list"]
}

KV Version2 stores the secrets in a path with the prefix data/

This will limit my access to read and list only.

Write the policy:

cat my-policy.hcl | oc -n vault exec -it vault-0 -- vault policy write expense-db-mysql -

Next, we are going to bind the Vault secret to a service account and a namespace. Both objects will be created later, when we deploy the application.

oc -n vault exec -it vault-0 -- vault write auth/kubernetes/role/expense-db-mysql \ (1)
bound_service_account_names=expense-db-mysql \ (2)
bound_service_account_namespaces=expenses \ (3)
policies=expense-db-mysql \ (4)
ttl=1h (5)

1	Path or our new role
2	Name of the service account we will create and that will be used by the application
3	Name of the namespace we will create
4	Name of the policy we created earlier
5	The token is valid for 1 hour, after this period the service account must re-authenticate

Let’s start an Application

Now we will create our MySQL application into the namespace expenses. Use the following command to create the namespace and the application containing the objects Deployment, ServiceAccount (expense-db-mysql) and Service.

See at the Github Page for a full yaml specification of the three objects.

oc new-project expenses
oc apply -f https://raw.githubusercontent.com/joatmon08/vault-argocd/part-1/database/deployment.yaml

The deployment will start a Pod with a sidecar container vault-agent. This sidecar is automatically created and must not be defined inside the Deployment specification. Instead, some annotations in the Deployment define what the container should be automatically injected and also where to find our secret:

 annotations:
vault.hashicorp.com/agent-inject: "true" (1)
vault.hashicorp.com/role: "expense-db-mysql" (2)
vault.hashicorp.com/agent-inject-secret-db: "expense/static/data/mysql" (3)
vault.hashicorp.com/agent-inject-template-db: | (4)
{{ with secret "expense/static/data/mysql" -}}
export MYSQL_ROOT_PASSWORD="{{ .Data.data.db_login_password }}"
{{- end }}
...
spec:
serviceAccountName: expense-db-mysql (5)

1	Defines that the vault-agent side car container shall be automatically injected. This is the most important annotation.
2	Name of the role that was created created previously
3	The agent will inject the data from expense/static/data/mysql and stores it in a file db The file name is everything that comes after vault.hashicorp.com/agent-inject-secret-
4	Configuration… the template that defines how the secret will be rendered
5	The service account name we bound our secret to, using the Consul language. In this case the MySQL password is simply exported

The vault-agent is requesting the database password from Vault and provides it to the application where it is stored at /vault/secrets/db

oc -n expenses exec -it $(oc get pods -l=app=expense-db-mysql -o jsonpath='{.items[0].metadata.name}') -c expense-db-mysql -- cat /vault/secrets/db
# output
export MYSQL_ROOT_PASSWORD="mysuperpassword$"

The Deployment sources this file when it starts and MySQL will take this information to configure itself.

TIP: Using vault CLI on your local environment

All above commands that are dealing with Vault commands, first login to a pod and then execure the commands from there.

If you have the Vault CLI installed on your local machine, you can open a port forwarding to your Vault cluster at OpenShift and execute the commands locally:

oc port-forward -n vault svc/vault 8200
export VAULT_ADDR=http://localhost:8200
vault login
...

Thanks

Thanks to the wonderful Rosemary Wang and her Github repository: https://github.com/joatmon08/vault-argocd/tree/part-1

Also check out the Youtube Video: GitOps Guide to the Galaxy (Ep 31) | GitOps With Vault Part 1 in which Rosemary and Christian discuss this setup

Automated ETCD Backup

Sat, 29 Jan 2022 00:00:00 +0000

Securing ETCD is one of the major Day-2 tasks for a Kubernetes cluster. This article will explain how to create a backup using OpenShift Cronjob.

There is absolutely no warranty. Verify your backups regularly and perform restore tests.

Prerequisites

The following is required:

OpenShift Cluster 4.x
Integrated Storage, might be NFS or anything. Best practice would be a RWX enabled storage.

Configure Project & Cronjob

Create the following objects in OpenShift. This fill create:

A Project called ocp-etcd-backup
A PersistentVolumeClaim to store the backups. Change to your appropriate StorageClass and accessMode
A ServiceAccount called openshift-backup
A dedicated ClusterRole which is able to start (debug pods)
A ClusterRoleBinding between the created ServiceAccount and the customer ClusterRole
A 2nd ClusterRoleBinding, which gives our ServiceAccount the permission to start privileged containers. This is required to start a debug pod on a control plane node.
A CronJob which performs the backup … see Callouts for inline explanations.

A helm chart, which would create these objects below, can be found at: https://github.com/tjungbauer/ocp-auto-backup. This is probably a better way to manage the variables via the values.yaml file.

kind: Namespace
apiVersion: v1
metadata:
name: ocp-etcd-backup
annotations:
openshift.io/description: Openshift Backup Automation Tool
openshift.io/display-name: Backup ETCD Automation
openshift.io/node-selector: ''
spec: {}
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: etcd-backup-pvc
namespace: ocp-etcd-backup
spec:
accessModes:
- ReadWriteOnce (1)
resources:
requests:
storage: 100Gi
storageClassName: gp2
volumeMode: Filesystem
---
kind: ServiceAccount
apiVersion: v1
metadata:
name: openshift-backup
namespace: ocp-etcd-backup
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cluster-etcd-backup
rules:
- apiGroups: [""]
resources:
- "nodes"
verbs: ["get", "list"]
- apiGroups: [""]
resources:
- "pods"
- "pods/log"
verbs: ["get", "list", "create", "delete", "watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: openshift-backup
subjects:
- kind: ServiceAccount
name: openshift-backup
namespace: ocp-etcd-backup
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-etcd-backup
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: etcd-backup-scc-privileged
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:openshift:scc:privileged
subjects:
- kind: ServiceAccount
name: openshift-backup
namespace: ocp-etcd-backup
---
kind: CronJob
apiVersion: batch/v1
metadata:
name: cronjob-etcd-backup
namespace: ocp-etcd-backup
labels:
purpose: etcd-backup
spec:
schedule: '*/5 * * * *' (2)
startingDeadlineSeconds: 200
concurrencyPolicy: Forbid
suspend: false
jobTemplate:
metadata:
creationTimestamp: null
spec:
backoffLimit: 0
template:
metadata:
creationTimestamp: null
spec:
nodeSelector:
node-role.kubernetes.io/master: '' (3)
restartPolicy: Never
activeDeadlineSeconds: 200
serviceAccountName: openshift-backup
schedulerName: default-scheduler
hostNetwork: true
terminationGracePeriodSeconds: 30
securityContext: {}
containers:
- resources:
requests:
cpu: 300m
memory: 250Mi
terminationMessagePath: /dev/termination-log
name: etcd-backup
command: (4)
- /bin/bash
- '-c'
- >-
oc get no -l node-role.kubernetes.io/master --no-headers -o
name | grep `hostname` | head -n 1 | xargs -I {} -- oc debug
{} -- bash -c 'chroot /host sudo -E
/usr/local/bin/cluster-backup.sh /home/core/backup' ; echo
'Moving Local Master Backups to target directory (from
/home/core/backup to mounted PVC)'; mv /home/core/backup/*
/etcd-backup/; echo 'Deleting files older than 30 days' ; find
/etcd-backup/ -type f -mtime +30 -exec rm {} \;
securityContext:
privileged: true
runAsUser: 0
imagePullPolicy: IfNotPresent
volumeMounts:
- name: temp-backup
mountPath: /home/core/backup (5)
- name: etcd-backup
mountPath: /etcd-backup (6)
terminationMessagePolicy: FallbackToLogsOnError
image: registry.redhat.io/openshift4/ose-cli
serviceAccount: openshift-backup
volumes:
- name: temp-backup
hostPath:
path: /home/core/backup
type: ''
- name: etcd-backup
persistentVolumeClaim:
claimName: etcd-backup-pvc
dnsPolicy: ClusterFirst
tolerations:
- operator: Exists
effect: NoSchedule
- operator: Exists
effect: NoExecute
successfulJobsHistoryLimit: 5
failedJobsHistoryLimit: 5

1	RWO is used here, since I have no other available storage on my test cluster.
2	How often shall the job be executed. Here, every 5 minutes.
3	Bind the job to "Master" nodes.
4	Command to be executed… It fetches the actual local master nodename and starts a debugging Pod there. The backup script is called and moves the backup to /home/core/backup which is a folder on the control plane itself. The move command will move the backups from the local folder to the actual backup target volume. Finally, it will remove backups older than 30 days.
5	Mounted /home/core/backup on the master nodes, here the command will store the backups before they are moved
6	Target destination for the etcd backup on the mounted PVC

Start a Job

If you do not want to wait until the CronJob is triggered, you can manually start the Job using the following commands:

oc create job backup --from=cronjob/cronjob-etcd-backup -n ocp-etcd-backup

This will start a Pod which will do the backup:

Starting pod/ip-10-0-196-187us-east-2computeinternal-debug ...
To use host binaries, run `chroot /host`
found latest kube-apiserver: /etc/kubernetes/static-pod-resources/kube-apiserver-pod-15
found latest kube-controller-manager: /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-10
found latest kube-scheduler: /etc/kubernetes/static-pod-resources/kube-scheduler-pod-9
found latest etcd: /etc/kubernetes/static-pod-resources/etcd-pod-3
etcdctl is already installed
{"level":"info","ts":1638199790.980932,"caller":"snapshot/v3_snapshot.go:119","msg":"created temporary db file","path":"/home/core/backup/snapshot_2021-11-29_152949.db.part"}
{"level":"info","ts":"2021-11-29T15:29:50.991Z","caller":"clientv3/maintenance.go:200","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":1638199790.9912837,"caller":"snapshot/v3_snapshot.go:127","msg":"fetching snapshot","endpoint":"https://10.0.196.187:2379"}
{"level":"info","ts":"2021-11-29T15:29:53.306Z","caller":"clientv3/maintenance.go:208","msg":"completed snapshot read; closing"}
Snapshot saved at /home/core/backup/snapshot_2021-11-29_152949.db
{"level":"info","ts":1638199793.3482974,"caller":"snapshot/v3_snapshot.go:142","msg":"fetched snapshot","endpoint":"https://10.0.196.187:2379","size":"180 MB","took":2.367303503}
{"level":"info","ts":1638199793.348459,"caller":"snapshot/v3_snapshot.go:152","msg":"saved","path":"/home/core/backup/snapshot_2021-11-29_152949.db"}
{"hash":1180914745,"revision":10182252,"totalKey":19360,"totalSize":179896320}
snapshot db and kube resources are successfully saved to /home/core/backup
Removing debug pod ...
Moving Local Master Backups to target directory (from /home/core/backup to mounted PVC)

Verifying the Backup

Let’s start a dummy Pod which can access the PVC to verify if the backup is really there.

apiVersion: v1
kind: Pod
metadata:
name: verify-etcd-backup
spec:
containers:
- name: verify-etcd-backup
image: registry.access.redhat.com/ubi8/ubi
command: ["sleep", "3000"]
volumeMounts:
- name: etcd-backup
mountPath: /etcd-backup
volumes:
- name: etcd-backup
persistentVolumeClaim:
claimName: etcd-backup-pvc

Logging into that Pod will show the available backups stored at /etcd-backup which is the mounted PVC.

oc rsh -n ocp-etcd-backup verify-etcd-backup ls -la etcd-backup
total 1406196
drwxr-xr-x. 3 root root 4096 Nov 29 17:00 .
dr-xr-xr-x. 1 root root 25 Nov 29 17:06 ..
drwx------. 2 root root 16384 Nov 29 15:21 lost+found
-rw-------. 1 root root 179896352 Nov 29 15:21 snapshot_2021-11-29_152150.db
-rw-------. 1 root root 179896352 Nov 29 15:29 snapshot_2021-11-29_152949.db
-rw-------. 1 root root 179896352 Nov 29 15:32 snapshot_2021-11-29_153159.db
-rw-------. 1 root root 179896352 Nov 29 15:36 snapshot_2021-11-29_153618.db
-rw-------. 1 root root 179896352 Nov 29 15:55 snapshot_2021-11-29_155513.db
-rw-------. 1 root root 179896352 Nov 29 16:00 snapshot_2021-11-29_160020.db
-rw-------. 1 root root 179896352 Nov 29 16:55 snapshot_2021-11-29_165521.db
-rw-------. 1 root root 179896352 Nov 29 17:00 snapshot_2021-11-29_170020.db
-rw-------. 1 root root 89875 Nov 29 15:21 static_kuberesources_2021-11-29_152150.tar.gz
-rw-------. 1 root root 89875 Nov 29 15:29 static_kuberesources_2021-11-29_152949.tar.gz
-rw-------. 1 root root 89875 Nov 29 15:32 static_kuberesources_2021-11-29_153159.tar.gz
-rw-------. 1 root root 89875 Nov 29 15:36 static_kuberesources_2021-11-29_153618.tar.gz
-rw-------. 1 root root 89875 Nov 29 15:55 static_kuberesources_2021-11-29_155513.tar.gz
-rw-------. 1 root root 89875 Nov 29 16:00 static_kuberesources_2021-11-29_160020.tar.gz
-rw-------. 1 root root 89875 Nov 29 16:55 static_kuberesources_2021-11-29_165521.tar.gz
-rw-------. 1 root root 89875 Nov 29 17:00 static_kuberesources_2021-11-29_170020.tar.gz

Advanced Cluster Security - Authentication

Sat, 11 Dec 2021 00:00:00 +0000

Red Hat Advanced Cluster Security (RHACS) Central is installed with one administrator user by default. Typically, customers request an integration with existing Identity Provider(s) (IDP). RHACS offers different options for such integration. In this article 2 IDPs will be configured as an example. First OpenShift Auth and second Red Hat Single Sign On (RHSSO) based on Keycloak

Prerequisites

OpenShift 4 Cluster
Advanced Cluster Security v3.66+
Red Hat SSO Operator installed

While RHSSO will be installed during this article, only default and example values are used. These are by no means examples for a production system.

Introduction

Advanced Cluster Security comes with several default roles, which can be assigned to users:

System role

Description

Admin

This role is targeted for administrators. Use it to provide read and write access to all resources.

Analyst

This role is targeted for a user who cannot make any changes, but can view everything. Use it to provide read-only access for all resources.

Continuous Integration

This role is targeted for CI (continuous integration) systems and includes the permission set required to enforce deployment policies.

None

This role has no read and write access to any resource. You can set this role as the minimum access role for all users.

Sensor Creator

Red Hat Advanced Cluster Security for Kubernetes uses this role to automate new cluster setups. It includes the permission set to create Sensors in secured clusters.

Scope Manager

This role includes the minimum permissions required to create and modify access scopes.

It is possible to create custom roles.

Configure RHACS Authentication: OpenShift Auth

It is assumed that RHACS is already installed and login to the Central UI is available.

Login to your RHACS and select “Platform Configuration” > “Access Control”
From the drop down menu Add auth provider select OpenShift Auth

Figure 1. ACS Auth Provider
Enter a Name for your provider and select a default role which is assigned to any user who can authenticate.

It is recommended to select the role None, so new accounts will have no privileges in RHACS.

With Rules you can assign roles to specific users, based on their userid, name, mail address or groups.

For example the user with the name poweruser gets the role Admin assigned.

Verify Authentication with OpenShift Auth

Logout from the Central UI and reload the browser.
Select from the drop down OpenShift Auth

Figure 2. ACS Login
Try to login with a valid OpenShift user.
Depending on the Rules which have been defined during previous steps the appropriate permissions should be assigned.
For example: If you login as user poweruser the role Admin is assigned.

Configure Red Hat Single Sign On

The following steps will create some basic example objects to an existing RHSSO or Keycloak to test the authentication at RHACS. Skip to step #5 if you have Keycloak already up and running and would like to reuse an existing client.

The RHSSO operator (or Keycloak) is installed at the namespace single-sign-on.

Create an instance of Keycloak

apiVersion: keycloak.org/v1alpha1
kind: Keycloak
metadata:
name: example-keycloak
namespace: single-sign-on
spec:
externalAccess:
enabled: true
instances: 1

Create a Realm
This will create a Realm called Basic

apiVersion: keycloak.org/v1alpha1
kind: KeycloakRealm
metadata:
name: example-keycloakrealm
namespace: single-sign-on
spec:
instanceSelector:
matchLabels:
app: sso
realm:
displayName: Basic Realm
enabled: true
id: basic
realm: basic

oc get route keycloak -n single-sign-on --template='{{ .spec.host }}'
# keycloak-single-sign-on.apps.cluster-29t8z.29t8z.sandbox677.opentlc.com

and log into the Administration Interface.

Extract the admin password for Keycloak

The secret name is build from "credential"<keycloak-instance-name>

oc extract secret/credential-example-keycloak -n single-sign-on --to=-
# ADMIN_PASSWORD
<you password>
# ADMIN_USERNAME
admin

Be sure to select your Realm (Basic in our case), goto Clients and select a ClientID.
1. In this example we select account
  
  Figure 3. ACS Login
  
  Of course you can create or use any other Client.
2. Enable the option Implicit Flow
Get the Issuer URL from your realm. This is typically your:
https://<KEYCLOAK_URL>/auth/realms/<REALM_NAME>;

For Example: https://keycloak-single-sign-on.apps.cluster-29t8z.29t8z.sandbox677.opentlc.com/auth/realms/basic

Create Test Users

In RHSSO create 2 user accounts to test the authentication later.

Goto Users and create the users:
1. User: acsadmin
  
  First Name: acsadmin
2. User: user1
  
  First Name: user 1

You can set any other values for these users. However, be sure to set a password for both, after they have been created.

Configure RHACS Authentication: RHSSO

It is assumed that RSACS is already installed and login to the Central UI is available.

Login to your RHACS and select “Platform Configuration” > “Access Control”
From the drop down menu Add auth provider select OpenID Connect
1. Enter a “Name” for your provider i.e. “Single Sign On”
2. Leave the “Callback Mode” to the “Auto-Select” setting
3. Enter your Issuer URL
4. As Client ID enter account (or the ClientID you would like to use)
5. Leave the Client Secret empty and select the checkbox Do not use Client Secret which is good enough for our tests.
  
  Remember the two callback URL from the blue box. They must be configured in Keycloak.
6. Select a default role which is assigned to any user who can authenticate.
  
  It is recommended to select the role None, so new accounts will have no privileges in RHACS.
7. With Rules you can assign roles to specific users, based on their userid, name, mail address or groups.
8. For example the user with the name acsadmin (which have been created previously in our RHSSO) gets the role Admin assigned.

The final settings are depict in the following image:

Figure 4. ACS Login

Continue RHSSO Configuration

What is left to do is the configuration of redirect URLs. These URLs are shown in the ACS Authentication Provider configuration (see blue field in the image above)

Log back into RHSSO and select “Clients” > “account”
Into Valid Redirect URLs enter the two URLs which you saved from the blue box in the RHACS configuration.

Troubleshoot: Test Login

In RHACS you can test the login to you SSO.

Goto "Platform Configuration" > "Access Control"
Click the button "Test login"

A popup will appear which asks you to enter SSO credentials. The connection to RHSSO will be validated:

Figure 5. ACS Test SSO

Verify Authentication with OpenShift Auth

Logout from the Central UI and reload the browser.
Select from the drop down Single Sign On

Figure 6. ACS Login SSO
Try to login with a valid SSO user.
Depending on the Rules which have been defined during previous steps the appropriate permissions should be assigned.
For example: If you login as user acsadmin the role Admin is assigned.

Secure your secrets with Sealed Secrets

Sat, 25 Sep 2021 00:00:00 +0000

Working with a GitOps approach is a good way to keep all configurations and settings versioned and in sync on Git. Sensitive data, such as passwords to a database connection, will quickly come around. Obviously, it is not a idea to store clear text strings in a, maybe even public, Git repository. Therefore, all sensitive information should be stored in a secret object. The problem with secrets in Kubernetes is that they are actually not encrypted. Instead, strings are base64 encoded which can be decoded as well. Thats not good … it should not be possible to decrypt secured data. Sealed Secret will help here…

Sealed Secrets by Bitnami[1] is one option to create real, encrypted secrets. It contains two parts:

A cluster-side controller / operator, which decrypts the secrets server-side on OpenShift installed in a dedicated namespace usually called sealed secrets.
kubeseal - a client-side command line tool

Prerequisites

An OpenShift 4 cluster with cluster-admin permissions.

Sealed Secrets Operator

Goto OperatorHub and search for Sealed Secrets (This is a Community Operator)

Figure 1. Search Sealed Secrets in OperatorHub

Install the operator, using the default settings, into the namespace sealed-secrets

Figure 2. Installed Sealed Secret Operator

Install the CRD SealedSecretController

Install the following object. For now the default values can be used.

apiVersion: bitnami.com/v1alpha1
kind: SealedSecretController
metadata:
name: controller (1)
namespace: sealed-secrets
spec:
networkPolicy: false
nodeSelector: {}
podLabels: {}
resources: {}
affinity: {}
securityContext:
fsGroup: ''
runAsUser: ''
rbac:
create: true
pspEnabled: false
crd:
create: true
keep: true
ingress:
annotations: {}
enabled: false
hosts:
- chart-example.local
path: /v1/cert.pem
tls: []
serviceAccount:
create: true
name: ''
image:
pullPolicy: IfNotPresent
repository: >-
quay.io/bitnami/sealed-secrets-controller@sha256:8e9a37bb2e1a6f3a8bee949e3af0e9dab0d7dca618f1a63048dc541b5d554985
secretName: sealed-secrets-key
tolerations: []
controller:
create: true
priorityClassName: ''
podAnnotations: {}

1	Be aware of the name of the controller OBJECT (name: controller). It is used lated as part of the actual controller name

Install the command line tool kubeseal

The kubeseal binary can be easily installed using either

on Mac: brew install kubeseal or

on Linux:

wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.16.0/kubeseal-linux-amd64 -O kubeseal
install -m 755 kubeseal /usr/local/bin/

Testing Sealed Secrets

Create a new project oc new-project myproject

Create a secret

echo -n "my_super_secret_string" \
| kubectl create secret generic mypasswords --dry-run=client --from-file=password=/dev/stdin -o json \
| kubeseal --controller-namespace=sealed-secrets --controller-name=controller-sealed-secrets --format json > mysealedsecret.json (1)

1	The switches --controller-namespace define the namespace where the operator is installed, --controller-name is a combination of the SealedSecretController object name and the name of the namespace

The password=my_super_secret_string is created and piped into kubeseal which is using the controller, where the server created a certificate for encryption, to create an encrypted json file mysealedsecret.json. It is important to note, that the actually Kubernetes secret object is not created at this stage.

The file mysealedsecret.json is encrypted now and it is safe to store this file on Github.

It looks like this:

{
"kind": "SealedSecret",
"apiVersion": "bitnami.com/v1alpha1",
"metadata": {
"name": "mypasswords",
"namespace": "myproject", (1)
"creationTimestamp": null
},
"spec": {
"template": {
"metadata": {
"name": "mypasswords",
"namespace": "myproject", (1)
"creationTimestamp": null
},
"data": null
},
"encryptedData": {
"password": "AgBsSZVcTfzfNFI7ZlCsH3/4b3L7m52/O9f70pMtn1myPWHeY1QJFoxpWkH0tWosfeIoko+iB0kCyFk/iJEYSvd31zgnr90hv4e2qVtEBmm6n5B7V40ZERdiy2Cz7UXakUKDdhTjA0BTjcf0f0b2FRDenGxCHJB7cyOVGOZ36jF6IdP2k6kbsZXklti/4MXK7oskDXGzU7rTsESK0ttk5uQgrpfWrhaUip5+Db5vcG1OlHhMJ7In3NlNr0mbl+YiXsKKDNvyw9T14L3rlfvHz1xe0lIqC72i5LSCarpGoSKNOr+Sev9+b/+no6P4VDPuSLORbwVXlP5kt+8xnpZJIEqnetwhr78dt8F3xmjXVBZncdwKk22Y/b9L+uUKWPAvOT78khpUIHQPo9dV/nmz1ldvu58fCFL4TjOOtyTBcUPD3qQJp+sEXgy63l8hEaMXuLUlk+srSnJfMtwkFhl0CG2fKsg4CsQoZlvq5oKOl50sujg3Trv4W9qVVCYHA7BUXEj6J0DxjOCqSQixHRr7Z7JqIyhhdLYdHwMH80scsIb6Ok7keC82v1yae770NWWxJJ4M7Ieb2ERzgwy825gkdq9nx9I6fVxYJkkZlpKKoTvL0uno4sKjC1yQjCgW1vpiZeLIJO2f9TpvVdK2nrag0/gXPMboAL2BGnMPMwjR7OZm+iHq3NXNKiIV1aWRO4wkd/spWziLjOpeS7T1k9w4XxoACwv3g4it"
}
}
}

1	The sealed secret will be created in your project

Upload the sealed secret oc create -f mysealedsecret.json

Verify Secret

The object SealedSecret is created:

oc get SealedSecret
NAME AGE
mypasswords 3s

The SealedSecretController will decrypt the and store the secret in the namespace. This can take a few seconds:

oc get secret mypasswords
NAME TYPE DATA AGE
mypasswords Opaque 1 25s

Extract the secret and verify that your string has been stored as "normal" secret

oc extract secret/mypasswords --to=-
# password
my_super_secret_string

Updating or appending new values

The process for updateing or appending a secret is similar. The only difference is that a new value for the key string is new.

# Updaing string
echo -n "my_NEW_super_secret_string" \
| kubectl create secret generic mypasswords --dry-run=client --from-file=password=/dev/stdin -o json \
| kubeseal --controller-namespace=sealed-secrets --controller-name=controller-sealed-secrets --format json --merge-into mysealedsecret.json
# Appending
echo -n "my_appended_string" \
| kubectl create secret generic mypasswords --dry-run=client --from-file=appendedstring=/dev/stdin -o json \
| kubeseal --controller-namespace=sealed-secrets --controller-name=controller-sealed-secrets --format json --merge-into mysealedsecret.json

Be sure that you are in the namespace you want to install the secret

Upload the sealed secret oc apply -f mysealedsecret.json and extract it again to validate:

oc extract secret/mypasswords --to=-
# appendedstring
my_appended_string
# password
my_NEW_super_secret_string

Sources

[1]: Bitname Readme on Github

oc compliance command line plugin

Tue, 20 Jul 2021 00:00:00 +0000

As described at Compliance Operator the Compliance Operator can be used to scan the OpenShift cluster environment against security benchmark, like CIS. Fetching the actual results might be a bit tricky tough.

With OpenShift 4.8 plugins to the oc command are allowed. One of these plugin os oc compliance, which allows you to easily fetch scan results, re-run scans and so on. Let’s install and try it out.

Installation

An oc plugin must be deployed into the same directory as the oc command itself.

The following describes the building and installation of the plugin.

You need Go installed on your node.

Clone the Git repository:

git clone https://github.com/openshift/oc-compliance.git

Build and install the plugin

make; make install
go build -o ./bin/oc-compliance ./cmd
which oc | xargs dirname | xargs -n1 cp ./bin/oc-compliance

The plugin allows the use of oc compliance

oc compliance
You must specify a sub-command.
Usage:
oc-compliance [flags]
oc-compliance [command]
Available Commands:
bind Creates a ScanSettingBinding for the given parameters
controls Get a report of what controls you\'re complying with
fetch-fixes Download the fixes/remediations
fetch-raw Download raw compliance results
help Help about any command
rerun-now Force a re-scan for one or more ComplianceScans
view-result View a ComplianceCheckResult
Flags:
-h, --help help for oc-compliance
Use "oc-compliance [command] --help" for more information about a command.

Fetch Raw Results

Without the oc-compliance plugin it was required to manually spin up a Pod and download the results from this Pod, where the PV is mounted. Now, with a simple command we can select the ScanSettingBinding and define an output folder. For example:

oc compliance fetch-raw <object-type> <object-name> -o <output-path>

Assuming the the compliance operator was configured as in the previous article, we have the ScanSettingBinding called cis-compliance:

oc compliance fetch-raw scansettingbindings cis-compliance -n openshift-compliance -o /tmp/

This starts downloading the result archives into /tmp

Fetching results for cis-compliance scans: ocp4-cis-node-worker, ocp4-cis-node-master, ocp4-cis
Fetching raw compliance results for pod 'raw-result-extractor-fxbw8'.Fetching raw compliance results for scan 'ocp4-cis-node-worker'.........
The raw compliance results are avaliable in the following directory: /tmp/ocp4-cis-node-worker
Fetching raw compliance results for pod 'raw-result-extractor-kqrw5'.Fetching raw compliance results for scan 'ocp4-cis-node-master'.....
The raw compliance results are avaliable in the following directory: /tmp/ocp4-cis-node-master
Fetching raw compliance results for pod 'raw-result-extractor-pfrgk'.Fetching raw compliance results for scan 'ocp4-cis'..
The raw compliance results are avaliable in the following directory: /tmp/ocp4-cis
ls -la /tmp/ocp4-cis*
/tmp/ocp4-cis:
total 172
drwx------ 2 root root 4096 Jul 30 16:05 .
drwxrwxrwt. 18 root root 4096 Jul 30 16:05 ..
-rw-r--r-- 1 root root 166676 Jul 30 16:05 ocp4-cis-api-checks-pod.xml.bzip2
/tmp/ocp4-cis-node-master:
total 504
drwx------ 2 root root 4096 Jul 30 16:05 .
drwxrwxrwt. 18 root root 4096 Jul 30 16:05 ..
-rw-r--r-- 1 root root 168256 Jul 30 16:05 ocp4-cis-node-master-master-0-pod.xml.bzip2
-rw-r--r-- 1 root root 165716 Jul 30 16:05 ocp4-cis-node-master-master-1-pod.xml.bzip2
-rw-r--r-- 1 root root 166945 Jul 30 16:05 ocp4-cis-node-master-master-2-pod.xml.bzip2
/tmp/ocp4-cis-node-worker:
total 1112
drwx------ 2 root root 4096 Jul 30 16:05 .
drwxrwxrwt. 18 root root 4096 Jul 30 16:05 ..
-rw-r--r-- 1 root root 154943 Jul 30 16:05 ocp4-cis-node-worker-compute-0-pod.xml.bzip2
-rw-r--r-- 1 root root 154903 Jul 30 16:05 ocp4-cis-node-worker-compute-1-pod.xml.bzip2
-rw-r--r-- 1 root root 154939 Jul 30 16:05 ocp4-cis-node-worker-compute-2-pod.xml.bzip2
-rw-r--r-- 1 root root 154890 Jul 30 16:05 ocp4-cis-node-worker-compute-3-pod.xml.bzip2
-rw-r--r-- 1 root root 168175 Jul 30 16:05 ocp4-cis-node-worker-master-0-pod.xml.bzip2
-rw-r--r-- 1 root root 165603 Jul 30 16:05 ocp4-cis-node-worker-master-1-pod.xml.bzip2
-rw-r--r-- 1 root root 166914 Jul 30 16:05 ocp4-cis-node-worker-master-2-pod.xml.bzip2

Re-Run Scans

Sometimes it is necessary to re-run scans. This can be done by annotating the appropriate scan as described at: Performing a Rescan

With the oc plugin you can simply trigger a re-scan with a single command:

oc compliance rerun-now scansettingbindings <name of scanbinding>

For example:

oc compliance rerun-now scansettingbindings cis-compliance

Example output:

Rerunning scans from 'cis-compliance': ocp4-cis-node-worker, ocp4-cis-node-master, ocp4-cis
Re-running scan 'openshift-compliance/ocp4-cis-node-worker'
Re-running scan 'openshift-compliance/ocp4-cis-node-master'
Re-running scan 'openshift-compliance/ocp4-cis'

With the command oc get compliancescan -n openshift-compliance you can check when the scan has been done:

NAME PHASE RESULT
ocp4-cis RUNNING NOT-AVAILABLE
ocp4-cis-node-master RUNNING NOT-AVAILABLE
ocp4-cis-node-worker AGGREGATING NOT-AVAILABLE

View Results on CLI

Once a scan process has finished you can verify the check results quick and easy using the command line:

oc get ComplianceCheckResult -A

This prints for example:

NAMESPACE NAME STATUS SEVERITY
[...]
openshift-compliance ocp4-cis-audit-log-forwarding-enabled FAIL medium
[...]

The view-result can print a human readable output, for example:

oc compliance view-result ocp4-cis-audit-log-forwarding-enabled -n openshift-compliance

Example:

+----------------------+-----------------------------------------------------------------------------------------+
| KEY | VALUE |
+----------------------+-----------------------------------------------------------------------------------------+
| Title | Ensure that Audit Log |
| | Forwarding Is Enabled |
+----------------------+-----------------------------------------------------------------------------------------+
| Status | FAIL |
+----------------------+-----------------------------------------------------------------------------------------+
| Severity | medium |
+----------------------+-----------------------------------------------------------------------------------------+
| Description | OpenShift audit works at the |
| | API server level, logging |
| | all requests coming to the |
| | server. Audit is on by default |
| | and the best practice is |
| | to ship audit logs off the |
| | cluster for retention. The |
| | cluster-logging-operator is |
| | able to do this with the |
| | |
| | |
| | |
| | ClusterLogForwarders |
| | |
| | |
| | |
| | resource. The forementioned resource can be configured to logs to different third party |
| | systems. For more information on this, please reference the official documentation: |
| | https://docs.openshift.com/container-platform/4.6/logging/cluster-logging-external.html |
+----------------------+-----------------------------------------------------------------------------------------+
| Rationale | Retaining logs ensures the |
| | ability to go back in time to |
| | investigate or correlate any |
| | events. Offloading audit logs |
| | from the cluster ensures that |
| | an attacker that has access |
| | to the cluster will not be |
| | able to tamper with the logs |
| | because of the logs being |
| | stored off-site. |
+----------------------+-----------------------------------------------------------------------------------------+
| Instructions | Run the following command: |
| | |
| | oc get clusterlogforwarders |
| | instance -n openshift-logging |
| | -ojson | jq -r |
| | '.spec.pipelines[].inputRefs | |
| | contains(["audit"])' |
| | |
| | The output should return true. |
+----------------------+-----------------------------------------------------------------------------------------+
| CIS-OCP Controls | 1.2.23 |
+----------------------+-----------------------------------------------------------------------------------------+
| NIST-800-53 Controls | AC-2(12), AU-6, AU-6(1), |
| | AU-6(3), AU-9(2), SI-4(16), |
| | AU-4(1), AU-11, AU-7, AU-7(1) |
+----------------------+-----------------------------------------------------------------------------------------+
| Available Fix | No |
+----------------------+-----------------------------------------------------------------------------------------+
| Result Object Name | ocp4-cis-audit-log-forwarding-enabled |
+----------------------+-----------------------------------------------------------------------------------------+
| Rule Object Name | ocp4-audit-log-forwarding-enabled |
+----------------------+-----------------------------------------------------------------------------------------+
| Remediation Created | No |
+----------------------+-----------------------------------------------------------------------------------------+

Compliance Operator

Mon, 19 Jul 2021 00:00:00 +0000

OpenShift comes out of the box with a highly secure operating system, called Red Hat CoreOS. This OS is immutable, which means that no direct changes are done inside the OS, instead any configuration is managed by OpenShift itself using MachineConfig objects. Nevertheless, hardening certain settings must still be considered. Red Hat released a hardening guide (CIS Benchmark) which can be downloaded at https://www.cisecurity.org/.

However, an automated way to perform such checks would be nice too. To achieve this the Compliance Operator can be leveraged, which runs an OpenSCAP check to create reports of the clusters is compliant or as the official documentation describes:

The Compliance Operator lets OpenShift Container Platform administrators describe the desired compliance state of a cluster and provides them with an overview of gaps and ways to remediate them. The Compliance Operator assesses compliance of both the Kubernetes API resources of OpenShift Container Platform, as well as the nodes running the cluster. The Compliance Operator uses OpenSCAP, a NIST-certified tool, to scan and enforce security policies provided by the content.

This article shall show how to quickly install the operator and retrieve the first result. It is not a full documentation, which is written by other people at: Compliance Operator, especially remediation is not covered here.

As prerequisites we have:

Installed OpenShift 4.6+ cluster

The Compliance Operator is available for Red Hat Enterprise Linux CoreOS (RHCOS) deployments only.

Install the Compliance Operator

The easiest way to deploy the Compliance Operator is by searching the OperatorHub which is available inside OpenShift.

Figure 1. Install Compliance Operator

Keep the default settings and wait until the operator has been installed.

Figure 2. Install Compliance Operator

Custom Resources (CRDs)

The operator brings a ton of new CRDs into the system:

ScanSetting … defines when and on which roles (worker, master …) a check shall be executed. It also defines a persistent volume (PV) to store the scan results. Two ScanSettings are created during the installation:
- default: just scans without automatically apply changes
- default-auto-apply: can automatically remediate without extra steps
ScanSettingBinding … binds one or more profiles to a scan
Profile … Represent different compliance benchmarks with a set of rules. For this blog we will use CIS Benchmark profiles
ProfileBundle … Bundles a security image, which is later used by Profiles.
Rule … Rules which are used by profiles to verify the state of the cluster.
TailoredProfile … Customized profile
ComplianceScan … scans which have been performed
ComplianceCheckResult … The results of a scan. Each ComplianceCheckResult represents the result of one compliance rule check
ComplianceRemediation … If a rule ca be remediated automatically, this object is created.

Create a ScanBinding object

The first step to do is to create a ScanBiding objects. (We reuse the default ScanSetting)

Let’s create the following object, which is using the profiles ocp4-cis and ocp4-cis-node

apiVersion: compliance.openshift.io/v1alpha1
kind: ScanSettingBinding
metadata:
name: cis-compliance
profiles:
- name: ocp4-cis-node (1)
kind: Profile
apiGroup: compliance.openshift.io/v1alpha1
- name: ocp4-cis (2)
kind: Profile
apiGroup: compliance.openshift.io/v1alpha1
settingsRef:
name: default (3)
kind: ScanSetting
apiGroup: compliance.openshift.io/v1alpha1

1	use the profile ocp4-cis-node
2	use the profile ocp4-cis
3	reference to the default scansetting

As soon as the object is created the cluster is scan is started. The objects ComplianceSuite and ComplianceScan are created automatically and will eventually reach the phase "DONE" when the scan is completed.

The following command will show the results of the scans

oc get compliancescan -n openshift-compliance
NAME PHASE RESULT
ocp4-cis DONE NON-COMPLIANT
ocp4-cis-node-master DONE NON-COMPLIANT
ocp4-cis-node-worker DONE INCONSISTENT

Three different checks have been done. One overall cluster check and 2 separated for master and worker nodes.

As we used the default ScanSetting the next check will run a 1 am.

Profiles

The operator comes with a set of standard profiles which represent different compliance benchmarks.

To view available profiles:

oc get profiles.compliance -n openshift-compliance

NAME AGE
ocp4-cis 28m
ocp4-cis-node 28m
ocp4-e8 28m
ocp4-moderate 28m
rhcos4-e8 28m
rhcos4-moderate 28m

Each profile contains a description which explains the intention and a list of rules which used in this profile.

For example the profile 'ocp4-cis-node' used above is containing:

oc get profiles.compliance -n openshift-compliance -oyaml ocp4-cis-node
# Output
description: This profile defines a baseline that aligns to the Center for Internet Security® Red
Hat OpenShift Container Platform 4 Benchmark™, V0.3, currently unreleased. This profile includes
Center for Internet Security® Red Hat OpenShift Container Platform 4 CIS Benchmarks™ content.
Note that this part of the profile is meant to run on the Operating System that Red Hat
OpenShift Container Platform 4 runs on top of. This profile is applicable to OpenShift versions
4.6 and greater.
[...]
name: ocp4-cis-node
namespace: openshift-compliance
[...]
rules:
- ocp4-etcd-unique-ca
- ocp4-file-groupowner-cni-conf
- ocp4-file-groupowner-controller-manager-kubeconfig
- ocp4-file-groupowner-etcd-data-dir
- ocp4-file-groupowner-etcd-data-files
- ocp4-file-groupowner-etcd-member
- ocp4-file-groupowner-etcd-pki-cert-files
- ocp4-file-groupowner-ip-allocations
[...]

Like the profiles the different rules can be inspected:

oc get rules.compliance -n openshift-compliance ocp4-file-groupowner-etcd-member
-o jsonpath='{"Title: "}{.title}{"\nDescription: \n"}{.description}'
# Output
Title: Verify Group Who Owns The etcd Member Pod Specification File
Description:
To properly set the group owner of /etc/kubernetes/static-pod-resources/etcd-pod-*/etcd-pod.yaml ,
run the command:
$ sudo chgrp root /etc/kubernetes/static-pod-resources/etcd-pod-*/etcd-pod.yaml

Profile Customization

Sometimes is it required to modify (tailor) a profile to fit specific needs. With the TailoredProfile object it is possible to enable or disable rules.

In this blog, I just want to share a quick example from the official documentaiton: https://docs.openshift.com/container-platform/4.7/security/compliance_operator/compliance-operator-tailor.html

The following TailoredProfile disables 2 rules and sets a value for another rule:

apiVersion: compliance.openshift.io/v1alpha1
kind: TailoredProfile
metadata:
name: nist-moderate-modified
spec:
extends: rhcos4-moderate
title: My modified NIST moderate profile
disableRules:
- name: rhcos4-file-permissions-node-config
rationale: This breaks X application.
- name: rhcos4-account-disable-post-pw-expiration
rationale: No need to check this as it comes from the IdP
setValues:
- name: rhcos4-var-selinux-state
rationale: Organizational requirements
value: permissive

Working with scan results

Once a scan finished you probably want to see what the status of the scan is.

As you sse above the cluster failed to be compliant.

oc get compliancescan -n openshift-compliance
NAME PHASE RESULT
ocp4-cis DONE NON-COMPLIANT
ocp4-cis-node-master DONE NON-COMPLIANT
ocp4-cis-node-worker DONE INCONSISTENT

Retrieving results via oc command

List all results which can be remediated automatically:

oc get compliancecheckresults -l 'compliance.openshift.io/check-status=FAIL,compliance.openshift.io/automated-remediation' -n openshift-compliance
NAME STATUS SEVERITY
ocp4-cis-api-server-encryption-provider-cipher FAIL medium
ocp4-cis-api-server-encryption-provider-config FAIL medium

Further information about remediation can be found at: Compliance Operator Remediation

List all results which cannot be remediated automatically and must be fixed manually instead:

oc get compliancecheckresults -l 'compliance.openshift.io/check-status=FAIL,!compliance.openshift.io/automated-remediation' -n openshift-compliance
NAME STATUS SEVERITY
ocp4-cis-audit-log-forwarding-enabled FAIL medium
ocp4-cis-file-permissions-proxy-kubeconfig FAIL medium
ocp4-cis-node-master-file-groupowner-ip-allocations FAIL medium
ocp4-cis-node-master-file-groupowner-openshift-sdn-cniserver-config FAIL medium
ocp4-cis-node-master-file-owner-ip-allocations FAIL medium
ocp4-cis-node-master-file-owner-openshift-sdn-cniserver-config FAIL medium
ocp4-cis-node-master-kubelet-configure-event-creation FAIL medium
ocp4-cis-node-master-kubelet-configure-tls-cipher-suites FAIL medium
ocp4-cis-node-master-kubelet-enable-protect-kernel-defaults FAIL medium
ocp4-cis-node-master-kubelet-eviction-thresholds-set-hard-imagefs-available FAIL medium
ocp4-cis-node-master-kubelet-eviction-thresholds-set-hard-imagefs-inodesfree FAIL medium
ocp4-cis-node-master-kubelet-eviction-thresholds-set-hard-memory-available FAIL medium
ocp4-cis-node-master-kubelet-eviction-thresholds-set-hard-nodefs-available FAIL medium
ocp4-cis-node-master-kubelet-eviction-thresholds-set-hard-nodefs-inodesfree FAIL medium
ocp4-cis-node-master-kubelet-eviction-thresholds-set-soft-imagefs-available FAIL medium
ocp4-cis-node-master-kubelet-eviction-thresholds-set-soft-imagefs-inodesfree FAIL medium
ocp4-cis-node-master-kubelet-eviction-thresholds-set-soft-memory-available FAIL medium
ocp4-cis-node-master-kubelet-eviction-thresholds-set-soft-nodefs-available FAIL medium
ocp4-cis-node-master-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree FAIL medium
ocp4-cis-node-worker-file-groupowner-ip-allocations FAIL medium
ocp4-cis-node-worker-file-groupowner-openshift-sdn-cniserver-config FAIL medium
ocp4-cis-node-worker-file-owner-ip-allocations FAIL medium
ocp4-cis-node-worker-file-owner-openshift-sdn-cniserver-config FAIL medium
ocp4-cis-node-worker-kubelet-configure-event-creation FAIL medium
ocp4-cis-node-worker-kubelet-configure-tls-cipher-suites FAIL medium
ocp4-cis-node-worker-kubelet-enable-protect-kernel-defaults FAIL medium
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-hard-imagefs-available FAIL medium
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-hard-imagefs-inodesfree FAIL medium
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-hard-memory-available FAIL medium
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-hard-nodefs-available FAIL medium
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-hard-nodefs-inodesfree FAIL medium
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-imagefs-available FAIL medium
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-imagefs-inodesfree FAIL medium
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-memory-available FAIL medium
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-nodefs-available FAIL medium
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree FAIL medium

Retrieving RAW results

Let’s first retrieve the raw result of the scan. For each of the ComplianceScans a volume claim (PVC) is created to store he results. We can use a Pod to mount the volume to download the scan results.

The following PVC have been created on our example:

oc get pvc -n openshift-compliance
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
ocp4-cis Bound pvc-cc026ae3-2f42-4e19-bc55-016c6dd31d22 1Gi RWO managed-nfs-storage 4h17m
ocp4-cis-node-master Bound pvc-3bd47c5e-2008-4759-9d53-ba41b568688d 1Gi RWO managed-nfs-storage 4h17m
ocp4-cis-node-worker Bound pvc-77200e5f-0f15-410c-a4ee-f2fb3e316f84 1Gi RWO managed-nfs-storage 4h17m

Now we can create a Pod which mounts all PVCs at once:

apiVersion: "v1"
kind: Pod
metadata:
name: pv-extract
namespace: openshift-compliance
spec:
containers:
- name: pv-extract-pod
image: registry.access.redhat.com/ubi8/ubi
command: ["sleep", "3000"]
volumeMounts: (1)
- mountPath: "/workers-scan-results"
name: workers-scan-vol
- mountPath: "/masters-scan-results"
name: masters-scan-vol
- mountPath: "/ocp4-scan-results"
name: ocp4-scan-vol
volumes: (2)
- name: workers-scan-vol
persistentVolumeClaim:
claimName: ocp4-cis-node-worker
- name: masters-scan-vol
persistentVolumeClaim:
claimName: ocp4-cis-node-master
- name: ocp4-scan-vol
persistentVolumeClaim:
claimName: ocp4-cis

1	mount paths
2	volumesclaims to mount

This creates a Pod with the PVCs mounted inside:

sh-4.4# ls -la | grep scan
drwxrwxrwx. 3 root root 4096 Jul 20 05:20 master-scan-results
drwxrwxrwx. 3 root root 4096 Jul 20 05:20 ocp4-scan-results
drwxrwxrwx. 3 root root 4096 Jul 20 05:20 workers-scan-results

We can download the result-files to our local machine for further auditing. Therefore, we create the folder scan_results in which we copy everything:

mkdir scan-results; cd scan-results
oc -n openshift-compliance cp pv-extract:ocp4-scan-results ocp4-scan-results/.
oc -n openshift-compliance cp pv-extract:workers-scan-results workers-scan-results/.
oc -n openshift-compliance cp pv-extract:masters-scan-results masters-scan-results/.

This will download several bzip2 archives for the appropriate scan result.

Once done, you can delete the "download pod" using: oc delete pod pv-extract -n openshift-compliance

Work wth RAW results

So above section described the download of the bzip2 files but what to do with it? First, you can import it into a tool which is able to read openScap reports. Or, secondly, you can use the oscap command to create a html output.

We have downloaded the following files:

./ocp4-scan-results/0/ocp4-cis-api-checks-pod.xml.bzip2
./masters-scan-results/0/ocp4-cis-node-master-master-0-pod.xml.bzip2
./masters-scan-results/0/ocp4-cis-node-master-master-2-pod.xml.bzip2
./masters-scan-results/0/ocp4-cis-node-master-master-1-pod.xml.bzip2
./workers-scan-results/0/ocp4-cis-node-worker-compute-0-pod.xml.bzip2
./workers-scan-results/0/ocp4-cis-node-worker-compute-1-pod.xml.bzip2
./workers-scan-results/0/ocp4-cis-node-worker-compute-3-pod.xml.bzip2
./workers-scan-results/0/ocp4-cis-node-worker-compute-2-pod.xml.bzip2

To create the html output (be sure that open-scap is installed on you host):

mkdir html
oscap xccdf generate report ocp4-scan-results/0/ocp4-cis-api-checks-pod.xml.bzip2 >> html/ocp4-cis-api-checks.html
oscap xccdf generate report masters-scan-results/0/ocp4-cis-node-master-master-0-pod.xml.bzip2 >> html/ocp4-cis-node-master-master-0.html
oscap xccdf generate report masters-scan-results/0/ocp4-cis-node-master-master-1-pod.xml.bzip2 >> html/ocp4-cis-node-master-master-1.html
oscap xccdf generate report masters-scan-results/0/ocp4-cis-node-master-master-2-pod.xml.bzip2 >> html/ocp4-cis-node-master-master-2.html
oscap xccdf generate report workers-scan-results/0/ocp4-cis-node-worker-compute-0-pod.xml.bzip2 >> html/ocp4-cis-node-worker-compute-0.html
...

The resulted html files are too big to be show here, but some snippets should give an overview:

To view the html output as an example I have linked the html files:

Overall Scoring of the result:

Figure 3. Scoring

A list if passed or failed checks:

Figure 4. Scan Result list

Scan details with a link to the CIS Benchmark section and further explainations on how to fix the issue:

Figure 5. Scan details

Performing a rescan

If it is necessary to run a rescan, the ComplianceScan object is simply annotated with:

oc annotate compliancescans/<scan_name> compliance.openshift.io/rescan=

If default-auto-apply is enabled, remediation which changes MachineConfigs will trigger a cluster reboot.

1	The namespace where the OpenBao deployment is running.
2	The self-signed CA Issuer.

Security on TechBlog about OpenShift/Ansible/Satellite and much more

The Guide to OpenBao - Authentication Methods - Part 7

Introduction

Authentication Workflow

Prerequisites for Authentication Methods

Using the CLI

Using the UI

Understanding Policies

Basic Policy Structure

Policy Capabilities

Default Policies

Creating Policies using the CLI

Kubernetes Authentication Method

Step 1: Enable Kubernetes Auth

Step 2: Configure Kubernetes Auth

Step 3: Create a Policy for the Application

Step 4: Create a Kubernetes Auth Role

Ensure the KV v2 secrets engine is mounted at secret/

Step 5: Store the database secret in OpenBao (one-time)

Step 6: Create the expense namespace and demo application

Step 7: Test Kubernetes authentication manually

Summary: Kubernetes auth method

LDAP Authentication Method

Step 1: Enable LDAP Auth

Step 2: Configure LDAP (Active Directory Example)

Step 3: Create the policies used by the group mapping

Step 4: Map LDAP groups to policies

Step 5: Test LDAP authentication

What jdoe can and cannot do (user-policy)

Common Practices for Authentication

Common Authentication Patterns

What is Coming Next?

Conclusion

Resources

The Guide to OpenBao - Initialisation, Unsealing, and Auto-Unseal - Part 6

What is happening?

Handling Initialisation and Unsealing

Approach 1: Manual Initialisation (simplest)

Approach 2: Kubernetes Job for initialisation

How the Secret is created

RBAC Configuration

Init Job that creates the Secret

Using the Secret in a Job (unseal)

Approach 3: Auto-unseal with Key Management Service KMS (recommended for production)

Auto-unseal options overview

Why use OpenBao when the seal is in KMS? Why not keep all secrets in KMS and skip OpenBao?

Option: AWS KMS

Create a KMS key

Key policy for OpenBao

Create an IAM user for the seal

Create the Secret

Helm / Argo CD configuration (AWS)

Approach 4: Init container with external secret store

Option: init Container example

Option: Transit seal

Dedicated unseal service (seal-only OpenBao)

Migration from Shamir to auto-unseal

Resources

The Guide to OpenBao - GitOps Deployment with Argo CD - Part 5

Introduction

Prerequisites

The (Wrapper) Helm Chart

Helm Chart for the CA Certificate

Helm Chart for the OpenBao Deployment

Creating Argo CD Applications

OpenBao is running—what next?

Resources

The Guide to OpenBao - Enabling TLS on OpenShift - Part 4

Introduction

Why Enable TLS?

What Must Be Considered?

Prerequisites

Overview of Steps

Step 1: Certificate Authority (CA) for OpenBao

Step 2: Create an Issuer for the OpenBao Server and Agent Injector

Step 3: Certificate for the OpenBao Server

Step 4: Certificate for the Agent Injector

Step 5: Helm Values for Server TLS

Step 6: Upgrade the Helm Release

New cluster: initialize and unseal openbao-0

Ensure the KV v2 secrets engine is mounted at `secret/`

But wait there is more…