Ansible Automation Platform on TechBlog about OpenShift/Ansible/Satellite and much more

Creating an auditd rule to monitor Ansible Automation Platform static secrets

Tue, 09 Jun 2026 00:00:00 +0000

Ansible Automation Platform (AAP) comes with an elaborate credential management solution. All credentials within AAP are stored encrypted in a PostgreSQL database. But the keys used for encrypting database fields are currently stored on the machine(s) running AAP. In this blog post, we show we created an auditd rule to get at least some kind of monitoring if those keys are accessed.

Ansible Automation Platform local SECRET_KEY

AAP comes with a credential management solution. Credentials are used to

access remote machines
provide configuration data for automation, like API keys
connect AAP to source code repositories and inventories

and much more. You can even integrate external credential management solutions like HashiCorp Vault or OpenBao.

All credentials stored within AAP are saved as encrypted database fields in a PostgreSQL database. There is a main key and AAP creates for every encrypted database field a derived AES-256 encryption key. For more details see the documentation.

But there is one caveat: the main secret key is stored on the local file system. There is currently no supported option to integrate something like a HSM (High Security Module) for storing the main secret securely.

To mitigate this issue we deployed a Linux audit rule that monitors read/write and attribute changes to the main secret.

Configuring Linux auditd

We use the containerized version of AAP 2.6. In the containerized version of AAP those secrets are stored as podman secrets as the user running the AAP containers.

To list AAP podman secrets execute

$ podman secret list
ID NAME DRIVER CREATED UPDATED
02a2eb397dc5cedefde6ae411 hub_resource_server file 2 hours ago 2 hours ago
0dd84dc7b808917c4efb6ea9f controller_channels file 2 hours ago 2 hours ago
85fb73ff57729250c019976ed hub_database_fields file 2 hours ago 2 hours ago
c482ff7c07f908e443d327293 controller_postgres file 2 hours ago 2 hours ago
e6750121a8997cce8185072bf gateway_admin_password file 2 hours ago 2 hours ago
2731b2d03806f88f35809de66 eda_admin_password file 2 hours ago 2 hours ago
31ca44577b4c5b61dd3f0a0b6 eda_secret_key file 2 hours ago 2 hours ago
856473626bad9ff37b1b979d4 eda_resource_server file 2 hours ago 2 hours ago
b20ec0897fc2b6533544c4441 gateway_redis_url file 2 hours ago 2 hours ago
1dfdd812847ed4c1c01b194e2 gateway_db_password file 2 hours ago 2 hours ago
32d04575881dd781be192ef49 eda_db_password file 2 hours ago 2 hours ago
5102f69350717c15177b80821 hub_secret_key file 2 hours ago 2 hours ago
d1faf92922182604cfa746318 controller_resource_server file 2 hours ago 2 hours ago
d29f0d8c2fc709655dd918f2c postgresql_admin_password file 2 hours ago 2 hours ago
1b3ee24261bdff50d2c70e3f0 gateway_secret_key file 2 hours ago 2 hours ago
ae6c9e78e91dea180d22db6ee hub_settings file 2 hours ago 2 hours ago
b348d89bfb7d3b18f33a67d07 controller_secret_key file 2 hours ago 2 hours ago (1)

1	the controller secret key

If we want to know where those secrets are stored we can execute

$ podman secret inspect controller_secret_key
[
{
"ID": "b348d89bfb7d3b18f33a67d07",
"CreatedAt": "2026-06-09T10:57:28.278138715Z",
"UpdatedAt": "2026-06-09T10:57:28.278138715Z",
"Spec": {
"Name": "controller_secret_key",
"Driver": {
"Name": "file",
"Options": {
"path": "/home/admin/.local/share/containers/storage/secrets/filedriver"
}
},
"Labels": {}
}
}
]

Let’s see what’s in the filedriver directory:

$ ls /home/admin/.local/share/containers/storage/secrets/filedriver
secretsdata.json secretsdata.lock(1)

1	we are mainly interested in the secretsdata.json file

So the secretsdata.json file stores all podman secrets that are using the filedriver. For more information see the section Secret Drivers in the podman-secrets-create(1) manpage.

Let’s create a Linux auditd rule to monitor access to this file. As we are using Red Hat Enterprise Linux 10 a more detailed introduction to the Linux audit system is available here.

We created the file /etc/audit/rules.d/aap_secrets.rules with the following content:

-w /home/admin/.local/share/containers/storage/secrets/filedriver/secretsdata.json -p rwa -k aap_secrets (1)

1	-p rwa means monitor read/write/attribute changes, -k tells the audit system log events with an additional key "aap_secrets"

To activate the rule we need to load the audit rule via:

# auditctl -R /etc/audit/rules.d/aap_secrets.rules

You can list loaded audit rules via

# auditctl -l
-w /home/admin/.local/share/containers/storage/secrets/filedriver/secretsdata.json -p rwa -k aap_secrets

As a test we can cat the secrets file and check if auditd logged an event:

# cat /home/admin/.local/share/containers/storage/secrets/filedriver/secretsdata.json
...
# grep aap_secrets /var/log/audit/audit.log
type=SYSCALL msg=audit(1781009319.993:204): arch=c000003e syscall=257 success=yes exit=3 a0=ffffff9c a1=7ffc5276b65c a2=0 a3=0 items=1 ppid=6186 pid=6695 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts2 ses=3 comm="cat" exe="/usr/bin/cat" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="aap_secrets"ARCH=x86_64 SYSCALL=openat AUID="admin" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root" (1)

1	We can see that the cat command (comm=cat) was used by the root user to display the file contents

Summary

In this blog post we demonstrated how to configure the Linux audit system to log events if the AAP database secret key is accessed. The audit subsystem can also monitor system calls, for a detailed introduction see Audit system architecture.

Using a local Renovate bot to manage Ansible collection dependencies

Tue, 26 May 2026 00:00:00 +0000

We wanted to use renovate bot to automatically update collection dependencies for one of our Ansible playbook repositories. This blog post describes the basics of renovate, how we configured it and how we actually run renovate locally.

Understanding renovate

As this was our first try of using renovate for dependency updates, we struggled to understand how renovate works and how it is configured.

Renovate is a bot that is able to scan multiple repositories for dependencies and create pull/merge requests with updated dependencies.

Configuration is split between a configuration for the bot and a local one for each repository.

So you basically have

One or more global configuration files that configure the bot (config.js) written in JavaScript or TypeScript
One configuration file per repository (renovate.json) written in JSON that contains local configuration options just for this repository.

Renovate is very flexible on how to structure the configuration. For more information see config-overview in the docs.

Renovate also supports multiple platforms, like GitHub, GitLab or Forgejo. For a complete list see renovate platforms.

The idea is that you run the bot once, it scans all configured repositories and creates pull requests in every repository with dependency updates.

If your projects are hosted on GitHub and you would like to use GitHub Actions for running renovate the straight forward solution is to follow Mend Renovate on GitHub Marketplace.

But we wanted more control and tried to

run renovate on a local machine. This should be migrated to a Job Template or Kubernetes CronJob later (might be worth another post)
use a private Automation Hub for hosting dependencies. So renovate should query collections on our Private Automation Hub for dependency updates
credentials should be made available via environment variables to the bot

Configuring and running renovate

As a first test we created a local bot configuration file config.js in our home directory. The GIT repository we used for testing is https://github.com/tosmi-ansible/aap-setup.

The first thing to be aware of is that the renovate bot by default always searches the repository configuration file (renovate.json) in the upstream repository. Changes to local files are ignored.

One way to avoid this is using --platform=local. This actually uses the local renovate.json file, but has the caveat that not all renovate functionality is available.

But in our case this was good enough as we just wanted to test AAP Hub authentication.

It is also helpful to run renovate with debugging enabled, this dumps the configuration currently used to STDOUT.

For example we had the following renovate.json config for testing:

{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": [
"config:best-practices"
],
"timezone": "Europe/London",
"schedule": [
"at any time"
],
"packageRules": [
{
"matchDatasources": ["galaxy-collection"]
},
{
"matchDatasources": ["github-tags"],
"enabled": "false"
}
],
"hostRules": [
{
"matchHost": "aap.apps.hub.aws.tntinfra.net",
"token": "Token <the token>",
"authType": "Token-Only"
}
]
}

But running renovate with LOG_LEVEL=debug set in the environment revealed the following settings:

DEBUG: Repository config (repository=tosmi-ansible/aap-setup)
"fileName": "renovate.json",
"config": {
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": ["config:recommended"]
}

The configuration above is the configuration available in the remote GIT repository and not our local configuration.

Renovate does not use a local renovate.json repository config by default. It always pulls the configuration from the remote repository except when you use _—platform=local.

Furthermore renovate heavily relies on local caching. We wanted to test if renovate is using our hub token for connecting to the Hub, but we could not see any log entry in the AAP Hub container.

Renovate creates its cache files in a directory called baseDir. You can specify the location (and later wipe the cache) with the --base-dir option.

As we are using a self signed certificate in our AAP Hub test installation we had to specify NODE_TLS_REJECT_UNAUTHORIZED=0 to disable TLS verification.

To summarize our test configuration:

We used the following config.js global renovate configuration file:

module.exports = {
token: '<github token>',
platform: 'github',
onboardingConfig: {
extends: ['config:recommended'],
},
//repositories: ['tosmi-ansible/aap-setup'], (1)
};

1	Repository listing is not allowed when using --platform=local

In the repository where we want to test automatic dependency updates we used the following renovate.json configuration:

{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": [
"config:best-practices"
],
"timezone": "Europe/London",
"schedule": [
"at any time"
],
"packageRules": [
{
"matchDatasources": ["galaxy-collection"]
},
{
"matchDatasources": ["github-tags"],
"enabled": "false" (1)
}
],
"hostRules": [
{
"matchHost": "aap.apps.hub.aws.tntinfra.net",
"token": "Token <the token>",
"authType": "Token-Only"
}
]
}

1	We disabled GitHub tag lookup explicitly as renovate would find some of our dependencies on GitHub

This is our collections/requirements.yaml file with a test dependency:

roles:
collections:
- name: kubernetes.core
version: 6.3.0
source: https://aap.apps.hub.aws.tntinfra.net/api/galaxy/content/rh-certified/

On our AAP Private Automation Hub we had kubernetes.core version 6.4.0 available.

So to run and test renovate locally we used the following command line:

NODE_TLS_REJECT_UNAUTHORIZED=0 LOG_LEVEL=debug RENOVATE_CONFIG_FILE=~/config.js renovate --platform=github --repository-cache=disabled --base-dir=/tmp/cache

Secret management and configuration cleanup

The next step is to remove the hard-coded tokens from our configuration. We wanted to use environment variables, as this allows us to run renovate as:

A Kubernetes CronJob with a Pod. We could inject the token via a Secret
As a job template within AAP with tokens stored as AAP credentials or
A normal Unix cron job with environment variables set

As the global renovate configuration config.js is written in JavaScript we could leverage process.env to retrieve required tokens from environment variables.

Here is our modified config.js:

module.exports = {
platform: 'github',
onboardingConfig: {
extends: ['config:recommended'],
},
// repositories: ['tosmi-ansible/aap-setup'], (1)
secrets: {
AAP_HUB_TOKEN: process.env.AAP_HUB_TOKEN, (2)
},
hostRules: [
{
matchHost: "aap.apps.hub.aws.tntinfra.net",
token: "{{ secrets.AAP_HUB_TOKEN }}", (3)
authType: "Token" (4)
},
],
};

1	We disable repository listing for --platform=local. Otherwise renovate will raise an error.
2	We define a new secret AAP_HUB_TOKEN and read the value from the environment variable AAP_HUB_TOKEN
3	We use the secret as our token for authenticating to AAP Hub
4	The authType setting is a little bit special, see the following section

For GitHub authentication renovate expects a GitHub token in the environment variable RENOVATE_TOKEN.

A few words about the authType setting. AAP Hub requires the following authorization header in the HTTP request for authentication:

Authorization: Token <the token>

With renovate we have two options to achieve this:

Either set authType to "Token-Only" and specify the token as "Token <the token>". So with token-only renovate takes the token string verbatim
Or set the authType to the string we would like to prepend to the token as we did

We had to dig into the source code of renovate to understand how it functions.

And here is the renovate.json repository configuration that we used:

{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": [
"config:best-practices"
],
"timezone": "Europe/London",
"schedule": [
"at any time"
],
"packageRules": [
{
"matchDatasources": ["galaxy-collection"]
},
{
"matchDatasources": ["github-tags"],
"enabled": "false"
}
]
}

You could also move the hostRules to the repository local renovate.json file and still use secrets.AAP_HUB_TOKEN, but as this is going to be our global configuration for Ansible code base covering multiple repositories our current idea is to store the authentication in the global configuration.

To set the required environment variables containing our secrets we execute

$ export AAP_HUB_TOKEN=$(pass show other/ansible/aws-aap-tokens | awk '/hub/ {print $2}') (1)
$ export RENOVATE_TOKEN=$(pass show other/github/tosmi-rw-token | awk '/token/ {print $2}')

1	We use the pass utility for secret management.

Integration with GitHub

So the last step is to store everything (the global and local configuration) on GitHub.

For the global repository we created the tosmi-ansible/renovate-config repository.

Here is our final config.js global configuration file in this repository:

module.exports = {
platform: 'github',
gitAuthor: "Renovate Bot <renovate-bot@stderr.at>",
dryRun: null,
onboardingConfig: {
extends: ['config:recommended'],
},
repositories: ['tosmi-ansible/aap-setup'], (1)
secrets: {
AAP_HUB_TOKEN: process.env.AAP_HUB_TOKEN,
},
hostRules: [
{
matchHost: "aap.apps.hub.aws.tntinfra.net",
token: "{{ secrets.AAP_HUB_TOKEN }}",
authType: "Token"
},
],
};

1	This time we tell renovate which repositories it should scan.

The default configuration default.json, also stored in the global repository:

{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": [
"config:best-practices"
],
"timezone": "Europe/London",
"schedule": [
"at any time"
],
"enabledManagers": ["ansible", "ansible-galaxy"], (1)
"packageRules": [
{
"matchDatasources": ["galaxy-collection"]
},
{
"matchDatasources": ["github-tags"],
"enabled": "false"
}
]
}

1	We only enable ansible and ansible-galaxy managers. Otherwise renovate would search for various other sources for dependencies.

tosmi-ansible/aap-setup contains our local configuration:

{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": [
"tosmi-ansible/renovate-config",
"config:best-practices"
]
}

This only contains a reference to our global configuration and we would like to use the best-practices preset.

We can now run renovate from the renovate-config repository and it will scan the configured repository and create a pull request with updated dependencies if required.

$ git clone git@github.com:tosmi-ansible/renovate-config.git
$ cd renovate-config
$ NODE_TLS_REJECT_UNAUTHORIZED=0 renovate --platform=github --base-dir=/tmp/renovate-cache (1)

1	We still need to set NODE_TLS_REJECT_UNAUTHORIZED to 0, because our AAP Hub instance uses a private certificate

One minor issue that we discovered is that if a pull request for an update already existed (even a closed one) Renovate would not open a new one. If you run renovate with LOG_LEVEL=debug you will see the following message:

DEBUG: Closed PR #19 already exists. Skipping branch. (repository=tosmi-ansible/aap-setup, branch=renovate/kubernetes.core-6.x)
"prTitle": "Update dependency kubernetes.core to v6.4.0"

To convince Renovate to create a new pull request just rename the closed pull request on GitHub. We just added the prefix "RENAMED". After renaming the pull request, renovate will happily create a new one.

Tips and tricks

This section list various things we learning while running renovate locally

Renovate does not honor new dependencies in requirements.yaml

We had an issues with new dependencies in requirements.yaml. Wiping the renovate cache (/tmp/renovate_cache) in our case fixed this and renovate created new pull requests for the added dependencies.

Renovate does not open new PR (pull requests) for updated dependencies

We double checked AAP and our renovate configuration and there was clearly a newer version of a collection but renovate just did not open a new PR.

Running renovate with LOG_LEVEL=debug revealed the following message (we tried to update the redhat.openshift_virtualization collection):

 {
"branchName": "renovate/redhat.openshift_virtualization-2.x",
"prNo": null,
"prTitle": "Update dependency redhat.openshift_virtualization to v2.2.4",
"result": "branch-limit-reached", (1)

1	renovate does not open a new branch because of "branch-limit-reached"

We check our upstream repository on GitHub and there was just the main branch, hm…

Looking further up in the log we found:

DEBUG: prHourlyLimit of the upgrades present in this branch (repository=tosmi-ansible/aap-setup, branch=renovate/infra.aap_utilities-3.x)
"limits": [{"depName": "infra.aap_utilities", "prHourlyLimit": 2}]
DEBUG: Calculated lowest prHourlyLimit among the upgrades present in this branch is 2. (repository=tosmi-ansible/aap-setup, branch=renovate/infra.aap_utilities-3.x)
DEBUG: Hourly PRs limit reached (repository=tosmi-ansible/aap-setup, branch=renovate/infra.aap_utilities-3.x) (1)
"hourlyPrCount": 2,
"hourlyPrLimit": 2
DEBUG: Reached branch limit - skipping branch creation (repository=tosmi-ansible/aap-setup, branch=renovate/infra.aap_utilities-3.x)

1	There is a hourly branch limit of two in place

As we just ran renovate to update another dependency, we seem to be hitting this hourly limit. The docs mention a default of 2.

The solution was to run renovate with the option --pr-hourly-limit=0:

NODE_TLS_REJECT_UNAUTHORIZED=0 renovate --platform=github --base-dir=/tmp/renovate-cache --pr-hourly-limit=0

Onboarding to Ansible Automation Platform with Configuration as Code

Wed, 04 Mar 2026 00:00:00 +0000

We had the honor of presenting at the 2nd Ansible Anwendertreffen in Austria. The topic was the onboarding of application teams to the Ansible Automation Platform.

We created an extensive demo that:

Onboards a new tenant into an AAP organization.
Each tenant gets a configuration as code repository, cloned from a template.
A push event to the config-as-code repository triggers an update of AAP objects.
Provides an example repository for each tenant to get started.
The example repository provides a webhook that creates an OpenShift Virtualization VM for testing code changes when a feature branch is created.
Contains a playbook to display objects that are not currently managed by the tenant’s configuration-as-code repository.

The source code for the demo is available on GitHub.

The README contains more details about the implementation, and the slide deck is also available on GitHub.