Skip to main content

Creating an auditd rule to monitor Ansible Automation Platform static secrets

Ansible Automation Platform (AAP) comes with an elaborate credential management solution. All credentials within AAP are stored encrypted in a PostgreSQL database. But the keys used for encrypting database fields are currently stored on the machine(s) running AAP. In this blog post, we show we created an auditd rule to get at least some kind of monitoring if those keys are accessed.

Ansible Automation Platform local SECRET_KEY

AAP comes with a credential management solution. Credentials are used to

  • access remote machines

  • provide configuration data for automation, like API keys

  • connect AAP to source code repositories and inventories

and much more. You can even integrate external credential management solutions like HashiCorp Vault or OpenBao.

All credentials stored within AAP are saved as encrypted database fields in a PostgreSQL database. There is a main key and AAP creates for every encrypted database field a derived AES-256 encryption key. For more details see the documentation.

But there is one caveat: the main secret key is stored on the local file system. There is currently no supported option to integrate something like a HSM (High Security Module) for storing the main secret securely.

To mitigate this issue we deployed a Linux audit rule that monitors read/write and attribute changes to the main secret.

Configuring Linux auditd

We use the containerized version of AAP 2.6. In the containerized version of AAP those secrets are stored as podman secrets as the user running the AAP containers.

To list AAP podman secrets execute

$ podman secret list
ID                         NAME                        DRIVER      CREATED      UPDATED
02a2eb397dc5cedefde6ae411  hub_resource_server         file        2 hours ago  2 hours ago
0dd84dc7b808917c4efb6ea9f  controller_channels         file        2 hours ago  2 hours ago
85fb73ff57729250c019976ed  hub_database_fields         file        2 hours ago  2 hours ago
c482ff7c07f908e443d327293  controller_postgres         file        2 hours ago  2 hours ago
e6750121a8997cce8185072bf  gateway_admin_password      file        2 hours ago  2 hours ago
2731b2d03806f88f35809de66  eda_admin_password          file        2 hours ago  2 hours ago
31ca44577b4c5b61dd3f0a0b6  eda_secret_key              file        2 hours ago  2 hours ago
856473626bad9ff37b1b979d4  eda_resource_server         file        2 hours ago  2 hours ago
b20ec0897fc2b6533544c4441  gateway_redis_url           file        2 hours ago  2 hours ago
1dfdd812847ed4c1c01b194e2  gateway_db_password         file        2 hours ago  2 hours ago
32d04575881dd781be192ef49  eda_db_password             file        2 hours ago  2 hours ago
5102f69350717c15177b80821  hub_secret_key              file        2 hours ago  2 hours ago
d1faf92922182604cfa746318  controller_resource_server  file        2 hours ago  2 hours ago
d29f0d8c2fc709655dd918f2c  postgresql_admin_password   file        2 hours ago  2 hours ago
1b3ee24261bdff50d2c70e3f0  gateway_secret_key          file        2 hours ago  2 hours ago
ae6c9e78e91dea180d22db6ee  hub_settings                file        2 hours ago  2 hours ago
b348d89bfb7d3b18f33a67d07  controller_secret_key       file        2 hours ago  2 hours ago (1)
1the controller secret key

If we want to know where those secrets are stored we can execute

$ podman secret inspect controller_secret_key
[
    {
        "ID": "b348d89bfb7d3b18f33a67d07",
        "CreatedAt": "2026-06-09T10:57:28.278138715Z",
        "UpdatedAt": "2026-06-09T10:57:28.278138715Z",
        "Spec": {
            "Name": "controller_secret_key",
            "Driver": {
                "Name": "file",
                "Options": {
                    "path": "/home/admin/.local/share/containers/storage/secrets/filedriver"
                }
            },
            "Labels": {}
        }
    }
]

Let’s see what’s in the filedriver directory:

$ ls /home/admin/.local/share/containers/storage/secrets/filedriver
secretsdata.json  secretsdata.lock(1)
1we are mainly interested in the secretsdata.json file

So the secretsdata.json file stores all podman secrets that are using the filedriver. For more information see the section Secret Drivers in the podman-secrets-create(1) manpage.

Let’s create a Linux auditd rule to monitor access to this file. As we are using Red Hat Enterprise Linux 10 a more detailed introduction to the Linux audit system is available here.

We created the file /etc/audit/rules.d/aap_secrets.rules with the following content:

-w /home/admin/.local/share/containers/storage/secrets/filedriver/secretsdata.json -p rwa -k aap_secrets (1)
1-p rwa means monitor read/write/attribute changes, -k tells the audit system log events with an additional key "aap_secrets"

To activate the rule we need to load the audit rule via:

# auditctl -R /etc/audit/rules.d/aap_secrets.rules

You can list loaded audit rules via

# auditctl -l
-w /home/admin/.local/share/containers/storage/secrets/filedriver/secretsdata.json -p rwa -k aap_secrets

As a test we can cat the secrets file and check if auditd logged an event:

# cat /home/admin/.local/share/containers/storage/secrets/filedriver/secretsdata.json
...
# grep aap_secrets  /var/log/audit/audit.log
type=SYSCALL msg=audit(1781009319.993:204): arch=c000003e syscall=257 success=yes exit=3 a0=ffffff9c a1=7ffc5276b65c a2=0 a3=0 items=1 ppid=6186 pid=6695 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts2 ses=3 comm="cat" exe="/usr/bin/cat" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="aap_secrets"ARCH=x86_64 SYSCALL=openat AUID="admin" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root" (1)
1We can see that the cat command (comm=cat) was used by the root user to display the file contents

Summary

In this blog post we demonstrated how to configure the Linux audit system to log events if the AAP database secret key is accessed. The audit subsystem can also monitor system calls, for a detailed introduction see Audit system architecture.


Discussion

Previous
Use arrow keys to navigate
Next