Setup & Configure Compliance Operator using GitOps

OpenShift GitOps Security Compliance

- Thomas Jungbauer Thomas Jungbauer ( Lastmod: 2024-05-08 ) - 7 min read

In the previous articles, we have discussed the Git repository folder structure and the configuration of the App-Of-Apps. Now it is time to deploy our first configuration. One of the first things I usually deploy is the Compliance Operator. This Operator is recommended for any cluster and can be deployed without any addition to the Subscription.

In this article, I will describe how it is installed and how the Helm Chart is configured.

Prerequisites

  1. Argo CD (OpenShift GitOps) deployed

  2. App-Of-Apps deployed

Introduction

As a reminder, at Git repository folder structure I described my preferred folder structure. I would like to deploy the Compliance Operator in the Management Cluster now. All my examples can be found at GitHub repository OpenShift Clusterconfig GitOps. The folder clusters/management-cluster/setup-compliance-operator is the one I am interested in.

Inside this folder, you will find another Helm Chart. The Helm Chart has no local templates, instead, it uses dependencies to call other (sub-) charts. However, the values.yaml is the main part to configure everything.

In case you want to have any local template, that you do NOT want to integrate into one of the sub-charts, you can easily do so, by storing them in the templates folder.

Why "empty" Helm Charts?

Actually, it would be possible to use the Helm Chart of the Chart repository directly, without creating a separate chart, that does nothing else than using dependency charts.

The reasons why I am using such an "empty" Chart are the following (in no particular order):

  1. With that way it is possible to add templates (i.e. SealedSecrets) and modify the values-file without packaging and releasing a new Chart version every time you change a small thing.

  2. The Multi-Source Option, which allows you to use a Helm Chart from repository A and a values file from repository B is still a TechPreview feature (Argo CD 2.10). I am using this for the App-of-Apps already, but I did not do this for all charts. This feature is on the list for Argo CD version 2.11 to become globally available.

As an alternative, it is also possible to mix Kustomize and Helm. That way you only need a kustomization.yaml file and reference to a Helm Chart. In the folder clusters/management-cluster/ingresscontroller I have such an example.

Installing Compliance Operator

Analysing Chart.yaml

As any Helm Chart a Chart.yaml file exists, that stores the basic information. The most important ones for now are the dependencies.

The file looks like the following. Three sub-charts are defined as required to deploy and configure the Compliance Operator.

apiVersion: v2
name: setup-compliance-operator
description: Deploy and configure the Compliance Operator
version: 1.0.1
dependencies:
  - name: compliance-operator-full-stack (1)
    version: ~1.0.0 (2)
    repository: https://charts.stderr.at/
  - name: helper-operator (3)
    version: ~1.0.21
    repository: https://charts.stderr.at/
  - name: helper-status-checker (4)
    version: ~4.0.0
    repository: https://charts.stderr.at/
    condition: helper-status-checker.enabled (5)
1Dependency: Compliance Operator Full Stack
2Version that will be used. The "~" means that the latest version of 1.0.X will be used.
3Dependency: Helper Operator
4Dependency: Helper Status Checker
5Only use this dependency when "enabled" is set
Verify the READMEs of the different Charts for detailed information on how to configure them.

As you can see three other Helm Charts are used to actually deploy and configure the Operator.

Configuration of the Chart

To configure the Compliance Operator, the values files must be prepared accordingly.

The important thing here is, that any value that should be bypassed to a sub-chart is defined under the name of the sub-chart. For example, everything under helper-operator: will be sent to the helper-operator Chart and is used there for its configuration.

The following is a full example of the values I typically use.

# Install Operator Compliance Operator
# Deploys Operator --> Subscription and Operatorgroup
helper-operator:
  operators:
    compliance-operator:
      enabled: true
      syncwave: '0'
      namespace:
        name: openshift-compliance
        create: true
      subscription:
        channel: stable
        approval: Automatic
        operatorName: compliance-operator
        source: redhat-operators
        sourceNamespace: openshift-marketplace
      operatorgroup:
        create: true
        notownnamespace: true

# Verify if the Operator has been successfully deployed
helper-status-checker:
  enabled: true

  checks:
    - operatorName: compliance-operator
      namespace:
        name: openshift-compliance
      serviceAccount:
        name: "status-checker-compliance"

# Setting for the Compliance Operator
compliance-operator-full-stack:
  compliance:
    namespace:
      name: openshift-compliance
      syncwave: '0'
      descr: 'Red Hat Compliance'
    scansettingbinding:
      enabled: true
      syncwave: '3'

      profiles:
        - name: ocp4-cis-node
          kind: Profile  # Could be Profile or TailedProfile
        - name: ocp4-cis
          kind: Profile
      scansetting: default

Let us walk through the settings in more detail.

Installing the Operator

The first thing to do is to deploy the Operator. Two resources are relevant to install an Operator:

  1. Subscription

  2. OperatorGroup

Both objects should be deployed at the very beginning of Argo CD synchronisation. This is done by setting the Syncwave to 0.

The main settings are the operatorName, the channel (which is the version of the operator) and the approval (which defines if the Operator is updated automatically or manually).

In addition, a Namespace object is deployed, because this Operator should run in its very own namespace.

This will start the Operator installation process.

helper-operator:
  operators:
    compliance-operator: (1)
      enabled: true (2)
      syncwave: '0' (3)
      namespace:
        name: openshift-compliance (4)
        create: true
      subscription: (5)
        channel: stable # Version of the Operator
        approval: Automatic # Automatic or Manual
        operatorName: compliance-operator # Name of the Operator
        source: redhat-operators
        sourceNamespace: openshift-marketplace
      operatorgroup: (6)
        create: true
        notownnamespace: true
1Key that can be freely defined. Theoretically, you can deploy multiple operators at once.
2Is this Operator enabled yes/no.
3Syncwave for the Operator deployment. (Subscription and OperatorGroup etc.)
4The Namespace where the Operator shall be deployed and if this namespace shall be created.
5Configuration of the Subscription resource.
6Configuration of the OperatorGroup
Verify the README at Helper Operator to find additional possible configurations.

Verify the Status of the Operator

After Argo CD creates the subscription and operatorgroup resources (and namespace), OpenShift will start the installation of the Operator. This installation will take a while but Argo CD does not see this. All it sees is that the Subscription resource is available and it tries to continue with the configuration of the Operator. Here it will fail because the CRDs are not available yet.

Therefore, I created a mechanism to verify if an Operator is ready or not.

Also verify the separate article Operator Installation with Argo CD that addresses the problem in more detail.

All it does is to start a small Job inside OpenShift and to verify the status of the Operator installation. If everything is fine, the Job will end successfully and Argo CD will continue with the next syncwave. Argo CD Hook and syncwaves are required here. The Job should be started after the Subscription/OperatorGroup resources have been created, which means any syncwave after "0".

The following annotations will be used by the Job:

    argocd.argoproj.io/hook: Sync (1)
    argocd.argoproj.io/hook-delete-policy: HookSucceeded (2)
    argocd.argoproj.io/sync-wave: {{ .syncwave | default 1 | quote }} (3)
1Hooks are ways to run scripts before, during, and after a Sync operation.
2Deletes the OpenShift Job again. The hook resource is deleted after the hook succeeded (e.g. Job/Workflow completed successfully).
3Syncwave: can be configured. Must be after helper-operator (default 0) and before the Operator is configured further. Default value is 1.

The configuration for hepler_status_checker will look like the following:

# Verify if the Operator has been successfully deployed
helper-status-checker:
  enabled: true (1)

  checks: (2)
    - operatorName: compliance-operator (3)
      namespace:
        name: openshift-compliance (4)
      serviceAccount:
        name: "status-checker-compliance" (5)
1Enable status checker or not. Default: false
2List of operators to check. Typically, only one is checked, but there could be more.
3Name of the Operator to check (same as for helper-operator)
4Namespace where the Operator has been installed (same as for helper-operator)
5Name of the ServiceAccount that will be created to check the status.
Verify the README at Helper Operator Status Checker to find additional possible configurations.

Configuring Compliance Operator

Finally, the Operator has been deployed and has been verified. Now the time is right to configure the Operator with any configuration we would like. This means, using CRDs to do whatever the Operator offers.

This is reflected in the following part of the values file. All these settings are handed over to the sub-chart compliance-operator-full-stack.

Verify the README at Compliance Operator Chart to find additional possible configurations. Especially, if you like to do Tailored Profiles.

The compliance operator requires a so-called ScanSettingBinding that uses Profiles which are used to check the cluster compliance once a day. In this case, I am using CIS Benchmarks. There are two profiles:

  1. ocp4-cis-node: will check the node operating system for missing but suggested configuration.

  2. ocp4-cis: will check the OpenShift cluster for missing but suggested configuration.

# Setting for the Compliance Operator
compliance-operator-full-stack: (1)
  compliance:
    namespace:
      name: openshift-compliance (2)
      syncwave: '0'
      descr: 'Red Hat Compliance'
    scansettingbinding: (3)
      enabled: true
      syncwave: '3'

      profiles: (4)
        - name: ocp4-cis-node
          kind: Profile  # Could be Profile or TailedProfile
        - name: ocp4-cis
          kind: Profile
      scansetting: default
1Handing everything that comes below to the sub-chart compliance-operator-full-stack
2Namespace where the configuration should be deployed. The Syncwave at this point could be omitted.
3The configuration for the ScanSettingBinding. It is enabled (default = false) and has a Syncwave AFTER the helper-status-checker.
4The list of profiles that shall be used. These must exist. The Compliance Operator offers several profiles. I usually use these two for full CIS compliance check.

Conclusion

With this configuration, the Compliance Operator will not only be installed but also configured with the same Argo CD Application. All you need to do is to synchronize Argo CD and let the magic happen. After a few minutes, everything should be in sync.

Sync Compliance Operator
Figure 1. Sync Compliance Operator

Inside OpenShift the Operator is configured and starts doing its job:

Configured Compliance Operator
Figure 2. Configured Compliance Operator

This concludes the deployment of the Compliance Operator. For further information about the Operator itself, please read the documentation or articles:

  1. Official Documentation: Compliance Operator

  2. Blog: Compliance Operator

Also, be sure to check out the READMEs of the different Charts:

  1. Helper Operator

  2. Helper Operator Status Checker

  3. Compliance Operator Chart

  4. Compliance Operator Setup

If you have any questions or problems, feel free to create a GitHub issue at any time.

See Also:

  • Introducing AdminNetworkPolicies –

    Classic Kubernetes/OpenShift offer a feature called NetworkPolicy that allows users to control the traffic to and from their assigned Namespace. NetworkPolicies are designed to give project owners or tenants the ability to protect their own namespace. Sometimes, however, I worked with customers where the cluster administrators or a dedicated (network) team need to enforce these policies. Since the NetworkPolicy API is namespace-scoped, it is not possible to enforce policies across namespaces. The only solution was to create custom (project) admin and edit roles, and remove the ability of creating, modifying or deleting NetworkPolicy objects. Technically, this is possible and easily done. But shifts the whole network security to cluster administrators. Luckily, this is where AdminNetworkPolicy (ANP) and BaselineAdminNetworkPolicy (BANP) comes into play.

  • Using Kustomize to post render a Helm Chart –

    Lately I came across several issues where a given Helm Chart must be modified after it has been rendered by Argo CD. Argo CD does a helm template to render a Chart. Sometimes, especially when you work with Subcharts or when a specific setting is not yet supported by the Chart, you need to modify it later …​ you need to post-render the Chart. In this very short article, I would like to demonstrate this on a real-live example I had to do. I would like to inject annotations to a Route objects, so that the certificate can be injected. This is done by the cert-utils operator. For the post-rendering the Argo CD repo pod will be extended with a sidecar container, that is watching for the repos and patches them if required.

  • Managing Certificates using GitOps approach –

    The article SSL Certificate Management for OpenShift on AWS explains how to use the Cert-Manager Operator to request and install a new SSL Certificate. This time, I would like to leverage the GitOps approach using the Helm Chart cert-manager I have prepared to deploy the Operator and order new Certificates. I will use an ACME Letsencrypt issuer with a DNS challenge. My domain is hosted at AWS Route 53. However, any other integration can be easily used.

  • Update Cluster Version using GitOps approach –

    During a GitOps journey at one point, the question arises, how to update a cluster? Nowadays it is very easy to update a cluster using CLI or WebUI, so why bother with GitOps in that case? The reason is simple: Using GitOps you can be sure that all clusters are updated to the correct, required version and the version of each cluster is also managed in Git. All you need is the channel you want to use and the desired cluster version. Optionally, you can define the exact image SHA. This might be required when you are operating in a restricted environment.

  • Multiple Sources for Applications in Argo CD –

    Argo CD or OpenShift GitOps uses Applications or ApplicationSets to define the relationship between a source (Git) and a cluster. Typically, this is a 1:1 link, which means one Application is using one source to compare the cluster status. This can be a limitation. For example, if you are working with Helm Charts and a Helm repository, you do not want to re-build (or re-release) the whole chart just because you made a small change in the values file that is packaged into the repository. You want to separate the configuration of the chart with the Helm package. The most common scenarios for multiple sources are (see: Argo CD documentation): Your organization wants to use an external/public Helm chart You want to override the Helm values with your own local values You don’t want to clone the Helm chart locally as well because that would lead to duplication and you would need to monitor it manually for upstream changes. This small article describes three different ways with a working example and tries to cover the advantages and disadvantages of each of them. They might be opinionated but some of them proved to be easier to use and manage.

Copyright © 2020 - 2024 Toni Schmidbauer & Thomas Jungbauer