Compliance on TechBlog about OpenShift/Ansible/Satellite and much more

[Ep.5] Setup & Configure Compliance Operator

Thu, 25 Apr 2024 00:00:00 +0000

In the previous articles, we have discussed the Git repository folder structure and the configuration of the App-Of-Apps. Now it is time to deploy our first configuration. One of the first things I usually deploy is the Compliance Operator. This Operator is recommended for any cluster and can be deployed without any addition to the Subscription.

In this article, I will describe how it is installed and how the Helm Chart is configured.

Prerequisites

Argo CD (OpenShift GitOps) deployed
App-Of-Apps deployed

Introduction

As a reminder, at Git repository folder structure I described my preferred folder structure. I would like to deploy the Compliance Operator in the Management Cluster now. All my examples can be found at GitHub repository OpenShift Clusterconfig GitOps. The folder clusters/management-cluster/setup-compliance-operator is the one I am interested in.

Inside this folder, you will find another Helm Chart. The Helm Chart has no local templates, instead, it uses dependencies to call other (sub-) charts. However, the values.yaml is the main part to configure everything.

In case you want to have any local template, that you do NOT want to integrate into one of the sub-charts, you can easily do so, by storing them in the templates folder.

Why "empty" Helm Charts?

Actually, it would be possible to use the Helm Chart of the Chart repository directly, without creating a separate chart, that does nothing else than using dependency charts.

The reasons why I am using such an "empty" Chart are the following (in no particular order):

With that way it is possible to add templates (i.e. SealedSecrets) and modify the values-file without packaging and releasing a new Chart version every time you change a small thing.
The Multi-Source Option, which allows you to use a Helm Chart from repository A and a values file from repository B is still a TechPreview feature (Argo CD 2.10). I am using this for the App-of-Apps already, but I did not do this for all charts. This feature is on the list for Argo CD version 2.11 to become globally available.

As an alternative, it is also possible to mix Kustomize and Helm. That way you only need a kustomization.yaml file and reference to a Helm Chart. In the folder clusters/management-cluster/ingresscontroller I have such an example.

Installing Compliance Operator

Analysing Chart.yaml

As any Helm Chart a Chart.yaml file exists, that stores the basic information. The most important ones for now are the dependencies.

The file looks like the following. Three sub-charts are defined as required to deploy and configure the Compliance Operator.

apiVersion: v2
name: setup-compliance-operator
description: Deploy and configure the Compliance Operator
version: 1.0.1
dependencies:
- name: compliance-operator-full-stack (1)
version: ~1.0.0 (2)
repository: https://charts.stderr.at/
- name: helper-operator (3)
version: ~1.0.21
repository: https://charts.stderr.at/
- name: helper-status-checker (4)
version: ~4.0.0
repository: https://charts.stderr.at/
condition: helper-status-checker.enabled (5)

1	Dependency: Compliance Operator Full Stack
2	Version that will be used. The "~" means that the latest version of 1.0.X will be used.
3	Dependency: Helper Operator
4	Dependency: Helper Status Checker
5	Only use this dependency when "enabled" is set

Verify the READMEs of the different Charts for detailed information on how to configure them.

As you can see three other Helm Charts are used to actually deploy and configure the Operator.

Configuration of the Chart

To configure the Compliance Operator, the values files must be prepared accordingly.

The important thing here is, that any value that should be bypassed to a sub-chart is defined under the name of the sub-chart. For example, everything under helper-operator: will be sent to the helper-operator Chart and is used there for its configuration.

The following is a full example of the values I typically use.

# Install Operator Compliance Operator
# Deploys Operator --> Subscription and Operatorgroup
helper-operator:
operators:
compliance-operator:
enabled: true
syncwave: '0'
namespace:
name: openshift-compliance
create: true
subscription:
channel: stable
approval: Automatic
operatorName: compliance-operator
source: redhat-operators
sourceNamespace: openshift-marketplace
operatorgroup:
create: true
notownnamespace: true
# Verify if the Operator has been successfully deployed
helper-status-checker:
enabled: true
checks:
- operatorName: compliance-operator
namespace:
name: openshift-compliance
serviceAccount:
name: "status-checker-compliance"
# Setting for the Compliance Operator
compliance-operator-full-stack:
compliance:
namespace:
name: openshift-compliance
syncwave: '0'
descr: 'Red Hat Compliance'
scansettingbinding:
enabled: true
syncwave: '3'
profiles:
- name: ocp4-cis-node
kind: Profile # Could be Profile or TailedProfile
- name: ocp4-cis
kind: Profile
scansetting: default

Let us walk through the settings in more detail.

Installing the Operator

The first thing to do is to deploy the Operator. Two resources are relevant to install an Operator:

Subscription
OperatorGroup

Both objects should be deployed at the very beginning of Argo CD synchronisation. This is done by setting the Syncwave to 0.

The main settings are the operatorName, the channel (which is the version of the operator) and the approval (which defines if the Operator is updated automatically or manually).

In addition, a Namespace object is deployed, because this Operator should run in its very own namespace.

This will start the Operator installation process.

helper-operator:
operators:
compliance-operator: (1)
enabled: true (2)
syncwave: '0' (3)
namespace:
name: openshift-compliance (4)
create: true
subscription: (5)
channel: stable # Version of the Operator
approval: Automatic # Automatic or Manual
operatorName: compliance-operator # Name of the Operator
source: redhat-operators
sourceNamespace: openshift-marketplace
operatorgroup: (6)
create: true
notownnamespace: true

1	Key that can be freely defined. Theoretically, you can deploy multiple operators at once.
2	Is this Operator enabled yes/no.
3	Syncwave for the Operator deployment. (Subscription and OperatorGroup etc.)
4	The Namespace where the Operator shall be deployed and if this namespace shall be created.
5	Configuration of the Subscription resource.
6	Configuration of the OperatorGroup

Verify the README at Helper Operator to find additional possible configurations.

Verify the Status of the Operator

After Argo CD creates the subscription and operatorgroup resources (and namespace), OpenShift will start the installation of the Operator. This installation will take a while but Argo CD does not see this. All it sees is that the Subscription resource is available and it tries to continue with the configuration of the Operator. Here it will fail because the CRDs are not available yet.

Therefore, I created a mechanism to verify if an Operator is ready or not.

Also verify the separate article Operator Installation with Argo CD that addresses the problem in more detail.

All it does is to start a small Job inside OpenShift and to verify the status of the Operator installation. If everything is fine, the Job will end successfully and Argo CD will continue with the next syncwave. Argo CD Hook and syncwaves are required here. The Job should be started after the Subscription/OperatorGroup resources have been created, which means any syncwave after "0".

The following annotations will be used by the Job:

 argocd.argoproj.io/hook: Sync (1)
argocd.argoproj.io/hook-delete-policy: HookSucceeded (2)
argocd.argoproj.io/sync-wave: {{ .syncwave | default 1 | quote }} (3)

1	Hooks are ways to run scripts before, during, and after a Sync operation.
2	Deletes the OpenShift Job again. The hook resource is deleted after the hook succeeded (e.g. Job/Workflow completed successfully).
3	Syncwave: can be configured. Must be after helper-operator (default 0) and before the Operator is configured further. Default value is 1.

The configuration for hepler_status_checker will look like the following:

# Verify if the Operator has been successfully deployed
helper-status-checker:
enabled: true (1)
checks: (2)
- operatorName: compliance-operator (3)
namespace:
name: openshift-compliance (4)
serviceAccount:
name: "status-checker-compliance" (5)

1	Enable status checker or not. Default: false
2	List of operators to check. Typically, only one is checked, but there could be more.
3	Name of the Operator to check (same as for helper-operator)
4	Namespace where the Operator has been installed (same as for helper-operator)
5	Name of the ServiceAccount that will be created to check the status.

Verify the README at Helper Operator Status Checker to find additional possible configurations.

Configuring Compliance Operator

Finally, the Operator has been deployed and has been verified. Now the time is right to configure the Operator with any configuration we would like. This means, using CRDs to do whatever the Operator offers.

This is reflected in the following part of the values file. All these settings are handed over to the sub-chart compliance-operator-full-stack.

Verify the README at Compliance Operator Chart to find additional possible configurations. Especially, if you like to do Tailored Profiles.

The compliance operator requires a so-called ScanSettingBinding that uses Profiles which are used to check the cluster compliance once a day. In this case, I am using CIS Benchmarks. There are two profiles:

ocp4-cis-node: will check the node operating system for missing but suggested configuration.
ocp4-cis: will check the OpenShift cluster for missing but suggested configuration.

# Setting for the Compliance Operator
compliance-operator-full-stack: (1)
compliance:
namespace:
name: openshift-compliance (2)
syncwave: '0'
descr: 'Red Hat Compliance'
scansettingbinding: (3)
enabled: true
syncwave: '3'
profiles: (4)
- name: ocp4-cis-node
kind: Profile # Could be Profile or TailedProfile
- name: ocp4-cis
kind: Profile
scansetting: default

1	Handing everything that comes below to the sub-chart compliance-operator-full-stack
2	Namespace where the configuration should be deployed. The Syncwave at this point could be omitted.
3	The configuration for the ScanSettingBinding. It is enabled (default = false) and has a Syncwave AFTER the helper-status-checker.
4	The list of profiles that shall be used. These must exist. The Compliance Operator offers several profiles. I usually use these two for full CIS compliance check.

Conclusion

With this configuration, the Compliance Operator will not only be installed but also configured with the same Argo CD Application. All you need to do is to synchronize Argo CD and let the magic happen. After a few minutes, everything should be in sync.

Figure 1. Sync Compliance Operator

Inside OpenShift the Operator is configured and starts doing its job:

Figure 2. Configured Compliance Operator

This concludes the deployment of the Compliance Operator. For further information about the Operator itself, please read the documentation or articles:

Also, be sure to check out the READMEs of the different Charts:

If you have any questions or problems, feel free to create a GitHub issue at any time.

oc compliance command line plugin

Tue, 20 Jul 2021 00:00:00 +0000

As described at Compliance Operator the Compliance Operator can be used to scan the OpenShift cluster environment against security benchmark, like CIS. Fetching the actual results might be a bit tricky tough.

With OpenShift 4.8 plugins to the oc command are allowed. One of these plugin os oc compliance, which allows you to easily fetch scan results, re-run scans and so on. Let’s install and try it out.

Installation

An oc plugin must be deployed into the same directory as the oc command itself.

The following describes the building and installation of the plugin.

You need Go installed on your node.

Clone the Git repository:

git clone https://github.com/openshift/oc-compliance.git

Build and install the plugin

make; make install
go build -o ./bin/oc-compliance ./cmd
which oc | xargs dirname | xargs -n1 cp ./bin/oc-compliance

The plugin allows the use of oc compliance

oc compliance
You must specify a sub-command.
Usage:
oc-compliance [flags]
oc-compliance [command]
Available Commands:
bind Creates a ScanSettingBinding for the given parameters
controls Get a report of what controls you\'re complying with
fetch-fixes Download the fixes/remediations
fetch-raw Download raw compliance results
help Help about any command
rerun-now Force a re-scan for one or more ComplianceScans
view-result View a ComplianceCheckResult
Flags:
-h, --help help for oc-compliance
Use "oc-compliance [command] --help" for more information about a command.

Fetch Raw Results

Without the oc-compliance plugin it was required to manually spin up a Pod and download the results from this Pod, where the PV is mounted. Now, with a simple command we can select the ScanSettingBinding and define an output folder. For example:

oc compliance fetch-raw <object-type> <object-name> -o <output-path>

Assuming the the compliance operator was configured as in the previous article, we have the ScanSettingBinding called cis-compliance:

oc compliance fetch-raw scansettingbindings cis-compliance -n openshift-compliance -o /tmp/

This starts downloading the result archives into /tmp

Fetching results for cis-compliance scans: ocp4-cis-node-worker, ocp4-cis-node-master, ocp4-cis
Fetching raw compliance results for pod 'raw-result-extractor-fxbw8'.Fetching raw compliance results for scan 'ocp4-cis-node-worker'.........
The raw compliance results are avaliable in the following directory: /tmp/ocp4-cis-node-worker
Fetching raw compliance results for pod 'raw-result-extractor-kqrw5'.Fetching raw compliance results for scan 'ocp4-cis-node-master'.....
The raw compliance results are avaliable in the following directory: /tmp/ocp4-cis-node-master
Fetching raw compliance results for pod 'raw-result-extractor-pfrgk'.Fetching raw compliance results for scan 'ocp4-cis'..
The raw compliance results are avaliable in the following directory: /tmp/ocp4-cis
ls -la /tmp/ocp4-cis*
/tmp/ocp4-cis:
total 172
drwx------ 2 root root 4096 Jul 30 16:05 .
drwxrwxrwt. 18 root root 4096 Jul 30 16:05 ..
-rw-r--r-- 1 root root 166676 Jul 30 16:05 ocp4-cis-api-checks-pod.xml.bzip2
/tmp/ocp4-cis-node-master:
total 504
drwx------ 2 root root 4096 Jul 30 16:05 .
drwxrwxrwt. 18 root root 4096 Jul 30 16:05 ..
-rw-r--r-- 1 root root 168256 Jul 30 16:05 ocp4-cis-node-master-master-0-pod.xml.bzip2
-rw-r--r-- 1 root root 165716 Jul 30 16:05 ocp4-cis-node-master-master-1-pod.xml.bzip2
-rw-r--r-- 1 root root 166945 Jul 30 16:05 ocp4-cis-node-master-master-2-pod.xml.bzip2
/tmp/ocp4-cis-node-worker:
total 1112
drwx------ 2 root root 4096 Jul 30 16:05 .
drwxrwxrwt. 18 root root 4096 Jul 30 16:05 ..
-rw-r--r-- 1 root root 154943 Jul 30 16:05 ocp4-cis-node-worker-compute-0-pod.xml.bzip2
-rw-r--r-- 1 root root 154903 Jul 30 16:05 ocp4-cis-node-worker-compute-1-pod.xml.bzip2
-rw-r--r-- 1 root root 154939 Jul 30 16:05 ocp4-cis-node-worker-compute-2-pod.xml.bzip2
-rw-r--r-- 1 root root 154890 Jul 30 16:05 ocp4-cis-node-worker-compute-3-pod.xml.bzip2
-rw-r--r-- 1 root root 168175 Jul 30 16:05 ocp4-cis-node-worker-master-0-pod.xml.bzip2
-rw-r--r-- 1 root root 165603 Jul 30 16:05 ocp4-cis-node-worker-master-1-pod.xml.bzip2
-rw-r--r-- 1 root root 166914 Jul 30 16:05 ocp4-cis-node-worker-master-2-pod.xml.bzip2

Re-Run Scans

Sometimes it is necessary to re-run scans. This can be done by annotating the appropriate scan as described at: Performing a Rescan

With the oc plugin you can simply trigger a re-scan with a single command:

oc compliance rerun-now scansettingbindings <name of scanbinding>

For example:

oc compliance rerun-now scansettingbindings cis-compliance

Example output:

Rerunning scans from 'cis-compliance': ocp4-cis-node-worker, ocp4-cis-node-master, ocp4-cis
Re-running scan 'openshift-compliance/ocp4-cis-node-worker'
Re-running scan 'openshift-compliance/ocp4-cis-node-master'
Re-running scan 'openshift-compliance/ocp4-cis'

With the command oc get compliancescan -n openshift-compliance you can check when the scan has been done:

NAME PHASE RESULT
ocp4-cis RUNNING NOT-AVAILABLE
ocp4-cis-node-master RUNNING NOT-AVAILABLE
ocp4-cis-node-worker AGGREGATING NOT-AVAILABLE

View Results on CLI

Once a scan process has finished you can verify the check results quick and easy using the command line:

oc get ComplianceCheckResult -A

This prints for example:

NAMESPACE NAME STATUS SEVERITY
[...]
openshift-compliance ocp4-cis-audit-log-forwarding-enabled FAIL medium
[...]

The view-result can print a human readable output, for example:

oc compliance view-result ocp4-cis-audit-log-forwarding-enabled -n openshift-compliance

Example:

+----------------------+-----------------------------------------------------------------------------------------+
| KEY | VALUE |
+----------------------+-----------------------------------------------------------------------------------------+
| Title | Ensure that Audit Log |
| | Forwarding Is Enabled |
+----------------------+-----------------------------------------------------------------------------------------+
| Status | FAIL |
+----------------------+-----------------------------------------------------------------------------------------+
| Severity | medium |
+----------------------+-----------------------------------------------------------------------------------------+
| Description | OpenShift audit works at the |
| | API server level, logging |
| | all requests coming to the |
| | server. Audit is on by default |
| | and the best practice is |
| | to ship audit logs off the |
| | cluster for retention. The |
| | cluster-logging-operator is |
| | able to do this with the |
| | |
| | |
| | |
| | ClusterLogForwarders |
| | |
| | |
| | |
| | resource. The forementioned resource can be configured to logs to different third party |
| | systems. For more information on this, please reference the official documentation: |
| | https://docs.openshift.com/container-platform/4.6/logging/cluster-logging-external.html |
+----------------------+-----------------------------------------------------------------------------------------+
| Rationale | Retaining logs ensures the |
| | ability to go back in time to |
| | investigate or correlate any |
| | events. Offloading audit logs |
| | from the cluster ensures that |
| | an attacker that has access |
| | to the cluster will not be |
| | able to tamper with the logs |
| | because of the logs being |
| | stored off-site. |
+----------------------+-----------------------------------------------------------------------------------------+
| Instructions | Run the following command: |
| | |
| | oc get clusterlogforwarders |
| | instance -n openshift-logging |
| | -ojson | jq -r |
| | '.spec.pipelines[].inputRefs | |
| | contains(["audit"])' |
| | |
| | The output should return true. |
+----------------------+-----------------------------------------------------------------------------------------+
| CIS-OCP Controls | 1.2.23 |
+----------------------+-----------------------------------------------------------------------------------------+
| NIST-800-53 Controls | AC-2(12), AU-6, AU-6(1), |
| | AU-6(3), AU-9(2), SI-4(16), |
| | AU-4(1), AU-11, AU-7, AU-7(1) |
+----------------------+-----------------------------------------------------------------------------------------+
| Available Fix | No |
+----------------------+-----------------------------------------------------------------------------------------+
| Result Object Name | ocp4-cis-audit-log-forwarding-enabled |
+----------------------+-----------------------------------------------------------------------------------------+
| Rule Object Name | ocp4-audit-log-forwarding-enabled |
+----------------------+-----------------------------------------------------------------------------------------+
| Remediation Created | No |
+----------------------+-----------------------------------------------------------------------------------------+

Compliance Operator

Mon, 19 Jul 2021 00:00:00 +0000

OpenShift comes out of the box with a highly secure operating system, called Red Hat CoreOS. This OS is immutable, which means that no direct changes are done inside the OS, instead any configuration is managed by OpenShift itself using MachineConfig objects. Nevertheless, hardening certain settings must still be considered. Red Hat released a hardening guide (CIS Benchmark) which can be downloaded at https://www.cisecurity.org/.

However, an automated way to perform such checks would be nice too. To achieve this the Compliance Operator can be leveraged, which runs an OpenSCAP check to create reports of the clusters is compliant or as the official documentation describes:

The Compliance Operator lets OpenShift Container Platform administrators describe the desired compliance state of a cluster and provides them with an overview of gaps and ways to remediate them. The Compliance Operator assesses compliance of both the Kubernetes API resources of OpenShift Container Platform, as well as the nodes running the cluster. The Compliance Operator uses OpenSCAP, a NIST-certified tool, to scan and enforce security policies provided by the content.

This article shall show how to quickly install the operator and retrieve the first result. It is not a full documentation, which is written by other people at: Compliance Operator, especially remediation is not covered here.

As prerequisites we have:

Installed OpenShift 4.6+ cluster

The Compliance Operator is available for Red Hat Enterprise Linux CoreOS (RHCOS) deployments only.

Install the Compliance Operator

The easiest way to deploy the Compliance Operator is by searching the OperatorHub which is available inside OpenShift.

Figure 1. Install Compliance Operator

Keep the default settings and wait until the operator has been installed.

Figure 2. Install Compliance Operator

Custom Resources (CRDs)

The operator brings a ton of new CRDs into the system:

ScanSetting … defines when and on which roles (worker, master …) a check shall be executed. It also defines a persistent volume (PV) to store the scan results. Two ScanSettings are created during the installation:
- default: just scans without automatically apply changes
- default-auto-apply: can automatically remediate without extra steps
ScanSettingBinding … binds one or more profiles to a scan
Profile … Represent different compliance benchmarks with a set of rules. For this blog we will use CIS Benchmark profiles
ProfileBundle … Bundles a security image, which is later used by Profiles.
Rule … Rules which are used by profiles to verify the state of the cluster.
TailoredProfile … Customized profile
ComplianceScan … scans which have been performed
ComplianceCheckResult … The results of a scan. Each ComplianceCheckResult represents the result of one compliance rule check
ComplianceRemediation … If a rule ca be remediated automatically, this object is created.

Create a ScanBinding object

The first step to do is to create a ScanBiding objects. (We reuse the default ScanSetting)

Let’s create the following object, which is using the profiles ocp4-cis and ocp4-cis-node

apiVersion: compliance.openshift.io/v1alpha1
kind: ScanSettingBinding
metadata:
name: cis-compliance
profiles:
- name: ocp4-cis-node (1)
kind: Profile
apiGroup: compliance.openshift.io/v1alpha1
- name: ocp4-cis (2)
kind: Profile
apiGroup: compliance.openshift.io/v1alpha1
settingsRef:
name: default (3)
kind: ScanSetting
apiGroup: compliance.openshift.io/v1alpha1

1	use the profile ocp4-cis-node
2	use the profile ocp4-cis
3	reference to the default scansetting

As soon as the object is created the cluster is scan is started. The objects ComplianceSuite and ComplianceScan are created automatically and will eventually reach the phase "DONE" when the scan is completed.

The following command will show the results of the scans

oc get compliancescan -n openshift-compliance
NAME PHASE RESULT
ocp4-cis DONE NON-COMPLIANT
ocp4-cis-node-master DONE NON-COMPLIANT
ocp4-cis-node-worker DONE INCONSISTENT

Three different checks have been done. One overall cluster check and 2 separated for master and worker nodes.

As we used the default ScanSetting the next check will run a 1 am.

Profiles

The operator comes with a set of standard profiles which represent different compliance benchmarks.

To view available profiles:

oc get profiles.compliance -n openshift-compliance

NAME AGE
ocp4-cis 28m
ocp4-cis-node 28m
ocp4-e8 28m
ocp4-moderate 28m
rhcos4-e8 28m
rhcos4-moderate 28m

Each profile contains a description which explains the intention and a list of rules which used in this profile.

For example the profile 'ocp4-cis-node' used above is containing:

oc get profiles.compliance -n openshift-compliance -oyaml ocp4-cis-node
# Output
description: This profile defines a baseline that aligns to the Center for Internet Security® Red
Hat OpenShift Container Platform 4 Benchmark™, V0.3, currently unreleased. This profile includes
Center for Internet Security® Red Hat OpenShift Container Platform 4 CIS Benchmarks™ content.
Note that this part of the profile is meant to run on the Operating System that Red Hat
OpenShift Container Platform 4 runs on top of. This profile is applicable to OpenShift versions
4.6 and greater.
[...]
name: ocp4-cis-node
namespace: openshift-compliance
[...]
rules:
- ocp4-etcd-unique-ca
- ocp4-file-groupowner-cni-conf
- ocp4-file-groupowner-controller-manager-kubeconfig
- ocp4-file-groupowner-etcd-data-dir
- ocp4-file-groupowner-etcd-data-files
- ocp4-file-groupowner-etcd-member
- ocp4-file-groupowner-etcd-pki-cert-files
- ocp4-file-groupowner-ip-allocations
[...]

Like the profiles the different rules can be inspected:

oc get rules.compliance -n openshift-compliance ocp4-file-groupowner-etcd-member
-o jsonpath='{"Title: "}{.title}{"\nDescription: \n"}{.description}'
# Output
Title: Verify Group Who Owns The etcd Member Pod Specification File
Description:
To properly set the group owner of /etc/kubernetes/static-pod-resources/etcd-pod-*/etcd-pod.yaml ,
run the command:
$ sudo chgrp root /etc/kubernetes/static-pod-resources/etcd-pod-*/etcd-pod.yaml

Profile Customization

Sometimes is it required to modify (tailor) a profile to fit specific needs. With the TailoredProfile object it is possible to enable or disable rules.

In this blog, I just want to share a quick example from the official documentaiton: https://docs.openshift.com/container-platform/4.7/security/compliance_operator/compliance-operator-tailor.html

The following TailoredProfile disables 2 rules and sets a value for another rule:

apiVersion: compliance.openshift.io/v1alpha1
kind: TailoredProfile
metadata:
name: nist-moderate-modified
spec:
extends: rhcos4-moderate
title: My modified NIST moderate profile
disableRules:
- name: rhcos4-file-permissions-node-config
rationale: This breaks X application.
- name: rhcos4-account-disable-post-pw-expiration
rationale: No need to check this as it comes from the IdP
setValues:
- name: rhcos4-var-selinux-state
rationale: Organizational requirements
value: permissive

Working with scan results

Once a scan finished you probably want to see what the status of the scan is.

As you sse above the cluster failed to be compliant.

oc get compliancescan -n openshift-compliance
NAME PHASE RESULT
ocp4-cis DONE NON-COMPLIANT
ocp4-cis-node-master DONE NON-COMPLIANT
ocp4-cis-node-worker DONE INCONSISTENT

Retrieving results via oc command

List all results which can be remediated automatically:

oc get compliancecheckresults -l 'compliance.openshift.io/check-status=FAIL,compliance.openshift.io/automated-remediation' -n openshift-compliance
NAME STATUS SEVERITY
ocp4-cis-api-server-encryption-provider-cipher FAIL medium
ocp4-cis-api-server-encryption-provider-config FAIL medium

Further information about remediation can be found at: Compliance Operator Remediation

List all results which cannot be remediated automatically and must be fixed manually instead:

oc get compliancecheckresults -l 'compliance.openshift.io/check-status=FAIL,!compliance.openshift.io/automated-remediation' -n openshift-compliance
NAME STATUS SEVERITY
ocp4-cis-audit-log-forwarding-enabled FAIL medium
ocp4-cis-file-permissions-proxy-kubeconfig FAIL medium
ocp4-cis-node-master-file-groupowner-ip-allocations FAIL medium
ocp4-cis-node-master-file-groupowner-openshift-sdn-cniserver-config FAIL medium
ocp4-cis-node-master-file-owner-ip-allocations FAIL medium
ocp4-cis-node-master-file-owner-openshift-sdn-cniserver-config FAIL medium
ocp4-cis-node-master-kubelet-configure-event-creation FAIL medium
ocp4-cis-node-master-kubelet-configure-tls-cipher-suites FAIL medium
ocp4-cis-node-master-kubelet-enable-protect-kernel-defaults FAIL medium
ocp4-cis-node-master-kubelet-eviction-thresholds-set-hard-imagefs-available FAIL medium
ocp4-cis-node-master-kubelet-eviction-thresholds-set-hard-imagefs-inodesfree FAIL medium
ocp4-cis-node-master-kubelet-eviction-thresholds-set-hard-memory-available FAIL medium
ocp4-cis-node-master-kubelet-eviction-thresholds-set-hard-nodefs-available FAIL medium
ocp4-cis-node-master-kubelet-eviction-thresholds-set-hard-nodefs-inodesfree FAIL medium
ocp4-cis-node-master-kubelet-eviction-thresholds-set-soft-imagefs-available FAIL medium
ocp4-cis-node-master-kubelet-eviction-thresholds-set-soft-imagefs-inodesfree FAIL medium
ocp4-cis-node-master-kubelet-eviction-thresholds-set-soft-memory-available FAIL medium
ocp4-cis-node-master-kubelet-eviction-thresholds-set-soft-nodefs-available FAIL medium
ocp4-cis-node-master-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree FAIL medium
ocp4-cis-node-worker-file-groupowner-ip-allocations FAIL medium
ocp4-cis-node-worker-file-groupowner-openshift-sdn-cniserver-config FAIL medium
ocp4-cis-node-worker-file-owner-ip-allocations FAIL medium
ocp4-cis-node-worker-file-owner-openshift-sdn-cniserver-config FAIL medium
ocp4-cis-node-worker-kubelet-configure-event-creation FAIL medium
ocp4-cis-node-worker-kubelet-configure-tls-cipher-suites FAIL medium
ocp4-cis-node-worker-kubelet-enable-protect-kernel-defaults FAIL medium
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-hard-imagefs-available FAIL medium
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-hard-imagefs-inodesfree FAIL medium
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-hard-memory-available FAIL medium
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-hard-nodefs-available FAIL medium
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-hard-nodefs-inodesfree FAIL medium
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-imagefs-available FAIL medium
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-imagefs-inodesfree FAIL medium
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-memory-available FAIL medium
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-nodefs-available FAIL medium
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree FAIL medium

Retrieving RAW results

Let’s first retrieve the raw result of the scan. For each of the ComplianceScans a volume claim (PVC) is created to store he results. We can use a Pod to mount the volume to download the scan results.

The following PVC have been created on our example:

oc get pvc -n openshift-compliance
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
ocp4-cis Bound pvc-cc026ae3-2f42-4e19-bc55-016c6dd31d22 1Gi RWO managed-nfs-storage 4h17m
ocp4-cis-node-master Bound pvc-3bd47c5e-2008-4759-9d53-ba41b568688d 1Gi RWO managed-nfs-storage 4h17m
ocp4-cis-node-worker Bound pvc-77200e5f-0f15-410c-a4ee-f2fb3e316f84 1Gi RWO managed-nfs-storage 4h17m

Now we can create a Pod which mounts all PVCs at once:

apiVersion: "v1"
kind: Pod
metadata:
name: pv-extract
namespace: openshift-compliance
spec:
containers:
- name: pv-extract-pod
image: registry.access.redhat.com/ubi8/ubi
command: ["sleep", "3000"]
volumeMounts: (1)
- mountPath: "/workers-scan-results"
name: workers-scan-vol
- mountPath: "/masters-scan-results"
name: masters-scan-vol
- mountPath: "/ocp4-scan-results"
name: ocp4-scan-vol
volumes: (2)
- name: workers-scan-vol
persistentVolumeClaim:
claimName: ocp4-cis-node-worker
- name: masters-scan-vol
persistentVolumeClaim:
claimName: ocp4-cis-node-master
- name: ocp4-scan-vol
persistentVolumeClaim:
claimName: ocp4-cis

1	mount paths
2	volumesclaims to mount

This creates a Pod with the PVCs mounted inside:

sh-4.4# ls -la | grep scan
drwxrwxrwx. 3 root root 4096 Jul 20 05:20 master-scan-results
drwxrwxrwx. 3 root root 4096 Jul 20 05:20 ocp4-scan-results
drwxrwxrwx. 3 root root 4096 Jul 20 05:20 workers-scan-results

We can download the result-files to our local machine for further auditing. Therefore, we create the folder scan_results in which we copy everything:

mkdir scan-results; cd scan-results
oc -n openshift-compliance cp pv-extract:ocp4-scan-results ocp4-scan-results/.
oc -n openshift-compliance cp pv-extract:workers-scan-results workers-scan-results/.
oc -n openshift-compliance cp pv-extract:masters-scan-results masters-scan-results/.

This will download several bzip2 archives for the appropriate scan result.

Once done, you can delete the "download pod" using: oc delete pod pv-extract -n openshift-compliance

Work wth RAW results

So above section described the download of the bzip2 files but what to do with it? First, you can import it into a tool which is able to read openScap reports. Or, secondly, you can use the oscap command to create a html output.

We have downloaded the following files:

./ocp4-scan-results/0/ocp4-cis-api-checks-pod.xml.bzip2
./masters-scan-results/0/ocp4-cis-node-master-master-0-pod.xml.bzip2
./masters-scan-results/0/ocp4-cis-node-master-master-2-pod.xml.bzip2
./masters-scan-results/0/ocp4-cis-node-master-master-1-pod.xml.bzip2
./workers-scan-results/0/ocp4-cis-node-worker-compute-0-pod.xml.bzip2
./workers-scan-results/0/ocp4-cis-node-worker-compute-1-pod.xml.bzip2
./workers-scan-results/0/ocp4-cis-node-worker-compute-3-pod.xml.bzip2
./workers-scan-results/0/ocp4-cis-node-worker-compute-2-pod.xml.bzip2

To create the html output (be sure that open-scap is installed on you host):

mkdir html
oscap xccdf generate report ocp4-scan-results/0/ocp4-cis-api-checks-pod.xml.bzip2 >> html/ocp4-cis-api-checks.html
oscap xccdf generate report masters-scan-results/0/ocp4-cis-node-master-master-0-pod.xml.bzip2 >> html/ocp4-cis-node-master-master-0.html
oscap xccdf generate report masters-scan-results/0/ocp4-cis-node-master-master-1-pod.xml.bzip2 >> html/ocp4-cis-node-master-master-1.html
oscap xccdf generate report masters-scan-results/0/ocp4-cis-node-master-master-2-pod.xml.bzip2 >> html/ocp4-cis-node-master-master-2.html
oscap xccdf generate report workers-scan-results/0/ocp4-cis-node-worker-compute-0-pod.xml.bzip2 >> html/ocp4-cis-node-worker-compute-0.html
...

The resulted html files are too big to be show here, but some snippets should give an overview:

To view the html output as an example I have linked the html files:

Overall Scoring of the result:

Figure 3. Scoring

A list if passed or failed checks:

Figure 4. Scan Result list

Scan details with a link to the CIS Benchmark section and further explainations on how to fix the issue:

Figure 5. Scan details

Performing a rescan

If it is necessary to run a rescan, the ComplianceScan object is simply annotated with:

oc annotate compliancescans/<scan_name> compliance.openshift.io/rescan=

If default-auto-apply is enabled, remediation which changes MachineConfigs will trigger a cluster reboot.