Step 6 - Scanning with ACS

OpenShift Supply Chain

- Thomas Jungbauer Thomas Jungbauer ( Lastmod: 2024-05-08 ) - 8 min read

In this step we will install Advanced Cluster Security (ACS) and create 2 new steps in our Pipeline to scan the image for vulnerabilities and security policy. A custom security policy, configured in ACS, will verify if the image is signed.

Goals

The goals of this step are:

  • Install ACS.

  • Create a custom security policy into ACS.

  • Configure CoSign integration in ACS.

  • Create a step in the Pipeline to perform an image scan (vulnerabilities).

  • Create a step in the Pipeline to perform an image check (security policies).

Install Advanced Cluster Security (ACS)

We will install ACS on our cluster which will be called local-cluster inside ACS.

The deployment will happen via Argo CD (OpenShift GitOps) that will use the following Helm Chart: RHACS

The Argo CD Application will trigger the deployment which will take several minutes.

The deployment will install a minimal version of ACS and tries to limit the required resources and replicas. For production environments, you probably want to increase the settings.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: in-cluster-init-rhacs
  namespace: openshift-gitops
spec:
  destination:
    namespace: default
    server: 'https://kubernetes.default.svc' (1)
  info:
    - name: Description
      value: >-
        Initialize Red Hat Advanced Cluster Security and deploy Central and
        SecuredCluster
  project: in-cluster
  source:
    path: charts/rhacs-full-stack (2)
    repoURL: 'https://github.com/tjungbauer/helm-charts' (3)
    targetRevision: main
1Installing on the local cluster.
2Path to the Helm chart.
3URL to the Helm chart repository.
Syncing ACS Deployment Application
Figure 1. Syncing ACS Deployment Application

This GitOps Application will start a full-stack ACS deployment. It does (in this exact order):

  1. Installs the ACS Operator into rhacs-operator

  2. Verifies if the Operator is ready

  3. Creates the Namespace stackrox

  4. Adds a ConsoleLink in the upper right action menu of OpenShift

    ACS ConsoleLine
    Figure 2. ACS ConsoleLink

  5. Creates the Central custom resource with minimum resources

  6. Creates an init-bundle to add the first (local) cluster

  7. Adds the custom resource SecureCluster to install the rest of the ACS components

  8. And finally, runs a basic configuration that enables authentication via OpenShift and provides the kubeadmin user admin privileges. This can be disabled in the values file if you prefer not to configure this, or if kubeadmin does not exist anymore in your cluster.

The whole process will take a while and uses syncwaves and hooks.

Configure ACS Integration

Generate Authentication Token

  1. Create an authentication Token in ACS

    Go to "Platform Configuration > Integrations > API Token"

    ACS Generate Token
    Figure 3. ACS Generate Token

    Generate a new Token with at least Continuous Integration privileges and save the created Token.

  2. Back in OpenShift, a Secret must be created with the key rox_api_token

    kind: Secret
apiVersion: v1
metadata:
  name: stackrox-secret
  namespace: ci
data:
  rox_api_token: <base 64 Token> (1)
type: Opaque
    1Base64 decoded ACS Token.

  3. Create a 2nd Secret that contains the ACS endpoint and its port:

    kind: Secret
apiVersion: v1
metadata:
  name: stackrox-endpoint
  namespace: ci
stringData:
  rox_central_endpoint: central-stackrox.apps.ocp.aws.ispworld.at:443 (1)
type: Opaque
    1The endpoint URL of my example cluster. Note: the port MUST be added here.

Integrate CoSign with ACS

  1. Configure CoSign Integration

    During Step 5 we have created a CoSign key pair to sign our images. To integrate with ACS, we need to retrieve the public key. In OpenShift, open the Secret signing-secrets in the Namespace openshift-pipelines and extract the key cosign.pub.

    It will look something like tjis:

    -----BEGIN PUBLIC KEY-----
key...
-----END PUBLIC KEY-----

  2. In ACS, go to "Platform Configuration > Integrations > Signature" and create a new CoSign integration.

    ACS CoSign Integration
    Figure 4. ACS CoSign Integration

    Enter a name and the public key and activate this integration

    ACS Create CoSign Integration
    Figure 5. ACS Create CoSign Integration

    This will enable ACS to verify the CoSign signature of the image.

Create a Custom Policy that verifies the Signature

Create a custom security policy, that verifies if our image has been signed or not.

To be sure that every image is correctly signed, we create a custom security policy that verifies this signature using our CoSign integration. This policy can be configured as inform or enforce. Enforce means that the pipeline will fail (during the Task acs-image-check) if the image signature cannot be checked.

Let’s create our policy by hand for this time. However, it is also possible to export/import rules. The important part here is that link to the CoSign integration is valid, otherwise ACS cannot verify the signature.

  1. Open "Platform Configuration > Policy > and click the button Create Policy"

  2. Name: Trusted Signature Policy

  3. Severity: High

  4. Categories: Security Best Practices

  5. Click Next

  6. Lifecycle stage: check Build and Deploy

  7. Response method: Inform and enforce > This will make the Task in the Pipeline fail when the signature cannot be verified.

  8. Configure enforcement behavior:

    1. Activate: Enforce on Build > Only fail for build lifecycles.

    2. Leave Deployment disabled for now. A good practice would be to enable it, but for other steps in this series, it will be required to keep it disabled for now.

    3. Leave Runtime disabled.

  9. Click Next

  10. As policy criteria find "Image Registry > Image signature" and drag and drop it to the policy section

    ACS Policy Criteria
    Figure 6. ACS Policy Criteria

  11. Click on Select and select the CoSign integration

    ACS Assign CoSign integration to policy
    Figure 7. ACS Assign CoSign integration to policy

  12. Click Next

  13. Leave the Policy scope for now and click Next

  14. Review the Policy and Save it.

Prepare the Pipeline Tasks

Now, finally, after all these preparations we are going to integrate ACS into our Pipeline. For this, we will create 2 tasks:

  1. acs-image-scan: Will scan the image for vulnerabilities

  2. acs-image-check: Will verify if the image would violate any security policy, especially the custom policy we have created.

Both checks will use the roxctl command line tool.

It is recommended to add these two Tasks to any Pipeline system you are using when you have ACS installed to leverage the full potential of ACS. It does not matter if you are using Tekton, Jenkins or something else. You can find examples of integrations at the Git repository: https://github.com/stackrox/contributions/.

Task acs-image-scan

To scan for vulnerabilities, create the following Task. It leverages the command line tool roxctl which can also be used on your local machine to perform such scans manually.

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: acs-image-scan
  namespace: ci
spec:
  description: >-
    Scan an image with StackRox/RHACS. This tasks allows you to check an
    image against vulnerabilities.
  params:
    - description: |
        Secret containing the address:port tuple for StackRox Central)
        (example - rox.stackrox.io:443)
      name: rox_central_endpoint
      type: string
    - description: Secret containing the StackRox API token with CI permissions
      name: rox_api_token
      type: string
    - description: |
        Full name of image to scan (example -- gcr.io/rox/sample:5.0-rc1)
      name: image
      type: string
    - default: 'false'
      description: |
        When set to `"true"`, skip verifying the TLS certs of the Central
        endpoint.  Defaults to `"false"`.
      name: insecure-skip-tls-verify
      type: string
    - default: 'registry.access.redhat.com/ubi9@sha256:089bd3b82a78ac45c0eed231bb58bfb43bfcd0560d9bba240fc6355502c92976'
      name: ubi9
      type: string
  results:
    - description: Output of `roxctl image check`
      name: check_output
      type: string
  steps:
    - env: (1)
        - name: ROX_API_TOKEN
          valueFrom:
            secretKeyRef:
              key: rox_api_token
              name: $(params.rox_api_token)
        - name: ROX_CENTRAL_ENDPOINT
          valueFrom:
            secretKeyRef:
              key: rox_central_endpoint
              name: $(params.rox_central_endpoint)
      image: $(params.ubi9)
      name: rox-image-scan
      resources: {}
      script: | (2)
        #!/usr/bin/env bash

        set +x

        curl -s -k -L -H "Authorization: Bearer $ROX_API_TOKEN" \
          "https://$ROX_CENTRAL_ENDPOINT/api/cli/download/roxctl-linux" \
          --output ./roxctl  \
          > /dev/null

        chmod +x ./roxctl  > /dev/null

        if [ "$(params.insecure-skip-tls-verify)" = "true" ]; then

          export ROX_INSECURE_CLIENT_SKIP_TLS_VERIFY=true

        fi

        ./roxctl image scan -e "$ROX_CENTRAL_ENDPOINT" --image "$(params.image)" (3)
1Token end endpoint which we defined as Secrets
2Script that downloads and executes the CLI roxctl
3Calling "roxctl image scan"

Task Image Check

To verify the policies that are configured in ACS we create the Task acs-image-check. We will use the command line tool roxctl again.

ACS comes with several (80+) build-in policies. Some of them are activated, but none of them configured to enforce a policy. Except, the policy Trusted Signature Policy we have created above.
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: acs-image-check
  namespace: ci
spec:
  description: >-
    Policy check an image with StackRox/RHACS This tasks allows you to check an
    image against build-time policies and apply enforcement to fail builds. It's
    a companion to the stackrox-image-scan task, which returns full
    vulnerability scan results for an image.
  params:
    - description: |
        Secret containing the address:port tuple for StackRox Central)
        (example - rox.stackrox.io:443)
      name: rox_central_endpoint
      type: string
    - description: Secret containing the StackRox API token with CI permissions
      name: rox_api_token
      type: string
    - description: |
        Full name of image to scan (example -- gcr.io/rox/sample:5.0-rc1)
      name: image
      type: string
    - default: 'false'
      description: |
        When set to `"true"`, skip verifying the TLS certs of the Central
        endpoint.  Defaults to `"false"`.
      name: insecure-skip-tls-verify
      type: string
    - default: 'registry.access.redhat.com/ubi9@sha256:089bd3b82a78ac45c0eed231bb58bfb43bfcd0560d9bba240fc6355502c92976'
      name: ubi9
      type: string
  results:
    - description: Output of `roxctl image check`
      name: check_output
      type: string
  steps:
    - env: (1)
        - name: ROX_API_TOKEN
          valueFrom:
            secretKeyRef:
              key: rox_api_token
              name: $(params.rox_api_token)
        - name: ROX_CENTRAL_ENDPOINT
          valueFrom:
            secretKeyRef:
              key: rox_central_endpoint
              name: $(params.rox_central_endpoint)
      image: $(params.ubi9)
      name: rox-image-check
      resources: {}
      script: |
        #!/usr/bin/env bash (2)

        set +x

        curl -s -k -L -H "Authorization: Bearer $ROX_API_TOKEN" \
          "https://$ROX_CENTRAL_ENDPOINT/api/cli/download/roxctl-linux" \
          --output ./roxctl  \
          > /dev/null

        chmod +x ./roxctl  > /dev/null

        if [ "$(params.insecure-skip-tls-verify)" = "true" ]; then

          export ROX_INSECURE_CLIENT_SKIP_TLS_VERIFY=true

        fi

        ./roxctl image check -e "$ROX_CENTRAL_ENDPOINT" --image "$(params.image)" (3)
1Token end endpoint which we defined as Secrets
2Script that downloads and executes the CLI roxctl
3Calling "roxctl image check"

Extend the Pipeline

Now it is time to extend our Pipeline with the two tasks. Add the following blocks to the Pipeline secure-supply-chain:

    - name: acs-image-scan (1)
      params: (2)
        - name: rox_central_endpoint
          value: stackrox-endpoint
        - name: rox_api_token
          value: stackrox-secret
        - name: image
          value: '$(params.IMAGE_REPO):$(params.IMAGE_TAG)'
        - name: insecure-skip-tls-verify (3)
          value: 'true'
      runAfter: (4)
        - build-sign-image
      taskRef:
        kind: Task
        name: acs-image-scan
    - name: acs-image-check (5)
      params: (6)
        - name: rox_central_endpoint
          value: stackrox-endpoint
        - name: rox_api_token
          value: stackrox-secret
        - name: image
          value: '$(params.IMAGE_REPO):$(params.IMAGE_TAG)'
        - name: insecure-skip-tls-verify (7)
          value: 'true'
      runAfter:
        - build-sign-image (8)
      taskRef:
        kind: Task
        name: acs-image-check
1Scanning for vulnerabilities
2Provide parameters to the Task
3Set to 'false' if you have valid certificates.
4Run after the Task "build-sign-image"
5Scanning security policies
6Provide parameters to the Task
7Set to 'false' if you have valid certificates.
8Run after the Task "build-sign-image"

The Pipeline will now look like this:

Pipeline Details
Figure 8. Pipeline Details

Execute the Pipeline

Let’s trigger another PipelineRun by updating the README.md of our source code. While the previous Tasks should run successfully, let’s monitor our two ACS tasks. However, now the Pipeline is running quite long already. This means it is time for a coffee.

Still here? Good, eventually our two ACS tasks have finished successfully.

The Task image-scan is looking for vulnerabilities and did not find any high-severity issues.

ACS Image Scan Result
Figure 9. ACS Image Scan Result

The Task image-check verified the security policies. You can see in the Logs that the *Trusted Signature Policy was not violated. Another might be shown, but since all other policies are configured to notify only (instead of "enforce"), the whole Task will end successfully. This is good enough for our first tests. The important bit is that the image has been signed.

ACS Image Check Result
Figure 10. ACS Image Check Result

Summary

Now we have integrated ACS and are using its powers to scan images for vulnerabilities and check them against security policies. Two Tasks have been created that are executed after the image has been built and are leveraging ACS’s command line tool roxctl

Whenever vulnerabilities are found, or security policies are violated (if the policy is set to enforce) the Pipeline will fail.

Such tasks should be built in any pipeline you are creating, independently of using ACS or any other security verification tool.

In the next step, we will create a Task that creates a Software Bill of Material (SBOM) for us.

See Also:

  • Introducing AdminNetworkPolicies –

    Classic Kubernetes/OpenShift offer a feature called NetworkPolicy that allows users to control the traffic to and from their assigned Namespace. NetworkPolicies are designed to give project owners or tenants the ability to protect their own namespace. Sometimes, however, I worked with customers where the cluster administrators or a dedicated (network) team need to enforce these policies. Since the NetworkPolicy API is namespace-scoped, it is not possible to enforce policies across namespaces. The only solution was to create custom (project) admin and edit roles, and remove the ability of creating, modifying or deleting NetworkPolicy objects. Technically, this is possible and easily done. But shifts the whole network security to cluster administrators. Luckily, this is where AdminNetworkPolicy (ANP) and BaselineAdminNetworkPolicy (BANP) comes into play.

  • Using Kustomize to post render a Helm Chart –

    Lately I came across several issues where a given Helm Chart must be modified after it has been rendered by Argo CD. Argo CD does a helm template to render a Chart. Sometimes, especially when you work with Subcharts or when a specific setting is not yet supported by the Chart, you need to modify it later …​ you need to post-render the Chart. In this very short article, I would like to demonstrate this on a real-live example I had to do. I would like to inject annotations to a Route objects, so that the certificate can be injected. This is done by the cert-utils operator. For the post-rendering the Argo CD repo pod will be extended with a sidecar container, that is watching for the repos and patches them if required.

  • Managing Certificates using GitOps approach –

    The article SSL Certificate Management for OpenShift on AWS explains how to use the Cert-Manager Operator to request and install a new SSL Certificate. This time, I would like to leverage the GitOps approach using the Helm Chart cert-manager I have prepared to deploy the Operator and order new Certificates. I will use an ACME Letsencrypt issuer with a DNS challenge. My domain is hosted at AWS Route 53. However, any other integration can be easily used.

  • Update Cluster Version using GitOps approach –

    During a GitOps journey at one point, the question arises, how to update a cluster? Nowadays it is very easy to update a cluster using CLI or WebUI, so why bother with GitOps in that case? The reason is simple: Using GitOps you can be sure that all clusters are updated to the correct, required version and the version of each cluster is also managed in Git. All you need is the channel you want to use and the desired cluster version. Optionally, you can define the exact image SHA. This might be required when you are operating in a restricted environment.

  • Multiple Sources for Applications in Argo CD –

    Argo CD or OpenShift GitOps uses Applications or ApplicationSets to define the relationship between a source (Git) and a cluster. Typically, this is a 1:1 link, which means one Application is using one source to compare the cluster status. This can be a limitation. For example, if you are working with Helm Charts and a Helm repository, you do not want to re-build (or re-release) the whole chart just because you made a small change in the values file that is packaged into the repository. You want to separate the configuration of the chart with the Helm package. The most common scenarios for multiple sources are (see: Argo CD documentation): Your organization wants to use an external/public Helm chart You want to override the Helm values with your own local values You don’t want to clone the Helm chart locally as well because that would lead to duplication and you would need to monitor it manually for upstream changes. This small article describes three different ways with a working example and tries to cover the advantages and disadvantages of each of them. They might be opinionated but some of them proved to be easier to use and manage.

Copyright © 2020 - 2024 Toni Schmidbauer & Thomas Jungbauer