Supply Chain on TechBlog about OpenShift/Ansible/Satellite and much more

Introduction to a Secure Supply Chain

Wed, 28 Jun 2023 00:00:00 +0000

The goal of the following ("short") series is to build a secure CI/CD pipeline step by step using OpenShift Pipelines (based on Tekton). The whole build process shall pull and build an image, upload it to a development environment and subsequently update the production environment.

The main focus here is security. Several steps and tools shall help to build and deploy a Secure Supply Chain.

The whole process is part of a Red Hat workshop which can present to your organization. I did some tweaks and created a step-by-step plan in order to remember it … since I am getting old :)

The Journey to Secure Supply Chain

This series includes the following articles:

Prerequisites

In order to develop our Secure Supply Chain, we need an OpenShift 4 Cluster. I am currently using OpenShift 4.13. Moreover, the OpenShift Pipelines operator must be deployed. It is based on Tekton and provides a Kubernetes-native way to create CI/CD pipelines.

The operator is deployed using the Operator Hub inside your cluster. Simply search for OpenShift Pipelines and install the operator using the default settings.

Figure 1. Install OpenShift Pipelines

Finally, you will need a GitHub account to be able to fork some repositories.

Some steps in the pipeline are working tightly with GitHub, especially the very last one that is talking GitHub’s API. However, any Git-system should work, and probably just minor changes will be required.

Everything else will be installed during the different steps described in the upcoming articles, while we build and tweak our pipeline.

Remember, the big goal of our pipeline is NOT to simply pull, build and push our code, but to integrate certain security tools like code scanning, image scanning and linting. Otherwise, it would be boring.

Used Tools

The following list of tools (or specifications) are used for our pipeline. They will be deployed when the appropriate step requires it.

Step 1 - Listen to Events

Wed, 28 Jun 2023 00:00:00 +0000

In this first step, we will simply prepare our environment to be able to retrieve calls from Git. In Git we will fork a prepared source code into a repository and any time a developer pushes a new code into our repository a webhook will notify OpenShift Pipelines to start the pipeline. Like most pipelines, the first task to be executed is to fetch the source code so it can be used for the next steps. The application I am going to use is called globex-ui and is an example webUI build with Angular.

Goals

The goals of this step are:

Create the EventListener and Trigger-Settings that will take care of notifications by GitHub.
Create a secret for GitHub authentication.
Fork a prepared source code and create a webhook inside Git.

Create the EventListener

The first thing we need to create is a Namespace that will be responsible for all our Pipeline-objects. In this example, it is called ci:
```
oc new-project ci
```

Now we need to create the so-called EventListener. This requires the creation of several objects:

Create a TriggerBinding: A TriggerBinding captures fields from an event and provides them as named parameters to the TriggerTemplate and subsequently to the PipelineRun.

We will create two TriggerBindings:
- globex-ui - For the required settings of our example application
- github-push - For the relevant parameters to push into git. These parameters will be provided by Git whenever Git is using the Webhook to inform OpenShift that a new push event happened.
  
  Copy the following examples into your cluster.
  
  The list of parameters in these manifests will be extended throughout this series.
  apiVersion: triggers.tekton.dev/v1beta1 kind: TriggerBinding metadata: name: globex-ui namespace: ci spec: params: - name: tlsVerify (1) value: "false" - name: gitRepoHost (2) value: github.com
  1 Default values for verifying SSL is "false" (Since I do not have certificates in place)
  
  2 The default value for the Git URL is github.com
  apiVersion: triggers.tekton.dev/v1alpha1 kind: TriggerBinding metadata: name: github-push namespace: ci spec: params: - name: gitrepositoryurl (1) value: $(body.repository.clone_url) - name: fullname value: $(body.repository.full_name) - name: io.openshift.build.commit.ref value: $(extensions.ref) - name: io.openshift.build.commit.id value: $(body.head_commit.id) - name: io.openshift.build.commit.date value: $(body.head_commit.timestamp) - name: io.openshift.build.commit.message value: $(body.head_commit.message) - name: io.openshift.build.commit.author value: $(body.head_commit.author.name)
  1 Several parameters, coming from Git via the Webhook, for example the exact URL to the repository, the ID of the commit, the date of the commit etc.

Create a TriggerTemplate: A TriggerTemplate acts as a blueprint for PipelineRuns (or TaskRuns). The resources and parameters here will be used when our Pipeline is executed. It also defines the workspaces, that will be used by the pipeline. For now, we are using the space shared-data where we will pull the source code for further checks.

apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerTemplate
metadata:
name: app-globex-ui-template
namespace: ci
spec:
params: (1)
- description: The git repository URL.
name: gitrepositoryurl
- description: The repository name for this PullRequest.
name: fullname
- description: The git branch for this PR.
name: io.openshift.build.commit.ref
- description: the specific commit SHA.
name: io.openshift.build.commit.id
- description: The date at which the commit was made
name: io.openshift.build.commit.date
- description: The commit message
name: io.openshift.build.commit.message
- description: The name of the github user handle that made the commit
name: io.openshift.build.commit.author
- description: The host name of the git repo
name: gitRepoHost
- description: Enable image repository TLS certification verification.
name: tlsVerify
- description: Extra parameters passed for the push command when pushing images.
name: build_extra_args
- description: Target image repository name
name: imageRepo
resourcetemplates: (2)
- apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
generateName: secure-supply-chain- (3)
spec:
params: (4)
- name: REPO_HOST
value: $(tt.params.gitRepoHost)
- name: GIT_REPO
value: $(tt.params.gitrepositoryurl)
- name: TLSVERIFY
value: $(tt.params.tlsVerify)
- name: BUILD_EXTRA_ARGS
value: $(tt.params.build_extra_args)
- name: IMAGE_REPO
value: $(tt.params.imageRepo)
- name: IMAGE_TAG
value: >-
$(tt.params.io.openshift.build.commit.ref)-$(tt.params.io.openshift.build.commit.id)
- name: COMMIT_SHA
value: $(tt.params.io.openshift.build.commit.id)
- name: GIT_REF
value: $(tt.params.io.openshift.build.commit.ref)
- name: COMMIT_DATE
value: $(tt.params.io.openshift.build.commit.date)
- name: COMMIT_AUTHOR
value: $(tt.params.io.openshift.build.commit.author)
- name: COMMIT_MESSAGE
value: $(tt.params.io.openshift.build.commit.message)
pipelineRef: (5)
name: secure-supply-chain
serviceAccountName: pipeline (6)
workspaces: (7)
- name: shared-data
volumeClaimTemplate:
metadata:
creationTimestamp: null
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
status: {}

1	List of parameters for this TriggerTemplate, that should be used further for the pipeline.
2	The resources we are going to use.
3	The name prefix of the generated PipelineRun
4	List of parameters that shall be provided to the pipeline
5	The reference to the pipeline that shall be executed.
6	Name of the ServiceAccount that will execute the Pipeline. Per default, this is pipeline which is managed by the Operator.
7	The workspaces that will be used by the PipelineRun. Currently shared-data only.

Create an EventListener that sets up a Service and listens for specific events and exposes a sink that receives incoming events, for example from a GitHub Webhook. It connects TriggerTemplate to a TriggerBinding. In this example, we create a Listener with 1 replica (that’s enough for testing) and connect our two TriggerBindings.

We also refer to the secret webhook-secret-globex-ui which will hold the password for GitHub to authenticate. We filter any push event coming from my Git repository tjungbauer/globex-ui

apiVersion: triggers.tekton.dev/v1alpha1
kind: EventListener
metadata:
name: globex-ui-event-listener
namespace: ci
spec:
namespaceSelector: {}
resources:
kubernetesResource:
replicas: 1
spec:
template:
metadata:
creationTimestamp: null
spec:
containers: null
serviceAccountName: pipeline
triggers: (1)
- bindings:
- kind: TriggerBinding
ref: globex-ui
- kind: TriggerBinding
ref: github-push
interceptors:
- params:
- name: secretRef
value:
secretKey: webhook-secret-key
secretName: webhook-secret-globex-ui (2)
ref:
kind: ClusterInterceptor
name: github
- params:
- name: filter (3)
value: >-
(header.match('X-GitHub-Event', 'push') &&
body.repository.full_name == 'tjungbauer/globex-ui')
- name: overlays
value:
- expression: 'body.ref.split(''/'')[2]'
key: ref
ref:
kind: ClusterInterceptor
name: cel
name: build-from-push-globex-ui
template: (4)
ref: app-globex-ui-template

1	TriggerBindings that are used.
2	Reference to the secret.
3	A filter for push events and our repository name.
4	The TriggerTemplate that will be used.

Now let us create a Route object to allow external traffic (from Git) to the EventListener.

apiVersion: route.openshift.io/v1
kind: Route
metadata:
name: el-event-listener
namespace: ci
spec:
port:
targetPort: http-listener
to:
kind: Service
name: el-globex-ui-event-listener (1)
weight: 100
tls:
termination: edge
insecureEdgeTerminationPolicy: Redirect
wildcardPolicy: None

1	Service that will be automatically created when the EventListener has been created.

And finally, we create a Secret to allow GitHub to authenticate. The name of the Secret is referenced inside the EventListener object.
```
kind: Secret
apiVersion: v1
metadata:
name: webhook-secret-globex-ui (1)
namespace: ci
stringData:
webhook-secret-key: yoursecret (2)
type: Opaque
```
1 Name as referenced in the EventListener

2 Your super secure password

Prepare GitHub

Now we have everything in place to prepare our source code in Git. All we need to do is to create a repository that holds our source code and a Webhook.

Fork the Source Code: https://github.com/redhat-gpte-devopsautomation/globex-ui

Why fork? I want to be able to update the files and trigger the Pipeline whenever I want to. My forked repository can be found at: https://github.com/tjungbauer/globex-ui
Create a Webhook in GitHub. Go to Settings > Webhooks and add a new Webhook using:

Figure 1. Create a new Webhook.
1. The Route URL that was created.
2. Content type: application/json.
3. Your Password as used in the secret above.
4. Enable or disable SSL verification, since I was too lazy to create a certificate at my demo cluster, I disabled it.
5. And select which events, shall be sent to the Listener. In our case, push events are just fine.
After a few seconds GitHub should have validated the Webhook (reload the page eventually)

Figure 2. Verify Webhook

Summary

That’s it, we now have a Git repository, that will send any push-event to the EventListener, which uses the Triggers to fill out any required parameters and starts the pipeline named: secure-supply-chain.

This pipeline does not exist yet and will be created in the next step together with its first task to pull from the Git repository.

Step 2 - Pipelines

Wed, 28 Jun 2023 00:00:00 +0000

We will now create the Pipeline and try to trigger it for the first time to verify if our Webhook works as intended.

Goals

The goals of this step are:

Create the Pipeline with a first task
Update the Github repository, to verify if the Webhook works
Verify if the PipelineRun is successful

Create the Pipeline

The Pipeline object is responsible to define the Tasks (steps) that should be executed. Whenever a Pipeline is started a PipelineRun is created that performs each defined Task in the defined order and logs the output. Tasks can run subsequently or in parallel.

Currently, the Pipeline has one task pull-source-code which is defined as a ClusterTask "git-clone". The purpose is to simply pull the source code to the workspace "shared-data".

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
name: secure-supply-chain (1)
namespace: ci
spec:
params: (2)
- name: REPO_HOST
type: string
- name: COMMIT_SHA
type: string
- name: TLSVERIFY
type: string
- name: BUILD_EXTRA_ARGS
type: string
- name: IMAGE_REPO
type: string
- name: IMAGE_TAG
type: string
- name: GIT_REF
type: string
- name: COMMIT_DATE
type: string
- name: COMMIT_AUTHOR
type: string
- name: COMMIT_MESSAGE
type: string
- name: GIT_REPO
type: string
tasks: (3)
- name: pull-source-code (4)
params:
- name: url (5)
value: $(params.GIT_REPO)
- name: revision
value: $(params.GIT_REF)
- name: deleteExisting
value: 'true'
taskRef: (6)
kind: ClusterTask
name: git-clone
workspaces: (7)
- name: output
workspace: shared-data
workspaces: (8)
- name: shared-data

1	Name of the Pipeline as referenced in the TriggerTemplate.
2	List of Parameters, hopefully, injected by the EventListener.
3	List of Tasks that will be executed.
4	Name of the Task.
5	Parameters used in this Task.
6	The Reference to the task. Here a ClusterTask named "git-clone" is used.
7	Workspace that shall be used in this Task.
8	Workspaces available in this Pipeline.

The initial Pipeline will now look like the following (Go to: Pipelines > Pipelines > secure-supply-chain)

Figure 1. Initial Pipeline

Our first Run

Now it is time to update something in our Git Repository and verify if everything can be executed successfully.

To update, it is enough to simply add a space in the README.md file and push it to Git.

If the Webhook works as expected, Git will notify our EventListener, which will then trigger the Pipeline. A PipelineRun is created, that executes all Tasks that are defined in the Pipeline (currently just 1)

You can monitor the progress of the PipelineRun:

Figure 2. PipelineRun Overview

On the Details-page you can see which step is currently executed:

Figure 3. PipelineRun Details

Eventually, the PipelineRun finishes successfully.

Figure 4. PipelineRun Finished

You can analyze the Logs in case of an Error or to get more details of a certain Task:

Figure 5. Task Logs

Summary

We have now created our first Pipeline and tested the GitHub Webhook. Whenever we push changes to the code, Git will notify the EventListener which will trigger the Pipeline with all required Parameters.

A PipelineRun is generated and is executing the defined Tasks. Currently, not much is done, expect cloning the Git repository.

In the next steps, we will evolve our Pipeline to perform security checks and sign our image.

Step 3 - SonarQube

Wed, 28 Jun 2023 00:00:00 +0000

After the Pipeline has been created and tested we will add another Task to verify the source code and check for possible security issues, leveraging the tool SonarQube by Sonar.

Goals

The goals of this step are:

Install and configure SonarQube
Add a Task to scan our source code for vulnerabilities
Verify the results.

SonarQube

SonarQube by Sonar helps developers to deliver clean code. With the integration into our CICD pipeline it will detect issues and reports them back to the developers. The results will be shown in a dashboard. We will install the Community version of SonarQube, which is enough for our showcase.

SonarQube Installation

To install SonarQube I have prepared a Helm Chart that I use with GitOps when I deploy a new lab environment. Feel free to use it. It simply calls the Chart that is provided by Sonar. In addition, it creates a Job that changes the default administrator password to a different one.

The values file is good to go, the only item you must change is the route in the very first line. Also, if you prefer not to deploy any plugins (for example, the German language pack), you can remove the appropriate line.

Before you run the Helm, you need to provide a Sealed Secret (or manually create a Secret) like the following:

kind: Secret
apiVersion: v1
metadata:
name: credentials
namespace: sonarqube
data:
adminpass: <your base64 password string>
type: Opaque

This password will be used by the Job "change-admin-password" and will configure a new password for the user "admin"

Once everything is installed (this will take several minutes), you can access SonarQube using the URL you defined in the values file.

Figure 1. SonarQube

SonarQube Create Token

Our Pipeline will talk to SonarQube and request a scan of the source code. To be able to do this, we need to create a Token in SonarQube and store it as a Secret in our ci namespace.

Click on "My Account" (Upper right corner) > Security and create a new token:

Figure 2. SonarQube Token

Copy the token and create the following Secret:

kind: Secret
apiVersion: v1
metadata:
name: globex-ui-sonarqube-secret
namespace: ci
stringData:
token: <your SonarQube Token> (1)
type: Opaque

1	The generated Token NOT base64 encrypted (I am using stringData in this case)

Modify the Pipeline

Now it is time to bring the SonarQube scan task into our Pipeline. This requires some modifications to existing objects and the creation of some new ones.

Modify the TriggerBinding.

Add the following lines to globex-ui TriggerBinding

 - name: sonarqubeHostUrl
value: http://sonarqube.apps.ocp.aws.ispworld.at/ (1)

1	The URL of SonarQube

So, it will look like this:

apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerBinding
metadata:
name: globex-ui
namespace: ci
spec:
params:
- name: tlsVerify
value: 'false'
- name: gitRepoHost
value: github.com
- name: sonarqubeHostUrl
value: https://sonarqube.apps.ocp.aws.ispworld.at/ (1)

1	The URL of SonarQube

Modify the TriggerTemplate

Add the following lines:

spec:
params: (1)
- description: Sonarqube host url
name: sonarqubeHostUrl
...
resourcetemplates:
- apiVersion: tekton.dev/v1beta1
spec:
params: (2)
- name: SONARQUBE_HOST_URL
value: $(tt.params.sonarqubeHostUrl)
- name: SONARQUBE_PROJECT_KEY
value: globex-ui (3)
- name: SONARQUBE_PROJECT_SECRET
value: globex-ui-sonarqube-secret (4)
...

1	Parameters provided by the TriggerBinding.
2	Parameters provided to the Pipeline.
3	Project that will be created in SonarQube.
4	Secret of the SonarQube token.

The result should look like the following:

apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerTemplate
metadata:
name: app-globex-ui-template
namespace: ci
spec:
params:
- description: The git repository URL.
name: gitrepositoryurl
- description: The repository name for this PullRequest.
name: fullname
- description: The git branch for this PR.
name: io.openshift.build.commit.ref
- description: the specific commit SHA.
name: io.openshift.build.commit.id
- description: The date at which the commit was made
name: io.openshift.build.commit.date
- description: The commit message
name: io.openshift.build.commit.message
- description: The name of the github user handle that made the commit
name: io.openshift.build.commit.author
- description: The host name of the git repo
name: gitRepoHost
- description: Enable image repository TLS certification verification.
name: tlsVerify
- description: Extra parameters passed for the push command when pushing images.
name: build_extra_args
- description: Target image repository name
name: imageRepo
- description: Sonarqube host url
name: sonarqubeHostUrl
resourcetemplates:
- apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
generateName: secure-supply-chain-
spec:
params:
- name: REPO_HOST
value: $(tt.params.gitRepoHost)
- name: GIT_REPO
value: $(tt.params.gitrepositoryurl)
- name: TLSVERIFY
value: $(tt.params.tlsVerify)
- name: BUILD_EXTRA_ARGS
value: $(tt.params.build_extra_args)
- name: IMAGE_REPO
value: $(tt.params.imageRepo)
- name: IMAGE_TAG
value: >-
$(tt.params.io.openshift.build.commit.ref)-$(tt.params.io.openshift.build.commit.id)
- name: COMMIT_SHA
value: $(tt.params.io.openshift.build.commit.id)
- name: GIT_REF
value: $(tt.params.io.openshift.build.commit.ref)
- name: COMMIT_DATE
value: $(tt.params.io.openshift.build.commit.date)
- name: COMMIT_AUTHOR
value: $(tt.params.io.openshift.build.commit.author)
- name: COMMIT_MESSAGE
value: $(tt.params.io.openshift.build.commit.message)
- name: SONARQUBE_HOST_URL
value: $(tt.params.sonarqubeHostUrl)
- name: SONARQUBE_PROJECT_KEY
value: globex-ui
- name: SONARQUBE_PROJECT_SECRET
value: globex-ui-sonarqube-secret
pipelineRef:
name: secure-supply-chain
serviceAccountName: pipeline
workspaces:
- name: shared-data
volumeClaimTemplate:
metadata:
creationTimestamp: null
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
status: {}

Create the Task scan-source. This task will use the pulled source code and uses SonarQube to let it scan our code.

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: scan-code
namespace: ci
spec:
description: >-
Source code scan using sonar-scanner and SonarQube.
params:
- default: 'docker.io/sonarsource/sonar-scanner-cli:latest' (1)
name: scanImage
type: string
- default: 'https://sonarqube-sonarqube.myplaceholder.com/' (2)
name: sonarqubeHostUrl
type: string
- default: object-detection-rest
name: sonarqubeProjectKey
type: string
- default: object-detection-rest-sonarqube-secret
name: sonarqubeProjectSecret
type: string
- default: 'true'
name: verbose
type: string
steps:
- env:
- name: SONAR_TOKEN_WEB_UI (3)
valueFrom:
secretKeyRef:
key: token
name: $(params.sonarqubeProjectSecret) (4)
image: $(params.scanImage)
name: scan-code
resources: {}
script: > (5)
set -x
echo $(ls -a)
sonar-scanner -X -Dsonar.projectKey=$(params.sonarqubeProjectKey)
-Dsonar.sources=./ -Dsonar.host.url=$(params.sonarqubeHostUrl)
-Dsonar.login=$SONAR_TOKEN_WEB_UI
workingDir: /workspace/repository
workspaces: (6)
- name: repository

1	Image containing SonarQube command line tool. The cluster must be able to connect to docker.io.
2	Default parameters for this Task that might be overwritten.
3	The Secret with the token.
4	Parameter as set by the PipelineRun which gets the value from the TriggerTemplate.
5	Script that is executed to scan the source code.
6	The workspace where we can find the source code.

Update your Pipeline and add the following task:

spec:
params:
...
- name: SONARQUBE_HOST_URL
type: string
- name: SONARQUBE_PROJECT_KEY
type: string
- name: SONARQUBE_PROJECT_SECRET
type: string
tasks:
...
- name: scan-source
params: (1)
- name: sonarqubeHostUrl
value: $(params.SONARQUBE_HOST_URL)
- name: sonarqubeProjectKey
value: $(params.SONARQUBE_PROJECT_KEY)
- name: sonarqubeProjectSecret
value: $(params.SONARQUBE_PROJECT_SECRET)
runAfter: (2)
- pull-source-code
taskRef: (3)
kind: Task
name: scan-code
workspaces: (4)
- name: repository
workspace: shared-data

1	Parameters that shall be provided for the Task.
2	The task should run AFTER the source has been pulled … which makes sense.
3	Reference to the Task we created above.
4	Workspace shared-data where the source code was pulled from the previous Task.

The full pipeline objects now look like the following:

Expand me...

The Pipeline now has a second task:

Figure 3. Pipeline

Execute the Pipeline

Let’s update the README.md of our source code again to trigger another PipelineRun. After the code has been pulled it should now perform the second task and scan the quality of the source code.

You can monitor the progress of the PipelineRun again:

Figure 4. PipelineRun Details

Once the PipelineRun executed both tasks successfully, we can check SonarQube.

The project globex-ui has been created which shows the results of our scan:

Figure 5. SonarQube Results

Summary

We have now added a Task to our Pipeline that performs a code analysis of our source code. The results are shown in SonarQube and the developers can react accordingly.

Step 4 - Verify Git Commit

Wed, 28 Jun 2023 00:00:00 +0000

Besides checking the source code quality, we should also verify if the commit into Git was done by someone/something we trust. It is a good practice to sign all commits to Git. You need to prepare your Git account and create trusted certificates.

I will not describe how exactly you need to configure Git to sign your commit. Verify the following link to learn more about Signing Commits

Goals

The goals of this step are:

Verify if the last commit has been signed

Prerequisites

Signing public key
Configured Git to verify your gpg signature

When your commit is signed, Git will show that:

Figure 1. Pipeline

Steps

Create the following Secret that contains your PUBLIC key.

kind: Secret
apiVersion: v1
metadata:
name: gpg-public-key
namespace: ci
data:
public.key: >-
<Base64 PUBLIC GPG KEY> (1)
type: Opaque

1	Public key, containing BEGIN/END lines base64 encoded.

Create the following Task:

apiVersion: tekton.dev/v1
kind: Task
metadata:
name: verify-source-code-commit-signature
namespace: ci
spec:
description: This task verifies the latest commit and signature against the gpg
public key
params:
- default: 'registry.redhat.io/openshift-pipelines/pipelines-git-init-rhel8:v1.10.4-4'
name: gitInit
type: string
steps:
- computeResources: {}
image: $(params.gitInit)
name: git-verify
script: |
set -x (1)
gpg --import /workspace/secrets/public.key
git config --global --add safe.directory /workspace/repository
git verify-commit HEAD || (echo "Unable to verify commit at HEAD!" && exit 1)
workingDir: /workspace/repository
workspaces:
- name: repository
- name: secrets (2)

1	The script to verify the signature of the commit,
2	The workspace that mounts the Secret containing the gpg key,

Modify the TriggerTemplate and add the following 3 lines
```
 workspaces:
...
- name: secrets
secret:
secretName: gpg-public-key (1)
```
1 The name of the Secret where the public key can be found.
Update the pipeline to execute the task verify-commit-signature, which is running in parallel to the SonarQube scan.
```
 - name: verify-commit-signature
runAfter:
- pull-source-code (1)
taskRef:
kind: Task
name: verify-source-code-commit-signature (2)
workspaces: (3)
- name: repository
workspace: shared-data
- name: secrets
workspace: secrets
workspaces:
...
- name: secrets (4)
```
1 This task runs after pull-source-code but in parallels with the SonarQube task.

2 Task reference

3 Workspaces that are used in this Task

4 Additional workspace for the Pipeline

The full pipeline objects now look like the following:

Expand me...

The status of the Pipeline now is:

Figure 2. Pipeline

Execute the Pipeline

Let’s update the README.md of our source code again to trigger another PipelineRun.

Now the 3rd task will verify if the commit was signed.

Figure 3. PipelineRun Details

In the logs of the Task, we can see that the commit was signed and could be verified. See:

...
gpg: Good signature from "Thomas Jungbauer <tjungbau@redhat.com>"
...

Figure 4. Signature Verification

Summary

At this stage we have a Pipeline, that pulls our code, does a code analysis, and verifies if the commit has been signed. The very next step is to build the image and push it into an Image Registry.

Step 5 - Build and Sign Image

Wed, 28 Jun 2023 00:00:00 +0000

Finally, after pulling and checking the code, we are going to create the image. During this process the image will be signed and uploaded to the public registry Quay.io.

Goals

The goals of this step are:

Build the image
Sign the image
Upload it Quay.io

You can use any registry you like.

Prerequisites

You need to have an account in Quay.io and create a repository there.

I have created the repository https://quay.io/repository/tjungbau/secure-supply-chain-demo which will contain the images and the signatures.

Steps

Create a Robot account in Quay.io and get the authentication json. Go to "Account Settings > Robot Accounts" and create a new account.

Figure 1. Quay Robot Account
Allow this account WRITE permissions to the repository

Figure 2. Quay Robot Account Permissions
Select the just-created account and go to Kubernetes Secret.

Here you can view or download the Secret. It should look like the following:
```
apiVersion: v1
kind: Secret
metadata:
name: tjungbau-secure-supply-chain-demo-pull-secret
data:
.dockerconfigjson: <SECURE>
type: kubernetes.io/dockerconfigjson
```
Copy the content and store it in your ci Namespace.

The name will be probably different in other examples.
To be able for the ServiceAccount, which is executing the Pipelines, to use this Secret, we need to modify the ServiceAccount object.

In our case the ServiceAccount is called pipeline. Modify it and add the following lines:
```
secrets:
...
- name: tjungbau-secure-supply-chain-demo-pull-secret (1)
imagePullSecrets:
- name: tjungbau-secure-supply-chain-demo-pull-secret (2)
```
1 Use your appropriate name for the secret

2 Use your appropriate name for the secret, required to be able to sign the image

Update the Pipeline object and add the following block

This time we do not need to add a Task object, because we are using a ClusterTask "buildah" that takes care of the building process. As workspace "shared-data" is used again, which has all the source code stored:

 - name: build-sign-image
params:
- name: TLSVERIFY
value: $(params.TLSVERIFY)
- name: BUILD_EXTRA_ARGS
value: >-
--label=io.openshift.build.commit.author='$(params.COMMIT_AUTHOR)'
--label=io.openshift.build.commit.date='$(params.COMMIT_DATE)'
--label=io.openshift.build.commit.id='$(params.COMMIT_SHA)'
--label=io.openshift.build.commit.message='$(params.COMMIT_MESSAGE)'
--label=io.openshift.build.commit.ref='$(params.GIT_REF)'
--ulimit=nofile=4096:4096
- name: IMAGE
value: '$(params.IMAGE_REPO):$(params.IMAGE_TAG)'
retries: 1
runAfter:
- scan-source
- verify-commit-signature
taskRef:
kind: ClusterTask
name: buildah
workspaces:
- name: source
workspace: shared-data

Update the TriggerBinding globex-ui and define the URL for your registry respectively for your image repository
```
spec:
parameters:
- name: imageRepo
value: quay.io/tjungbau/secure-supply-chain-demo
```

With this configuration, we could already build the image and push it into the registry. However, we also would like to sign it.

Prepare Tekton for Image Signing

To automatically sign images, we use TektonChains which is a controller that allows you to manage your supply chain security by signing TaskRuns and/or OCI images. Once the build-Task has pushed the images to Quay, Tekton will detect this event (by monitoring specific values of Task results) and then creates and pushes the signature for that image.

By default, Tekton Chains observes all task run executions. When the task runs complete, Tekton Chains signs and stores all artefacts.

Tekton Chain is part of OpenShift Pipelines and has the following main features:

You can sign task runs, task run results, and OCI registry images with cryptographic keys that are generated by tools such as cosign and skopeo.
You can use attestation formats such as in-toto.
You can securely store signatures and signed artefacts using OCI repository as a storage backend.

In our case, we will use CoSign for signing the images and Rekor for the attestation.

To activate image signing add the following to the TektonConfig object by removing the "chain{}" entry.

 chain:
artifacts.oci.storage: oci (1)
artifacts.taskrun.format: in-toto (2)
artifacts.taskrun.storage: oci (3)
transparency.enabled: true (4)

1	The storage backend for storing OCI signatures.
2	The format for storing TaskRun payloads. Can be in-toto or slsa/v1.
3	The storage backend for TaskRun signatures. Multiple can be defined.
4	Enable or disable automatic binary transparency uploads. The URL for uploading binary transparency attestations is https://rekor.sigstore.dev by default.

More details can be found at https://docs.openshift.com/container-platform/4.13/cicd/pipelines/using-tekton-chains-for-openshift-pipelines-supply-chain-security.html

CoSign - Signing the Image

We will use CoSign to sign our image. To do so we need to download the cosign binary and generate a key pair:

Be sure that you are logged into the OpenShift cluster.

cosign generate-key-pair k8s://openshift-pipelines/signing-secrets

Enter password for private key:
Enter password for private key again:
Successfully created secret signing-secrets in namespace openshift-pipelines
oc get secrets signing-secrets -n openshift-pipelines
NAME TYPE DATA AGE
signing-secrets Opaque 3 46h

Cosign will ask you to enter a password and will then create a Kubernetes secret in the Namespace openshift-pipelines.

When OpenShift Pipelines now execute a Task that is pushing an image, this Secret will be used to sign the image.

Let’s update our source code. The image will be built and pushed into Quay. The small black shield indicates that this image has been signed.

Figure 3. Quay Signed Image

Besides the actual image the files: *.sig and *.att can be seen. The first one is the signature, the second one shows metadata about the attestation retrieved from Rekor which can be compared with the Rekor URL.

Rekor

Rekor’s goals are to provide an immutable tamper resistant ledger of metadata generated within a software projects supply chain. Rekor will enable software maintainers and build systems to record signed metadata to an immutable record. Other parties can then query said metadata to enable them to make informed decisions on trust and non-repudiation of an object’s lifecycle. For more details visit the sigstore website.

The Rekor project provides a restful API based server for validation and a transparency log for storage. A CLI application is available to make and verify entries, query the transparency log for inclusion proof, integrity verification of the transparency log or retrieval of entries by either public key or artifact. Rekor Git

An important feature of Tekton Chains is that it integrates seamlessly with an application called Rekor. The configuration transparency.enabled: true enables the call to the Rekor API that can be found by default at: https://rekor.sigstore.dev. This will create a transparency log which can be used for verification purposes later.

Of course, it would be possible to install your own server. Learn in the official Docs how this can be achieved.

This means every time our build-Tasks successfully completes a URL to the transparency logs will be attached to the Task.

After our PipelineRuns were successful, we can verify if the image has an entry in the Rekor transparency log. This is important to ensure that the image went through a valid signing process. At a later step (before we create the pull request for the production update) we will verify if the log exists.

The following annotations should have been added to the TaskRun or PipelineRun secure-supply-chain-XXX-build-sign-image automatically:

metadata:
annotations:
chains.tekton.dev/signed: 'true'
chains.tekton.dev/transparency: 'https://rekor.sigstore.dev/api/v1/log/entries?logIndex=25593495'

This created the following transparency logs:

{
"24296fb24b8ad77a7ae84f4b950336d1872a066aeb9463dfbc231a7d3b73dd040dd5faefb3c013a7": {
"body": "eyJhcGlWZXJzaW9uIjoiMC4wLjEiLCJraW5kIjoiaGFzaGVkcmVrb3JkIiwic3BlYyI6eyJkYXRhIjp7Imhhc2giOnsiYWxnb3JpdGhtIjoic2hhMjU2IiwidmFsdWUiOiJmYzE2ODM3Mzc1ODUyNjc0MTQ4MjIyMDJhMWU2Y2RkZDkxNWI4NWUzYzhhNDVhZmI0YzEyYmE2YmNhMGNmNDQyIn19LCJzaWduYXR1cmUiOnsiY29udGVudCI6Ik1FWUNJUUNRMnlnU2ZGTllHS2pzbXRIYjVielAwdlRNMFgyNHVlNXlKbjJxdWFWSyt3SWhBSmhoSWpweEFybDJERjFsVmpMT0ZzVnRhMzJmbXJXRmxXWkdRQ015b2xrayIsInB1YmxpY0tleSI6eyJjb250ZW50IjoiTFMwdExTMUNSVWRKVGlCUVZVSk1TVU1nUzBWWkxTMHRMUzBLVFVacmQwVjNXVWhMYjFwSmVtb3dRMEZSV1VsTGIxcEplbW93UkVGUlkwUlJaMEZGVEc0clduaENNMHNyS3pWSk1pOWlNWFJqVFVKcVZXODRjWGx6YVFwNlQzUlpVbGRQUTBwTVRWQlNWVGR4WTFJeVMyWlZZazAzYlVwUlNtbHZjbXR6VW1KUmNHMTBTekV4WjJNM1pIVkZObGg0WmxOdlpWUkJQVDBLTFMwdExTMUZUa1FnVUZWQ1RFbERJRXRGV1MwdExTMHRDZz09In19fX0=",
"integratedTime": 1688060588,
"logID": "c0d23d6ad406973f9559f3ba2d1ca01f84147d8ffc5b8445c224f98b9591801d",
"logIndex": 25593495,
"verification": {
"inclusionProof": {
"checkpoint": "rekor.sigstore.dev - 2605736670972794746\n21430853\n10sx5mwekx2c8jltWbt0WeJu+tNM9FOBNQoyPiC3CPQ=\nTimestamp: 1688061275289390813\n\n— rekor.sigstore.dev wNI9ajBFAiEAyzFYKKghB+RQsbMIIzjywdA9vFjsusQoMWMUJPRjw3ECIDVeYSGKn7zwXWxGLhQlIGAI1Ca0Gu7ZuLJ64xR3SrY0\n",
"hashes": [
"5cc5dba1f5f420acc9ed049c77b04c4fdd66cf6ef1ebd46b66e26039bb9ba06c",
"3041520c9fec6177225063053503f6d20b09e86b72968b737e3211a233dab6ce",
"ddb57132bba122df920d4816af7ac02bef03aec533cc7b45f8552ce19c023d10",
"8a510da356706a0a822daf3c3a935176d3634c6e0cce6830fe90a8771a3bc83d",
"7d254234192620827306cc5024ffb8ae699bfe2725c8e6aa13757f5471d416d4",
"648a4e45da960f4ad286dfa22ae93721c9ba82ee05edd5a7134d9b19e35735e0",
"9dcf0f21690ec420bffbc1d10c383b7d3dc568d26bd8fd9e8771abb01f5976dc",
"86575ffec011d78ad393e7af267693c0c59360e2299b308369951d8362032733",
"0088ef1f6ec1e992fa6c91837287f2bd7eea238e5af7467e21e6df0dc8efe516",
"b149d95903a4e554ac1c381f1afff31bb62b6e12b6b2fc95f36b8e198e1a42cd",
"de73a694862cebd660ef1cecdd1bb66e273ab0651ca939bf7fa6f5f567bafe93",
"a3c84734ddae3102952584443dea70e3dec2cc085af7cf0ba3530751966861ec",
"0be5c7bbcf481d1efcfc63a27fce447cf6345f7bb3155cf5de39de592c16d52d",
"f597f4bae8df3d6fc6eebfe3eabd7d393e08781f6f16b42398eca8512398fff1",
"4e35fcb3c0a59e7f329994002b38db35f5d511f499ba009e10b31f5d27563607",
"47044b7ac3aab820e44f0010538d7de71e17a11f4140cbbe9eeb37f78b77cc7d",
"eee63677e2591eefe06ab537d6dd1b4060770682744c8287879b5dcd3365a5b2",
"ff41aa21106dbe03996b4335dd158c7ffafd144e45022193de19b2b9136c3e42",
"e6ebdeef2e23335d8d7049ba5a0049a90593efdfe9c1b4548946b44a19d7214f",
"dd51e840e892d70093ad7e1db1e2dea3d50334c7345d360e444d22fc49ed9f5e",
"ad712c98424de0f1284d4f144b8a95b5d22c181d4c0a246518e7a9a220bdf643"
],
"logIndex": 21430064,
"rootHash": "d74b31e66c1e931d9cf2396d59bb7459e26efad34cf45381350a323e20b708f4",
"treeSize": 21430853
},
"signedEntryTimestamp": "MEYCIQDqmqb4k95FjiBNogOAmjTskkIPaGslgvrSND4pQdUALwIhAMt85yLsa5Ei+3NsmJ906/T9Hx1YZDDHQfHlTEYqqcaE"
}
}
}

Summary

Finally, we have a signed image uploaded to Quay. In addition, a transparency log has been created. Now let us add some security checks in the next steps.

Step 6 - Scanning with ACS

Wed, 28 Jun 2023 00:00:00 +0000

In this step we will install Advanced Cluster Security (ACS) and create 2 new steps in our Pipeline to scan the image for vulnerabilities and security policy. A custom security policy, configured in ACS, will verify if the image is signed.

Goals

The goals of this step are:

Install ACS.
Create a custom security policy into ACS.
Configure CoSign integration in ACS.
Create a step in the Pipeline to perform an image scan (vulnerabilities).
Create a step in the Pipeline to perform an image check (security policies).

Install Advanced Cluster Security (ACS)

We will install ACS on our cluster which will be called local-cluster inside ACS.

The deployment will happen via Argo CD (OpenShift GitOps) that will use the following Helm Chart: RHACS

The Argo CD Application will trigger the deployment which will take several minutes.

The deployment will install a minimal version of ACS and tries to limit the required resources and replicas. For production environments, you probably want to increase the settings.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: in-cluster-init-rhacs
namespace: openshift-gitops
spec:
destination:
namespace: default
server: 'https://kubernetes.default.svc' (1)
info:
- name: Description
value: >-
Initialize Red Hat Advanced Cluster Security and deploy Central and
SecuredCluster
project: in-cluster
source:
path: charts/rhacs-full-stack (2)
repoURL: 'https://github.com/tjungbauer/helm-charts' (3)
targetRevision: main

1	Installing on the local cluster.
2	Path to the Helm chart.
3	URL to the Helm chart repository.

Figure 1. Syncing ACS Deployment Application

This GitOps Application will start a full-stack ACS deployment. It does (in this exact order):

Installs the ACS Operator into rhacs-operator
Verifies if the Operator is ready
Creates the Namespace stackrox
Adds a ConsoleLink in the upper right action menu of OpenShift

Figure 2. ACS ConsoleLink
Creates the Central custom resource with minimum resources
Creates an init-bundle to add the first (local) cluster
Adds the custom resource SecureCluster to install the rest of the ACS components
And finally, runs a basic configuration that enables authentication via OpenShift and provides the kubeadmin user admin privileges. This can be disabled in the values file if you prefer not to configure this, or if kubeadmin does not exist anymore in your cluster.

The whole process will take a while and uses syncwaves and hooks.

Configure ACS Integration

Generate Authentication Token

Create an authentication Token in ACS

Go to "Platform Configuration > Integrations > API Token"

Figure 3. ACS Generate Token

Generate a new Token with at least Continuous Integration privileges and save the created Token.

Back in OpenShift, a Secret must be created with the key rox_api_token

kind: Secret
apiVersion: v1
metadata:
name: stackrox-secret
namespace: ci
data:
rox_api_token: <base 64 Token> (1)
type: Opaque

1	Base64 decoded ACS Token.

Create a 2nd Secret that contains the ACS endpoint and its port:

kind: Secret
apiVersion: v1
metadata:
name: stackrox-endpoint
namespace: ci
stringData:
rox_central_endpoint: central-stackrox.apps.ocp.aws.ispworld.at:443 (1)
type: Opaque

1	The endpoint URL of my example cluster. Note: the port MUST be added here.

Integrate CoSign with ACS

Configure CoSign Integration

During Step 5 we have created a CoSign key pair to sign our images. To integrate with ACS, we need to retrieve the public key. In OpenShift, open the Secret signing-secrets in the Namespace openshift-pipelines and extract the key cosign.pub.

It will look something like tjis:
```
-----BEGIN PUBLIC KEY-----
key...
-----END PUBLIC KEY-----
```
In ACS, go to "Platform Configuration > Integrations > Signature" and create a new CoSign integration.

Figure 4. ACS CoSign Integration

Enter a name and the public key and activate this integration

Figure 5. ACS Create CoSign Integration

This will enable ACS to verify the CoSign signature of the image.

Create a Custom Policy that verifies the Signature

Create a custom security policy, that verifies if our image has been signed or not.

To be sure that every image is correctly signed, we create a custom security policy that verifies this signature using our CoSign integration. This policy can be configured as inform or enforce. Enforce means that the pipeline will fail (during the Task acs-image-check) if the image signature cannot be checked.

Let’s create our policy by hand for this time. However, it is also possible to export/import rules. The important part here is that link to the CoSign integration is valid, otherwise ACS cannot verify the signature.

Open "Platform Configuration > Policy > and click the button Create Policy"
Name: Trusted Signature Policy
Severity: High
Categories: Security Best Practices
Click Next
Lifecycle stage: check Build and Deploy
Response method: Inform and enforce > This will make the Task in the Pipeline fail when the signature cannot be verified.
Configure enforcement behavior:
1. Activate: Enforce on Build > Only fail for build lifecycles.
2. Leave Deployment disabled for now. A good practice would be to enable it, but for other steps in this series, it will be required to keep it disabled for now.
3. Leave Runtime disabled.
Click Next
As policy criteria find "Image Registry > Image signature" and drag and drop it to the policy section

Figure 6. ACS Policy Criteria
Click on Select and select the CoSign integration

Figure 7. ACS Assign CoSign integration to policy
Click Next
Leave the Policy scope for now and click Next
Review the Policy and Save it.

Prepare the Pipeline Tasks

Now, finally, after all these preparations we are going to integrate ACS into our Pipeline. For this, we will create 2 tasks:

acs-image-scan: Will scan the image for vulnerabilities
acs-image-check: Will verify if the image would violate any security policy, especially the custom policy we have created.

Both checks will use the roxctl command line tool.

It is recommended to add these two Tasks to any Pipeline system you are using when you have ACS installed to leverage the full potential of ACS. It does not matter if you are using Tekton, Jenkins or something else. You can find examples of integrations at the Git repository: https://github.com/stackrox/contributions/.

Task acs-image-scan

To scan for vulnerabilities, create the following Task. It leverages the command line tool roxctl which can also be used on your local machine to perform such scans manually.

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: acs-image-scan
namespace: ci
spec:
description: >-
Scan an image with StackRox/RHACS. This tasks allows you to check an
image against vulnerabilities.
params:
- description: |
Secret containing the address:port tuple for StackRox Central)
(example - rox.stackrox.io:443)
name: rox_central_endpoint
type: string
- description: Secret containing the StackRox API token with CI permissions
name: rox_api_token
type: string
- description: |
Full name of image to scan (example -- gcr.io/rox/sample:5.0-rc1)
name: image
type: string
- default: 'false'
description: |
When set to `"true"`, skip verifying the TLS certs of the Central
endpoint. Defaults to `"false"`.
name: insecure-skip-tls-verify
type: string
- default: 'registry.access.redhat.com/ubi9@sha256:089bd3b82a78ac45c0eed231bb58bfb43bfcd0560d9bba240fc6355502c92976'
name: ubi9
type: string
results:
- description: Output of `roxctl image check`
name: check_output
type: string
steps:
- env: (1)
- name: ROX_API_TOKEN
valueFrom:
secretKeyRef:
key: rox_api_token
name: $(params.rox_api_token)
- name: ROX_CENTRAL_ENDPOINT
valueFrom:
secretKeyRef:
key: rox_central_endpoint
name: $(params.rox_central_endpoint)
image: $(params.ubi9)
name: rox-image-scan
resources: {}
script: | (2)
#!/usr/bin/env bash
set +x
curl -s -k -L -H "Authorization: Bearer $ROX_API_TOKEN" \
"https://$ROX_CENTRAL_ENDPOINT/api/cli/download/roxctl-linux" \
--output ./roxctl \
> /dev/null
chmod +x ./roxctl > /dev/null
if [ "$(params.insecure-skip-tls-verify)" = "true" ]; then
export ROX_INSECURE_CLIENT_SKIP_TLS_VERIFY=true
fi
./roxctl image scan -e "$ROX_CENTRAL_ENDPOINT" --image "$(params.image)" (3)

1	Token end endpoint which we defined as Secrets
2	Script that downloads and executes the CLI roxctl
3	Calling "roxctl image scan"

Task Image Check

To verify the policies that are configured in ACS we create the Task acs-image-check. We will use the command line tool roxctl again.

ACS comes with several (80+) build-in policies. Some of them are activated, but none of them configured to enforce a policy. Except, the policy Trusted Signature Policy we have created above.

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: acs-image-check
namespace: ci
spec:
description: >-
Policy check an image with StackRox/RHACS This tasks allows you to check an
image against build-time policies and apply enforcement to fail builds. It's
a companion to the stackrox-image-scan task, which returns full
vulnerability scan results for an image.
params:
- description: |
Secret containing the address:port tuple for StackRox Central)
(example - rox.stackrox.io:443)
name: rox_central_endpoint
type: string
- description: Secret containing the StackRox API token with CI permissions
name: rox_api_token
type: string
- description: |
Full name of image to scan (example -- gcr.io/rox/sample:5.0-rc1)
name: image
type: string
- default: 'false'
description: |
When set to `"true"`, skip verifying the TLS certs of the Central
endpoint. Defaults to `"false"`.
name: insecure-skip-tls-verify
type: string
- default: 'registry.access.redhat.com/ubi9@sha256:089bd3b82a78ac45c0eed231bb58bfb43bfcd0560d9bba240fc6355502c92976'
name: ubi9
type: string
results:
- description: Output of `roxctl image check`
name: check_output
type: string
steps:
- env: (1)
- name: ROX_API_TOKEN
valueFrom:
secretKeyRef:
key: rox_api_token
name: $(params.rox_api_token)
- name: ROX_CENTRAL_ENDPOINT
valueFrom:
secretKeyRef:
key: rox_central_endpoint
name: $(params.rox_central_endpoint)
image: $(params.ubi9)
name: rox-image-check
resources: {}
script: |
#!/usr/bin/env bash (2)
set +x
curl -s -k -L -H "Authorization: Bearer $ROX_API_TOKEN" \
"https://$ROX_CENTRAL_ENDPOINT/api/cli/download/roxctl-linux" \
--output ./roxctl \
> /dev/null
chmod +x ./roxctl > /dev/null
if [ "$(params.insecure-skip-tls-verify)" = "true" ]; then
export ROX_INSECURE_CLIENT_SKIP_TLS_VERIFY=true
fi
./roxctl image check -e "$ROX_CENTRAL_ENDPOINT" --image "$(params.image)" (3)

1	Token end endpoint which we defined as Secrets
2	Script that downloads and executes the CLI roxctl
3	Calling "roxctl image check"

Extend the Pipeline

Now it is time to extend our Pipeline with the two tasks. Add the following blocks to the Pipeline secure-supply-chain:

 - name: acs-image-scan (1)
params: (2)
- name: rox_central_endpoint
value: stackrox-endpoint
- name: rox_api_token
value: stackrox-secret
- name: image
value: '$(params.IMAGE_REPO):$(params.IMAGE_TAG)'
- name: insecure-skip-tls-verify (3)
value: 'true'
runAfter: (4)
- build-sign-image
taskRef:
kind: Task
name: acs-image-scan
- name: acs-image-check (5)
params: (6)
- name: rox_central_endpoint
value: stackrox-endpoint
- name: rox_api_token
value: stackrox-secret
- name: image
value: '$(params.IMAGE_REPO):$(params.IMAGE_TAG)'
- name: insecure-skip-tls-verify (7)
value: 'true'
runAfter:
- build-sign-image (8)
taskRef:
kind: Task
name: acs-image-check

1	Scanning for vulnerabilities
2	Provide parameters to the Task
3	Set to 'false' if you have valid certificates.
4	Run after the Task "build-sign-image"
5	Scanning security policies
6	Provide parameters to the Task
7	Set to 'false' if you have valid certificates.
8	Run after the Task "build-sign-image"

The Pipeline will now look like this:

Figure 8. Pipeline Details

Execute the Pipeline

Let’s trigger another PipelineRun by updating the README.md of our source code. While the previous Tasks should run successfully, let’s monitor our two ACS tasks. However, now the Pipeline is running quite long already. This means it is time for a coffee.

Still here? Good, eventually our two ACS tasks have finished successfully.

The Task image-scan is looking for vulnerabilities and did not find any high-severity issues.

Figure 9. ACS Image Scan Result

The Task image-check verified the security policies. You can see in the Logs that the *Trusted Signature Policy was not violated. Another might be shown, but since all other policies are configured to notify only (instead of "enforce"), the whole Task will end successfully. This is good enough for our first tests. The important bit is that the image has been signed.

Figure 10. ACS Image Check Result

Summary

Now we have integrated ACS and are using its powers to scan images for vulnerabilities and check them against security policies. Two Tasks have been created that are executed after the image has been built and are leveraging ACS’s command line tool roxctl

Whenever vulnerabilities are found, or security policies are violated (if the policy is set to enforce) the Pipeline will fail.

Such tasks should be built in any pipeline you are creating, independently of using ACS or any other security verification tool.

In the next step, we will create a Task that creates a Software Bill of Material (SBOM) for us.

Step 7 - Generating a SBOM

Wed, 28 Jun 2023 00:00:00 +0000

A software bill of materials (SBOM) offers an insight into the makeup of an application developed using third-party commercial tools and open-source software. It is a machine-readable, formally structured list of all components. This list is important to gain visibility into what risks might exist in a package and their potential impact. It allows to prioritize the risks and define a remediation path. The teams can use the SBOM to monitor all components that are used across an application.

We will create a Task that will generate a SBOM and uploads it into an SBOM repository.

Goals

The goals of this step are:

Install a SBOM repository.
Generate a SBOM and upload it to the repository.

Tool/Standard CycloneDX

OWASP CycloneDX is a full-stack Bill of Materials (BOM) standard that provides advanced supply chain capabilities for cyber risk reduction. The specification supports:

Software Bill of Materials (SBOM)
Software-as-a-Service Bill of Materials (SaaSBOM)
Hardware Bill of Materials (HBOM)
Operations Bill of Materials (OBOM)
Vulnerability Disclosure Reports (VDR)
Vulnerability Exploitability eXchange (VEX)

Strategic direction of the specification is managed by the CycloneDX Core Working Group, is backed by the OWASP Foundation, and is supported by the global information security community.

Source: https://cyclonedx.org/

The CycloneDX module for Node.js creates a valid CycloneDX Software Bill-of-Materials (SBOM) containing an aggregate of all project dependencies. CycloneDX is a lightweight SBOM specification that is easily created, human and machine-readable, and simple to parse.

https://cyclonedx.org/capabilities/sbom/

We will use this standard and will deploy a small Pod in our cluster to be able to upload our SBOM locally.

Install CycloneDX Repository Server

We can install a CycloneDX Repository Server, which is a simple Pod running in our cluster. You can use the following Argo CD application, which will create the Namespace (cyclonedx),Deployment, Route, and Service.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: in-cluster-install-cyclonedx
namespace: openshift-gitops
spec:
destination:
namespace: cyclonedx
server: 'https://kubernetes.default.svc'
info:
- name: Description
value: Install CycloneDX SBOM Repository
project: in-cluster
source:
path: clusters/management-cluster/cyclonedx (1)
repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops'
targetRevision: main

1	Path and URL to the Git repository.

In Argo CD the following Application will be created:

Figure 1. CycloneDX Installation

Update the Tekton Pipeline

Update the TriggerBinding globex-ui and add the following:

 - name: cyclonedxHostUrl
value: >-
https://cyclonedx-bom-repo-server-cyclonedx.apps.ocp.aws.ispworld.at (1)

1	DO NOT add a trailing slash here.

Update the TriggerTemplate and add the following to the appropriate blocks:

spec:
parames:
...
- description: Cyclonedx host url
name: cyclonedxHostUrl (1)
resourcetemplates:
...
spec:
params:
...
- name: CYCLONEDX_HOST_URL
value: $(tt.params.cyclonedxHostUrl) (2)

1	The parameter defined in the TriggerBinding
2	The parameter as it will be provided to the Pipeline.

Install the following Task

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: generate-sbom
namespace: ci
spec:
params: (1)
- default: >-
https://cyclonedx-bom-repo-server-cyclonedx.apps.placeholder.com/
name: cyclonedxHostUrl
type: string
- default: 'cyclonedx/cyclonedx-node'
name: cycloneDXnodeImage
type: string
results:
- description: The url location of the generate SBOM
name: sbomUrl
type: string
steps:
- image: $(params.cycloneDXnodeImage)
name: generate-sbom
resources:
requests:
memory: 1Gi
script: > (2)
apk add git curl
npm install .
/usr/src/cyclonedx-bom/bin/make-bom.js -o bom.xml
curl -v -k $(params.cyclonedxHostUrl)/v1/bom -H "Content-Type:
application/vnd.cyclonedx+xml; version=1.4" -H "Accept: */*" -d @bom.xml
-D /tmp/header.txt
LOCATION=$(cat /tmp/header.txt | grep location: | awk '{print $2}' | sed
's|http:|https:|g')
printf "%s" "$LOCATION" > "$(results.sbomUrl.path)"
echo "SBOM URL accessible on Results of TaskRun $(context.taskRun.name)"
workingDir: /workspace/repository
workspaces:
- name: repository

1	Default parameters for this task. Might be overwritten, by the EventListener
2	Script to be executed. It will download a script and upload the SBOM to our CycloneDX server.

Update the Pipeline object

spec:
params:
...
- name: CYCLONEDX_HOST_URL (1)
type: string
tasks:
...
- name: generate-sbom (2)
params:
- name: cyclonedxHostUrl
value: $(params.CYCLONEDX_HOST_URL)
runAfter:
- build-sign-image (3)
taskRef:
kind: Task
name: generate-sbom
workspaces:
- name: repository
workspace: shared-data

1	Add this to the parameter section
2	The new Task to generate a SBOM
3	Run after build-sign-image (and in parallel to the ACS tasks)

Figure 2. Pipeline

Execute the Pipeline

Let’s trigger the pipeline again. If everything works well, the URL to the uploaded SBOM will be visible in the TaskRun log.

Figure 3. Pipeline

When you open this URL, you will get a huge list of components end dependencies.

For example, the following snippet shows that the angular component animations (version 13.2.6) is used.

<components>
<component type="library" bom-ref="pkg:npm/%40angular/animations@13.2.6">
<author>angular</author>
<group>@angular</group>
<name>animations</name>
<version>13.2.6</version>
<description>Angular - animations integration with web-animations</description>
<licenses>
<license>
<id>MIT</id>
</license>
</licenses>
<purl>pkg:npm/%40angular/animations@13.2.6</purl>
<externalReferences>
<reference type="website">
<url>https://github.com/angular/angular#readme</url>
</reference>
<reference type="issue-tracker">
<url>https://github.com/angular/angular/issues</url>
</reference>
<reference type="vcs">
<url>git+https://github.com/angular/angular.git</url>
</reference>
</externalReferences>
</component>

Summary

This concludes our security scans for the whole build process. With the SBOM you have now a complete list of components and dependencies of your images.

During the next steps, we will work with the Kubernetes manifests and update them, perform a linting on these manifests, and try to deploy the updated image to our DEV environment.

Step 8 - Updating Kubernetes Manifests

Wed, 28 Jun 2023 00:00:00 +0000

With the finalization of the build process and the security checks during this phase, it is now time to update the Kubernetes manifests and provide the new tag for the created image. I have forked (and cleaned up) another repository that will store all yaml specifications that are required for OpenShift. You can find this repository at: Kubernetes Manifests

Goals

The goals of this step are:

Update the manifest in Git with the new image tag

Preparing the Pipeline

Let’s create the Task object that will take care of the update.

The Task will use git commands to update the manifests accordingly.

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: update-manifest
namespace: ci
spec:
description: >-
This task updates the manifest for the current application to point to the
image tag created with the short commit.
params: (1)
- description: Used to tag the built image.
name: image
type: string
- default: main
description: Target branch to push to
name: target-branch
type: string
- default: Tekton Pipeline
description: Git user name for performing the push operation.
name: git_user_name
type: string
- default: tekton@tekton.com
description: Git user email for performing the push operation.
name: git_user_email
type: string
- description: File in which the image configuration is stored.
name: configuration_file
type: string
- description: Repo in which the image configuration is stored.
name: repository
type: string
- default: 'registry.redhat.io/openshift-pipelines/pipelines-git-init-rhel8:v1.10.4-4'
name: gitInit
type: string
- name: verbose
description: Verbose output
type: string
default: "true"
steps: (2)
- image: $(params.gitInit)
name: git
env:
- name: PARAM_VERBOSE
value: $(params.verbose)
resources: {}
script: >
#!/usr/bin/env sh
set -eu
if [ "${PARAM_VERBOSE}" = "true" ] ; then
set -x
fi
# Setting up the git config.
git config --global user.email "$(params.git_user_email)"
git config --global user.name "$(params.git_user_name)"
# Checkout target branch to avoid the detached HEAD state
TMPDIR=$(mktemp -d)
cd $TMPDIR
git clone $(params.repository)
cd securing-software-supply-chain
git checkout $(params.target-branch)
# Set to the short commit value passed as parameter.
# Notice the enclosing " to keep it as a string in the resulting YAML.
IMAGE=\"$(params.image)\"
sed -i "s#\(.*value:\s*\).*#\1 ${IMAGE}#" $(params.configuration_file)
git add $(params.configuration_file)
git commit -m "Automatically updated manifest to point to image tag
$IMAGE"
git push origin $(params.target-branch)

1	Required default parameters.
2	Script to clone and push the update to Git.

To the Pipeline we have to add the following section

spec:
params: (1)
- name: MANIFEST_FILE
type: string
- name: MANIFEST_FILE_PROD
type: string
- name: MANIFEST_REPO
type: string
- name: MANIFEST_REPO_NAME
type: string
- name: MANIFEST_GIT_REF
type: string
...
tasks:
...
- name: update-dev-manifest
params:
- name: image
value: '$(params.IMAGE_REPO):$(params.IMAGE_TAG)'
- name: configuration_file
value: $(params.MANIFEST_FILE)
- name: repository
value: $(params.MANIFEST_REPO)
- name: git_user_name
value: $(params.COMMIT_AUTHOR)
runAfter: (2)
- acs-image-check
- acs-image-scan
- generate-sbom
taskRef:
kind: Task
name: update-manifest

1	Parameters that define the settings for the manifest repository
2	This time we are running after these three tasks

Update the TriggerBinding globex-ui

 - name: manifestRepo
value: git@github.com:tjungbauer/securing-software-supply-chain.git (1)
- name: manifestRepoRef
value: main
- name: manifestFile (2)
value: application/globex/overlays/dev/kustomization.yaml
- name: manifestFileProd (3)
value: application/globex/overlays/prod/kustomization.yaml
- name: manifestRepoName
value: tjungbauer/securing-software-supply-chain

1	The SSH url to GitHub
2	The overlay that will be used for the DEV environment
3	The overlay that will be used for the PROD environment

Update the TriggerTemplate and add:

spec:
params:
- description: The file to update to point to newly built image
name: manifestFile
- description: The file to update to point to newly built image in prod
name: manifestFileProd
- description: The repo to update to point to newly built image
name: manifestRepo
- description: The reference to the repo
name: manifestRepoRef
- description: The full name of the repo
name: manifestRepoName
...
resourcetemplates:
...
spec:
params:
- name: MANIFEST_FILE
value: $(tt.params.manifestFile)
- name: MANIFEST_FILE_PROD
value: $(tt.params.manifestFileProd)
- name: MANIFEST_REPO
value: $(tt.params.manifestRepo)
- name: MANIFEST_GIT_REF
value: $(tt.params.manifestRepoRef)
- name: MANIFEST_REPO_NAME
value: $(tt.params.manifestRepoName)
...

Git SSH Key

To be able to do changes in Git, which is using SSH keys to authenticate, we need to specify a Secret that knows the SSH private key and Github’s host keys.

Be sure that to create an SSH key (for example using ssh-keygen) and that GitHub knows about this key. You can add your key at "Account > Settings > SSH and GPG keys".

Run the following command to get the host keys of GitHub:

ssh-keyscan -H github.com
# github.com:22 SSH-2.0-babeld-c89ab1f3
# github.com:22 SSH-2.0-babeld-c89ab1f3
|1|cr4dkqIl2SnZqktvRsFnx1lA5Ag=|eie8EHXqQHgCZJq7F/TtRRZcBbc= ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk=
# github.com:22 SSH-2.0-babeld-c89ab1f3
|1|r6z4yeEV9Cog/2I4stZp1A34BAE=|EU8VcdD+KHMJJt9uPL4jh5zK0fI= ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg=
# github.com:22 SSH-2.0-babeld-c89ab1f3
|1|hVnTyBzb/gSR8jpz+NUziUkHy1A=|ENMDVCVsLfz2CWLa1C+BnzZI8Yg= ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl
# github.com:22 SSH-2.0-babeld-c89ab1f3

Finally, we need to create a Secret with the following content:

apiVersion: v1
kind: Secret
metadata:
annotations:
tekton.dev/git-0: github.com (1)
name: git-ssh-key
namespace: ci
type: kubernetes.io/ssh-auth
data:
ssh-privatekey: <BASE64 PRIVAT SSH KEY > (2)
known_hosts: < BASE64 of Githubs hostkey> (3)

1	This annotation defines for which host Tekton will inject the Secret.
2	A base64 decoded string of your PRIVATE ssh key.
3	A base64 decoded string of the host keys

Update the pipeline ServiceAccount and add the new Secret:

secrets:
...
- name: git-ssh-key

Execute the Pipeline

Time to trigger the Pipeline, which now looks like:

Figure 1. Pipeline Details

The new Task will automatically push the new image tag to the Kubernetes manifests Git repository. From there it can later be used to roll out the new version.

In Git you can track the changes:

Figure 2. Git Commit

Summary

We have now updated the Kubernetes manifests with the new image tag. In the next steps, we will verify these manifests (linting) and then try to deploy the new image on the DEV environment.

Step 9 - Linting Kubernetes Manifests

Wed, 28 Jun 2023 00:00:00 +0000

At this point we have checked our source code, verified that it has been signed and has no vulnerabilities, generated a SBOM and updated the Kubernetes manifests, that are responsible to deploy our application on OpenShift. As everything with OpenShift these Kubernetes objects are simple yaml files. In this step we will perform a linting on these files, to verify if they follow certain rules and best practices.

Goals

The goals of this step are:

Clone the Kubernetes manifest into a workspace
Create Tasks that perform the linting
WARN when the manifests do not follow best practices (we will not let the Task fail here, because I would need to modify all manifests)

What is linting

A linter is an analysis tool that verifies if your code has any errors, bugs or stylistic errors. SonarQube could be seen as such tool. We used it to scan our source code. For Kubernetes manifests we can use tools that check the yaml files against best practices or security. For example, it could notify you if you forgot to define resources or probes in your manifests.

Tools

In this step, I will use three linting tools. In no way, this means you need to use all three. I only use them for demonstration purposes. However, to lint your yaml files or Helm charts is a common practice you should consider.

For this demonstration I am leveraging:

In the end, you should choose the tool that fits best for you.

Manifests

The manifest we are going to use can be found at my GitHub repository: Securing Software Supply Chain. It is a fork and the original can be found here.

Prepare Pipeline

Modify the TriggerTemplate and add a parameter LINTING_INFORM_ONLY that either sets the Tasks to inform or enforce (failing when linting does find issues). In addition, we define a new workspace, where we will download and store the Kubernetes manifests.

spec:
params:
...
- description: Only inform on linting errors (Log) but do not actually fail
name: lintingInformOnly (1)
resourcetemplates:
...
spec:
params:
...
- name: LINTING_INFORM_ONLY
value: $(tt.params.lintingInformOnly)
...
workspaces:
...
- name: shared-data-manifests (2)
volumeClaimTemplate:
metadata:
creationTimestamp: null
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
status: {}

1	New parameter to either inform only when linting is unsuccessful or to enforce that the Task will end with an error.
2	Workspace to download and store the yaml files.

Update the TriggerBinding to set the values for the PipelineRun.
```
spec:
params:
...
- name: lintingInformOnly (1)
value: 'true'
```
1 Set the parameter to 'true'

Update the Pipeline object. Here we will need to add the parameter and four Tasks (clone the Git Repo, KubeLinter, Yamllint and kube-score) as well as the workspace.

spec:
params:
...
- name: LINTING_INFORM_ONLY (1)
type: string
...
- name: pull-manifests (2)
params:
- name: url
value: $(params.MANIFEST_REPO)
- name: revision
value: $(params.MANIFEST_GIT_REF)
- name: deleteExisting
value: 'true'
runAfter:
- update-dev-manifest (3)
taskRef:
kind: ClusterTask (4)
name: git-clone
workspaces: (5)
- name: output
workspace: shared-data-manifests
- name: kube-linter (6)
params:
- name: informLintingOnly (7)
value: $(params.LINTING_INFORM_ONLY)
runAfter:
- pull-manifests (8)
taskRef:
kind: Task
name: kube-linter
workspaces:
- name: repository
workspace: shared-data-manifests
- name: kube-score (9)
params:
- name: informLintingOnly
value: $(params.LINTING_INFORM_ONLY)
runAfter:
- pull-manifests (10)
taskRef:
kind: Task
name: kube-score
workspaces:
- name: repository (11)
workspace: shared-data-manifests
- name: yaml-lint (12)
params:
- name: informLintingOnly
value: $(params.LINTING_INFORM_ONLY)
runAfter:
- pull-manifests (13)
taskRef:
kind: Task
name: yaml-lint
workspaces:
- name: repository (14)
workspace: shared-data-manifests
workspaces:
...
- name: shared-data-manifests

1	New parameter assigned to the Pipeline.
2	Task to clone the repository to the workspace.
3	Will run after the Pipeline has updated the manifests with the new image.
4	Is a child of the ClusterTask git-clone.
5	The workspace to clone the repository.
6	Task to execute KubeLinter.
7	Parameter to either enforce or inform only.
8	Will run after the repository has been cloned.
9	Task to execute kube-score.
10	Will run after the repository has been cloned.
11	Workspace where the cloned repository can be found.
12	Task to execute Yamllint.
13	Will run after the repository has been cloned.
14	Workspace where the cloned repository can be found.

Remember: It is not required to execute three different linter tools. It is only done as a showcase. I personally like KubeLinter. Choose whatever tool is suitable for you.

Create the different Task objects for the linter tools. Each Task will execute a linter program and provides its very own Log.

I have created the image linter-image that contains the three required binaries. It is available at Quay.io and its original Dockerfile can be found here. Use it at your own risk :).

KubeLinter

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: kube-linter
namespace: ci
spec:
description: >-
Task to run KubeLinter and perform a linting of Kubernetes manifets.
params:
- default: 'false'
name: informLintingOnly
type: string
- default: 'quay.io/tjungbau/linter-image:v1.0.2'
name: linterImage
type: string
steps:
- image: $(params.linterImage)
name: kube-linter
resources: {}
script: >
#!/usr/bin/env bash
RC=0
kube-linter lint /workspace/repository/. --config "/workspace/repository/.kube-linter.yaml" (1)
if [ $? -gt 0 ]; then
RC=1
fi
# We actually do not fail but inform only
if [ "$(params.informLintingOnly)" = "true" ]; then
echo "Informing only, task will not fail. Actual return code was $RC"
exit 0;
fi
(exit $RC)
workingDir: /workspace/repository
workspaces:
- name: repository

1	Execute kube-linter using the configuration stored in the repository.

kube-score

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: kube-score
namespace: ci
spec:
description: >-
Task to run kube-score and perform a linting of Kubernetes manifets.
params:
- default: 'false'
name: informLintingOnly
type: string
- default: 'quay.io/tjungbau/linter-image:v1.0.2'
name: linterImage
type: string
steps:
- image: $(params.linterImage)
name: kube-linter
resources: {}
script: >
#!/usr/bin/env bash
RC=0
KUBESCORE_IGNORE_TESTS="${KUBESCORE_IGNORE_TESTS:-container-image-pull-policy,pod-networkpolicy}" (1)
for i in `find . -name '*.yaml' -type f`; do kube-score score
--ignore-test ${KUBESCORE_IGNORE_TESTS} $i; let RC=RC+$?; done
if [ $? -gt 0 ]; then
RC=1
fi
# We actually do not fail but inform only
if [ "$(params.informLintingOnly)" = "true" ]; then
echo "Informing only, task will not fail. Actual return code was $RC"
exit 0;
fi
(exit $RC)
workingDir: /workspace/repository
workspaces:
- name: repository

1	Disable checks for Network Policies or image Pull policy for kube-score.

Yammllint

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: yaml-lint
namespace: ci
spec:
description: >-
Task to run yamllint and perform a linting of Kubernetes manifets.
params:
- default: 'false'
name: informLintingOnly
type: string
- default: 'quay.io/tjungbau/linter-image:v1.0.2'
name: linterImage
type: string
steps:
- image: $(params.linterImage)
name: yaml-lint
resources: {}
script: |
#!/usr/bin/env bash
for files in `find . -type f -name '*.yaml'`; do (1)
yamllint -c /workspace/repository/.yamllint.yaml ${files}; let var=var+$?
done
# We actually do not fail but inform only
if [ "$(params.informLintingOnly)" = "true" ]; then
echo "Informing only, task will not fail. Actual return code was $var"
exit 0;
fi
(exit $var)
workingDir: /workspace/repository
workspaces:
- name: repository

1	Execute kube-linter using the configuration stored in the repository.

Execute the Pipeline

The Pipeline now looks like this:

Figure 1. Pipeline Details

Remember, you typically need only one linter tool, not three different ones. Since we inform only you will see some errors in the logs. For example, for kube-linter:

Figure 2. Kube-Linter Results

Summary

Now, all our yaml manifests have been linted, with three different tools. And because we do not fail at this stage, we can continue. The next steps will be some deployment checks.

Since everything is done using Argo CD and the manifests have been updated during the step "update-manifest", the changes will be most likely already deployed. Even if the linting-step comes later and might even fail. This is fine because we first deploy on a DEV environment. So, if linting fails, it will prohibit the rollout to production, while some application testing can still be done on DEV.

Step 10 - The Example Application

Tue, 27 Jun 2023 00:00:00 +0000

If you read all articles up to here (congratulations) you know that we always update the README file of "The Application". But what is this application exactly and how can we update it like in a real-life example? The Globex UI application is built with Angular and was prepared for this journey. It is a quite complex application and requires Kafka to be installed as well. However, since I am not a developer and this tool was already available, I forked and re-used it. The original can be found at Globex UI

Goals

The goals of this step are:

Deploy AMQ Streams Operator
Deploy Globex DEV
Deploy Globex Prod
Verify Deployments

Introduction

The Globex UI, although quite complex, will provide a simple web interface. Since I do not have multiple clusters, I will use the Namespaces globex-dev and globex-prod to distinguish between the two environments. This means, our DEV environment will run inside globex-dev and PROD in globex-prod. Kafka related stuff will be deployed in the Namespace kafka.

Figure 1. Globex UI

The deployment will be done automatically by GitOps. As soon as our Pipeline will be executed and subsequently updates our image in the Kubernetes manifests (see step: Linting Kubernetes Manifests), a new version of Globex UI will be deployed. The production update will follow in later steps in the pipeline.

GitOps will monitor changes in our Kubernetes manifest repository. Here, we use Kustomize overlays to separate between DEV and PROD.

As a general recommendation: Do everything with GitOps. If it is not in Git, it does not exist.

Installation

To install all required components, we create several GitOps Applications.

Although we create these Argo CD Applications by hand, it is recommended to put these definitions into Git also.

Since this is just a demo, I am using the cluster instance of OpenShift GitOps. DO NOT deploy customer workload using the default instance, since it has privileged permissions on the cluster. Instead, create a second instance (or multiple) for the developers.

The following Applications will be created:

Figure 2. GitOps Applications

Install AMQ Streams Operator

As the very first step, we need to deploy the AMQ Streams Operator. Simply search for the operator in the Operator Hub and deploy it using the default values.

Also, this deployment could and should be done using Argo CD. For the sake of speed, I have done this manually here.

Figure 3. AMQ Streams Operator

Install Kafka for Globex

Now we can deploy all Kafka-related stuff using GitOps. Let’s create the following Application object:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: globex-kafka
namespace: openshift-gitops
spec:
destination:
namespace: kafka (1)
server: 'https://kubernetes.default.svc' (2)
project: default
source:
path: application/kafka/overlays/dev (3)
repoURL: 'https://github.com/tjungbauer/securing-software-supply-chain'
targetRevision: HEAD
syncPolicy:
automated: (4)
selfHeal: true
syncOptions:
- RespectIgnoreDifferences=true
- CreateNamespace=true

1	The deployment must be done in the Namespace kafka
2	We install using the local GitOps server instance.
3	Path/URL to the GitHub Kustomize repository.
4	Auto-Update in case of changes are detected.

Install Globex DEV

The DEV installation will deploy the test version of our application into the Namespace globex-dev

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: globex-dev (1)
namespace: openshift-gitops
spec:
destination:
namespace: globex-dev (2)
server: 'https://kubernetes.default.svc' (3)
ignoreDifferences:
- group: '*'
jqPathExpressions:
- '.imagePullSecrets[] | select(.name|test(".-dockercfg-."))'
kind: ServiceAccount
project: default
source:
path: application/globex/overlays/dev (4)
repoURL: 'https://github.com/tjungbauer/securing-software-supply-chain'
targetRevision: HEAD
syncPolicy:
automated: (5)
selfHeal: true
syncOptions:
- RespectIgnoreDifferences=true
- CreateNamespace=true (6)

1	Globex DEV instance
2	Install into the Namespace globex-dev
3	We install using the local GitOps server instance.
4	Path/URL to the GitHub Kustomize repository.
5	Auto-Update in case of changes are detected.
6	Create the Namespace if it does not exist.

Install Globex PROD

And finally, the same for the PROD instance:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: globex-prod (1)
namespace: openshift-gitops
spec:
destination:
namespace: globex-prod (2)
server: 'https://kubernetes.default.svc' (3)
ignoreDifferences:
- group: '*'
jqPathExpressions:
- '.imagePullSecrets[] | select(.name|test(".-dockercfg-."))'
kind: ServiceAccount
project: default
source:
path: application/globex/overlays/prod (4)
repoURL: 'https://github.com/tjungbauer/securing-software-supply-chain'
targetRevision: HEAD
syncPolicy:
automated: (5)
selfHeal: true
syncOptions:
- RespectIgnoreDifferences=true
- CreateNamespace=true (6)

1	Globex DEV instance
2	Install into the Namespace globex-prod
3	We install using the local GitOps server instance.
4	Path/URL to the GitHub Kustomize repository.
5	Auto-Update in case of changes are detected.
6	Create the Namespace if it does not exist.

Summary

During this step we have added nothing new to our pipeline, but deployed our example application Globex UI instead, including Kafka. The next steps will now do another verification against ACS and if the transparency logs are available, and then finally prepares everything for production deployment.

Step 11 - ACS Deployment Check

Tue, 27 Jun 2023 00:00:00 +0000

After the Pipeline prepared the new image for DEV it should be checked against ACS for policy compliance. This ensures that the deployment manifest adheres to policy requirements. The command line tool roxctl will be leveraged to perform this task.

Goals

The goals of this step are:

Create a Task that performs a deployment check using roxctl

Create the Task

Create the following Task object. This Task is a bit more complex, since we want to verify the Deployment, we first need to build the Kustomize files which would create a big yaml file and will store it into the output folder (all.yaml). Then we use kubesplit to split this huge file into different smaller ones. Finally, the roxctl step will search for a file called deployment—globex-ui.yml and performs the security checks (verify if any security policy is violated in ACS) against this file.

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: acs-deploy-check
namespace: ci
spec:
description: >-
Policy check a deployment with StackRox/RHACS This tasks allows you to check
a deployment against build-time policies and apply enforcement to fail
builds. It's a companion to the stackrox-image-scan task, which returns full
vulnerability scan results for an image.
params:
- description: |
Secret containing the address:port tuple for StackRox Central)
(example - rox.stackrox.io:443)
name: rox_central_endpoint
type: string
- description: Secret containing the StackRox API token with CI permissions
name: rox_api_token
type: string
- default: 'https://default-git-url.git'
name: gitRepositoryUrl
type: string
- default: main
name: gitRepositoryRevision
type: string
- default: 'true'
name: verbose
type: string
- default: 'false'
description: |
When set to `"true"`, skip verifying the TLS certs of the Central
endpoint. Defaults to `"false"`.
name: insecure-skip-tls-verify
type: string
- default: 'quay.io/wpernath/kustomize-ubi'
name: kustomieBuildImage
type: string
- default: 'looztra/kubesplit'
name: kubesplitImage
type: string
- default: 'registry.access.redhat.com/ubi8:8.7-1026'
name: ubi8Image
type: string
results:
- description: Output of `roxctl deployment check`
name: check_output
type: string
steps:
- image: $(params.kustomieBuildImage)
name: kustomize-build
resources: {}
script: | (1)
#!/usr/bin/env sh
set -eu -o pipefail
cd /workspace/repository/application/globex/overlays/dev
mkdir -p /workspace/repository/input
mkdir -p /workspace/repository/output
kustomize build . --output /workspace/repository/input/all.yaml
workingDir: /workspace/repository
- image: $(params.kubesplitImage) (2)
name: kustomize-split
resources: {}
script: >
#!/usr/bin/env sh
set -eu -o pipefail
kubesplit -i /workspace/repository/input/all.yaml -o
/workspace/repository/output
workingDir: /workspace/repository
- env:
- name: ROX_API_TOKEN
valueFrom:
secretKeyRef:
key: rox_api_token
name: $(params.rox_api_token)
- name: ROX_CENTRAL_ENDPOINT
valueFrom:
secretKeyRef:
key: rox_central_endpoint
name: $(params.rox_central_endpoint)
image: $(params.ubi8Image)
name: rox-deploy-scan (3)
resources: {}
script: |
#!/usr/bin/env bash
set +x
cd /workspace/repository/output
curl -s -k -L -H "Authorization: Bearer $ROX_API_TOKEN" \
"https://$ROX_CENTRAL_ENDPOINT/api/cli/download/roxctl-linux" \
--output ./roxctl \
> /dev/null
chmod +x ./roxctl > /dev/null
DEPLOYMENT_FILE=$(ls -1a | grep *deployment--globex-ui.yml)
if [ "$(params.insecure-skip-tls-verify)" = "true" ]; then
export ROX_INSECURE_CLIENT_SKIP_TLS_VERIFY=true
fi
./roxctl deployment check -e "$ROX_CENTRAL_ENDPOINT" --file "$DEPLOYMENT_FILE"
workingDir: /workspace/repository
workspaces:
- name: repository

1	Build Kustomize and store the output into /workspace/repository/input/all.yaml.
2	Split the huge yaml file into separate ones and store them into /workspace/repository/output.
3	Search for the file deployment—globex-ui.yml and perform a roxctl deployment check.

Update the Pipeline

The Pipeline object must be extended with another Task:

 - name: acs-deploy-check
params:
- name: rox_central_endpoint (1)
value: stackrox-endpoint
- name: rox_api_token
value: stackrox-secret
- name: insecure-skip-tls-verify
value: 'true'
runAfter: (2)
- yaml-lint
- kube-score
- kube-linter
taskRef:
kind: Task
name: acs-deploy-check
workspaces:
- name: repository
workspace: shared-data-manifests (3)

1	The parameters required for ACS.
2	This task runs after the linting tasks.
3	The workspace, where the manifests have been pulled.

Execute the Pipeline

Again, we trigger our pipeline by simply updating the README.md of our source code.

The ACS check will verify if any security policies, that are valid for Deployment-states, are violated.

My test returned the following result.

Figure 1. Pipeline Details

However, since the policies are configured to "inform only", the PipelineRun will finish successfully.

Summary

All ACS checks have been done now. The deployment does not violate any security policy that is configured in ACS … or to be more exact: It does not violate enforced policy, thus the Task will end successfully.

Step 12 - Verify TLog Signature

Tue, 27 Jun 2023 00:00:00 +0000

Since the image has been deployed on DEV now, we need to prepare everything for production. Before we start, we need to confirm that the whole signing process has been passed and that our signature has been applied correctly. With CoSign we signed our image and used Rekor to create a transparency log. We will use this log to confirm that the image was signed.

Goals

The goals of this step are:

Verify if the image has been signed using Rekor transparency log.

CoSign public key

Before we start, we need a secret in the Namespace ci that provides the PUBLIC key of CoSign. This key was generated in a previous Step 5 - CoSign - Signing the Image

kind: Secret
apiVersion: v1
metadata:
name: cosign-secret (1)
namespace: ci
data:
cosign.pub: >-
<base64 CoSign PUBLIC key>
type: Opaque

1	base64 decoded PUBLIC key of CoSign

Create the Task

The following task will use several steps to fetch and parse the transparency log, that has been created. The different tasks are using images from Redhat-gpte. The command line tool rekor-cli will be used to verify the entries in the log.

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: rekor-verify
namespace: ci
spec:
params:
- default: image-registry-secret (1)
name: registrySecret
type: string
- default: cosign-secret (2)
name: cosignSecret
type: string
- default: >-
quay.io/placeholder/test:latest
name: image
type: string
- default: 'quay.io/tjungbau/pipeline-tools:v1.1.0'
name: pipelinetoolsImage
type: string
steps:
- env:
- name: REGISTRY_SECRET
valueFrom:
secretKeyRef:
key: .dockerconfigjson
name: $(params.registrySecret)
- name: COSIGN_PUBLIC_KEY
valueFrom:
secretKeyRef:
key: cosign.pub
name: $(params.cosignSecret)
image: $(params.pipelinetoolsImage)
name: cosign-verify-image
resources: {}
script: >
mkdir -p /home/cosign/.docker/
echo "${REGISTRY_SECRET}" > /home/cosign/.docker/config.json
echo "${COSIGN_PUBLIC_KEY}" > /workspace/cosign.pub
cosign verify --key /workspace/cosign.pub $(params.image) --output-file
/workspace/cosign.verify
cat /workspace/cosign.verify | jq --raw-output '.[0] | .critical |
.image | .["docker-manifest-digest"]' > /workspace/cosign.sha
rekor-cli search --sha $(cat /workspace/cosign.sha) --format json >
/workspace/rekor.search
cat /workspace/rekor.search | jq '.UUIDs[0]' | sed 's/\"//g' >
/workspace/rekor.uuid
rekor-cli get --uuid $(cat /workspace/rekor.uuid) --format json >
/workspace/rekor.get
cat /workspace/rekor.get | jq -r .Attestation

1	Registry pull secret.
2	CoSign public key.

Update the Pipeline

The Pipeline object must be extended with another Task:

 - name: verify-tlog-signature
params: (1)
- name: registrySecret (2)
value: tjungbau-secure-supply-chain-demo-pull-secret
- name: cosignSecret (3)
value: cosign-secret
- name: image
value: '$(params.IMAGE_REPO):$(params.IMAGE_TAG)'
runAfter: (4)
- yaml-lint
- kube-score
- kube-linter
taskRef:
kind: Task
name: rekor-verify

1	The parameters required for this task.
2	Pull secret for quay.io, which was created during step 5.
3	CoSign public key.
4	This task runs after the linting tasks.

Execute the Pipeline

Triggering the pipeline will now check the signature of an image.

The logs should show something like this:

Figure 1. Rekor Image Signature Verification

As you can see: The image has been signed correctly and the transparency logs can be parsed.

Summary

Finally, everything has been verified and we can bring everything into production. The final two steps will create a new branch in the Git manifest repository and a pull request that can be manually merged.

Step 13 - Bring it to Production

Tue, 27 Jun 2023 00:00:00 +0000

If you reached this article, congratulations! You read through tons of pages to build up a Pipeline. The last two steps in our Pipeline are: Creating a new branch and creating a pull request, with the changes of the image tag that must be approved and will be merged then (We will not update the main branch directly!). Finally, we will do a "real" update to the application to see the actual changes.

Goals

The goals of this step are:

Create a task that creates a new branch in Git repository.
Create a task that creates a pull request in Git repository.
Perform a full End2End run of the Pipeline and approve the pull request.

Create a token at GitHub

To create a pull request we need to authenticate against the Git api. For this, we will need a token. In GitHub open your personal settings (by clicking on your avatar) and then go to "Developer settings".

Select "Personal access tokens" and "fine-graining tokens" to create a new token.

Figure 1. GitHub Personal Access Token

Enter the required fields:

Name: Secure Supply Chain
Expiration: Whatever your like (max is 1 year)
Repository access: Only select repositories
Select your repository, for example: tjungbauer/securing-software-supply-chain
Permissions: "Pull requests" > Read and write

Figure 2. GitHub Personal Access Token Created

Save the created token and create the following secret:

kind: Secret
apiVersion: v1
metadata:
name: github-token
namespace: ci
stringData:
token: <Token Value> (1)
type: Opaque

1	Clear-text token. If already base64 encoded, change stringData to data.

Task: Create a new branch

Since we do not update the main branch directly, we will create a new (feature) branch, that will be used for a pull request and can be deleted after the pull request has been merged.

Create the following Task object

This Task will use git commands to create a new feature-branch and pushes the changes into that branch.

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: new-branch-manifest-repo
namespace: ci
spec:
description: >-
This task creates a branch for a PR to point to the image tag created with
the short commit.
params:
- description: Used to tag the built image.
name: image
type: string
- default: main
description: Target branch to push to
name: target-branch
type: string
- default: Tekton Pipeline
description: Git user name for performing the push operation.
name: git_user_name
type: string
- default: tekton@tekton.com
description: Git user email for performing the push operation.
name: git_user_email
type: string
- description: File in which the image configuration is stored.
name: configuration_file
type: string
- description: Repo in which the image configuration is stored.
name: repository
type: string
- default: 'registry.redhat.io/openshift-pipelines/pipelines-git-init-rhel8:v1.10.4-4'
name: gitInit
type: string
steps:
- image: $(params.gitInit)
name: git
resources: {}
script: >-
# Setting up the git config.
git config --global user.email "$(params.git_user_email)"
git config --global user.name "$(params.git_user_name)"
# Checkout target branch to avoid the detached HEAD state
TMPDIR=$(mktemp -d)
cd $TMPDIR
git clone $(params.repository) (1)
cd securing-software-supply-chain
git checkout -b $(params.target-branch) (2)
# Set to the short commit value passed as parameter.
# Notice the enclosing " to keep it as a string in the resulting YAML.
IMAGE=\"$(params.image)\"
sed -i "s#\(.*value:\s*\).*#\1 ${IMAGE}#" $(params.configuration_file)
git add $(params.configuration_file) (3)
git commit -m "Automatically updated manifest to point to image tag
$IMAGE"
git push origin $(params.target-branch)

1	Clone the main repository.
2	Create a new feature branch.
3	Add, commit, and push everything to the new branch.

Modify the Pipeline object

The Task must be added to the Pipeline, it provides several required parameters.

 - name: create-prod-manifest-branch
params:
- name: image
value: '$(params.IMAGE_REPO):$(params.IMAGE_TAG)'
- name: configuration_file
value: $(params.MANIFEST_FILE_PROD)
- name: repository
value: $(params.MANIFEST_REPO)
- name: git_user_name
value: $(params.COMMIT_AUTHOR)
- name: target-branch
value: feature-for-$(params.COMMIT_SHA)
runAfter:
- acs-deploy-check
- verify-tlog-signature
taskRef:
kind: Task
name: new-branch-manifest-repo

Task: Create a Pull request

Create the following Task object

The following task will take the token and create a new pull request at GitHub:

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: git-open-pull-request
namespace: ci
spec:
description: >-
This task will open a PR on Github based on several parameters. This could
be useful in GitOps repositories for example.
params:
- default: api.github.com
description: |
The GitHub host, adjust this if you run a GitHub enteprise or Gitea
name: GITHUB_HOST_URL
type: string
- default: ''
description: |
The API path prefix, GitHub Enterprise has a prefix e.g. /api/v3
name: API_PATH_PREFIX
type: string
- description: |
The GitHub repository full name, e.g.: tektoncd/catalog
name: REPO_FULL_NAME
type: string
- default: github
description: >
The name of the kubernetes secret that contains the GitHub token,
default: github
name: GITHUB_TOKEN_SECRET_NAME
type: string
- default: token
description: >
The key within the kubernetes secret that contains the GitHub token,
default: token
name: GITHUB_TOKEN_SECRET_KEY
type: string
- default: Bearer
description: >
The type of authentication to use. You could use the less secure "Basic"
for example
name: AUTH_TYPE
type: string
- description: |
The name of the branch where your changes are implemented.
name: HEAD
type: string
- description: |
The name of the branch you want the changes pulled into.
name: BASE
type: string
- description: |
The body description of the pull request.
name: BODY
type: string
- description: |
The title of the pull request.
name: TITLE
type: string
- default: 'registry.access.redhat.com/ubi8/python-38:1'
name: ubi8PythonImage
type: string
results:
- description: Number of the created pull request.
name: NUMBER
type: string
- description: URL of the created pull request.
name: URL
type: string
steps:
- env:
- name: PULLREQUEST_NUMBER_PATH
value: $(results.NUMBER.path)
- name: PULLREQUEST_URL_PATH
value: $(results.URL.path)
image: $(params.ubi8PythonImage)
name: open-pr
resources: {}
script: >-
#!/usr/libexec/platform-python (1)
"""This script will open a PR on Github"""
import json
import os
import sys
import http.client
github_token = (2)
open("/etc/github-open-pr/$(params.GITHUB_TOKEN_SECRET_KEY)",
"r").read()
open_pr_url = "/repos/$(params.REPO_FULL_NAME)/pulls"
data = { (3)
"head": "$(params.HEAD)",
"base": "$(params.BASE)",
"title": """$(params.TITLE)""",
"body": """$(params.BODY)"""
}
print("Sending this data to GitHub: ")
print(data)
authHeader = "Bearer " + github_token
giturl = "api."+"$(params.GITHUB_HOST_URL)"
conn = http.client.HTTPSConnection(giturl)
conn.request(
"POST",
open_pr_url,
body=json.dumps(data),
headers={
"User-Agent": "OpenShift Pipelines",
"Authorization": authHeader.strip(),
"Accept": "application/vnd.github+json",
"Content-Type": "application/json",
"X-GitHub-Api-Version": "2022-11-28"
})
resp = conn.getresponse()
if not str(resp.status).startswith("2"):
print("Error: %d" % (resp.status))
print(resp.read())
sys.exit(1)
else:
# https://docs.github.com/en/rest/reference/pulls#create-a-pull-request
body = json.loads(resp.read().decode())
open(os.environ.get('PULLREQUEST_NUMBER_PATH'), 'w').write(f'{body["number"]}')
open(os.environ.get('PULLREQUEST_URL_PATH'), 'w').write(body["html_url"])
print("GitHub pull request created for $(params.REPO_FULL_NAME): "
f'number={body["number"]} url={body["html_url"]}')
volumeMounts:
- mountPath: /etc/github-open-pr
name: githubtoken
readOnly: true
volumes:
- name: githubtoken
secret:
secretName: $(params.GITHUB_TOKEN_SECRET_NAME)

1	Python script to create the pull request.
2	The token from the secret object.
3	The data we will send to GitHub.

Modify the Pipeline object

 - name: issue-prod-pull-request
params:
- name: GITHUB_HOST_URL
value: $(params.REPO_HOST)
- name: GITHUB_TOKEN_SECRET_NAME
value: github-token
- name: REPO_FULL_NAME
value: $(params.MANIFEST_REPO_NAME)
- name: HEAD
value: feature-for-$(params.COMMIT_SHA)
- name: BASE
value: main
- name: BODY
value: Update prod image for $(params.COMMIT_MESSAGE)
- name: TITLE
value: 'Production update: $(params.COMMIT_MESSAGE)'
runAfter:
- create-prod-manifest-branch
taskRef:
kind: Task
name: git-open-pull-request

Review the whole Pipeline

We did it, we created a Secure Supply Chain using Tekton Tasks. The full Pipeline now looks like this:

Figure 3. Pipeline Details

The last step will create a pull request on Git. When this request is approved and merged, the update will finally happen in the production environment. This is a manual process to have control what comes in production and what does not.

Execute full Pipeline E2E

It is time to execute the whole pipeline now end to end. We will do a real update to the application now, so we can see the differences.

As described in step 10, the DEV and PROD environments are running on the same cluster. In the field, this will probably not happen, but for now, it is good enough. GitOps/Argo CD monitors any changes and automatically updates whenever the Git repository (Kubernetes Manifests) is changed. During the PipelineRun we will update the image tag for DEV, which automatically rolls out and create a Pull request which is waiting for approval and will roll out the changes onto production.

Both environments have a route to access the application. At the moment both will look the same:

Figure 4. Globex DEV origin

Update application

The repository of Globex UI is forked at: https://github.com/tjungbauer/globex-ui. We used it throughout this journey to update the README.md file. The readme file does not really change anything. So, let’s update the UI itself.

look for the file src/index.html and add the following line before </body>

<center><strong>My very important update</strong></center>

Save this change and push it to GitHub. This will trigger the Pipeline which is running quite long. However, once it is finished, the DEV environment should now show the new line in the UI.

After the pipeline updated the image tag in Git, the GitOps process must fetch this change. This may take a while. You can speed this up by refreshing the "Application" inside the Argo CD interface. It should then automatically synchronize.

The update can now be seen in the browser. The "important update" is visible at the bottom of the page.

Figure 5. Globex DEV updated

The production environment was not yet updated. Instead, a pull request has been created:

Figure 6. Open pull request

This request can be reviewed and merged. As you can see there was only one change in the files:

Figure 7. Open pull request - changed files

Merge the pull request and wait until Argo CD fetched the changes and updates the production environment. This is it, the changes are done and promoted to production:

Figure 8. Globex PROD updated

Conclusion

This concludes this journey to a Secure Supply Chain using Tekton (OpenShift Pipelines). Is this the best must-have you need to do? No, it is an example, a demonstration. Feel free to use and modify it. You can also use other tools for the tasks or the pipeline as such. It does not matter if you use Tekton, Jenkins, Gitlab Runner etc. What is important is that you secure your whole supply chain as much as possible. Any image you create should be signed to ensure that the source can be trusted. Every source code should be verified against best practices and all images should be scanned for vulnerabilities and policy violations during the build AND the deployment process.

1	Default values for verifying SSL is "false" (Since I do not have certificates in place)
2	The default value for the Git URL is github.com

1	Name as referenced in the EventListener
2	Your super secure password

1	This task runs after pull-source-code but in parallels with the SonarQube task.
2	Task reference
3	Workspaces that are used in this Task
4	Additional workspace for the Pipeline

1	Use your appropriate name for the secret
2	Use your appropriate name for the secret, required to be able to sign the image