Step 5 - Build and Sign Image

OpenShift Supply Chain

- Thomas Jungbauer Thomas Jungbauer ( Lastmod: 2024-05-08 ) - 5 min read

Finally, after pulling and checking the code, we are going to create the image. During this process the image will be signed and uploaded to the public registry Quay.io.

Goals

The goals of this step are:

  • Build the image

  • Sign the image

  • Upload it Quay.io

You can use any registry you like.

Prerequisites

  • You need to have an account in Quay.io and create a repository there.

I have created the repository https://quay.io/repository/tjungbau/secure-supply-chain-demo which will contain the images and the signatures.

Steps

  1. Create a Robot account in Quay.io and get the authentication json. Go to "Account Settings > Robot Accounts" and create a new account.

    Quay Robot Account
    Figure 1. Quay Robot Account

  2. Allow this account WRITE permissions to the repository

    Quay Robot Account Permissions
    Figure 2. Quay Robot Account Permissions

  3. Select the just-created account and go to Kubernetes Secret.

    Here you can view or download the Secret. It should look like the following:

    apiVersion: v1
kind: Secret
metadata:
  name: tjungbau-secure-supply-chain-demo-pull-secret
data:
  .dockerconfigjson: <SECURE>
type: kubernetes.io/dockerconfigjson

    Copy the content and store it in your ci Namespace.

    The name will be probably different in other examples.

  4. To be able for the ServiceAccount, which is executing the Pipelines, to use this Secret, we need to modify the ServiceAccount object.

    In our case the ServiceAccount is called pipeline. Modify it and add the following lines:

    secrets:
...
  - name: tjungbau-secure-supply-chain-demo-pull-secret (1)
imagePullSecrets:
  - name: tjungbau-secure-supply-chain-demo-pull-secret (2)
    1Use your appropriate name for the secret
    2Use your appropriate name for the secret, required to be able to sign the image

  5. Update the Pipeline object and add the following block

    This time we do not need to add a Task object, because we are using a ClusterTask "buildah" that takes care of the building process. As workspace "shared-data" is used again, which has all the source code stored:

        - name: build-sign-image
      params:
        - name: TLSVERIFY
          value: $(params.TLSVERIFY)
        - name: BUILD_EXTRA_ARGS
          value: >-
            --label=io.openshift.build.commit.author='$(params.COMMIT_AUTHOR)'
            --label=io.openshift.build.commit.date='$(params.COMMIT_DATE)'
            --label=io.openshift.build.commit.id='$(params.COMMIT_SHA)'
            --label=io.openshift.build.commit.message='$(params.COMMIT_MESSAGE)'
            --label=io.openshift.build.commit.ref='$(params.GIT_REF)'
            --ulimit=nofile=4096:4096
        - name: IMAGE
          value: '$(params.IMAGE_REPO):$(params.IMAGE_TAG)'
      retries: 1
      runAfter:
        - scan-source
        - verify-commit-signature
      taskRef:
        kind: ClusterTask
        name: buildah
      workspaces:
        - name: source
          workspace: shared-data

  6. Update the TriggerBinding globex-ui and define the URL for your registry respectively for your image repository

    spec:
  parameters:
    - name: imageRepo
      value: quay.io/tjungbau/secure-supply-chain-demo
With this configuration, we could already build the image and push it into the registry. However, we also would like to sign it.

Prepare Tekton for Image Signing

To automatically sign images, we use TektonChains which is a controller that allows you to manage your supply chain security by signing TaskRuns and/or OCI images. Once the build-Task has pushed the images to Quay, Tekton will detect this event (by monitoring specific values of Task results) and then creates and pushes the signature for that image.

By default, Tekton Chains observes all task run executions. When the task runs complete, Tekton Chains signs and stores all artefacts.

Tekton Chain is part of OpenShift Pipelines and has the following main features:

  • You can sign task runs, task run results, and OCI registry images with cryptographic keys that are generated by tools such as cosign and skopeo.

  • You can use attestation formats such as in-toto.

  • You can securely store signatures and signed artefacts using OCI repository as a storage backend.

In our case, we will use CoSign for signing the images and Rekor for the attestation.

To activate image signing add the following to the TektonConfig object by removing the "chain{}" entry.

  chain:
    artifacts.oci.storage: oci (1)
    artifacts.taskrun.format: in-toto (2)
    artifacts.taskrun.storage: oci (3)
    transparency.enabled: true (4)
1The storage backend for storing OCI signatures.
2The format for storing TaskRun payloads. Can be in-toto or slsa/v1.
3The storage backend for TaskRun signatures. Multiple can be defined.
4Enable or disable automatic binary transparency uploads. The URL for uploading binary transparency attestations is https://rekor.sigstore.dev by default.
More details can be found at https://docs.openshift.com/container-platform/4.13/cicd/pipelines/using-tekton-chains-for-openshift-pipelines-supply-chain-security.html

CoSign - Signing the Image

We will use CoSign to sign our image. To do so we need to download the cosign binary and generate a key pair:

Be sure that you are logged into the OpenShift cluster.
cosign generate-key-pair k8s://openshift-pipelines/signing-secrets
Enter password for private key:
Enter password for private key again:
Successfully created secret signing-secrets in namespace openshift-pipelines

oc get secrets signing-secrets -n openshift-pipelines
NAME              TYPE     DATA   AGE
signing-secrets   Opaque   3      46h

Cosign will ask you to enter a password and will then create a Kubernetes secret in the Namespace openshift-pipelines.

When OpenShift Pipelines now execute a Task that is pushing an image, this Secret will be used to sign the image.

Let’s update our source code. The image will be built and pushed into Quay. The small black shield indicates that this image has been signed.

Quay Signed Image
Figure 3. Quay Signed Image

Besides the actual image the files: *.sig and *.att can be seen. The first one is the signature, the second one shows metadata about the attestation retrieved from Rekor which can be compared with the Rekor URL.

Rekor

Rekor’s goals are to provide an immutable tamper resistant ledger of metadata generated within a software projects supply chain. Rekor will enable software maintainers and build systems to record signed metadata to an immutable record. Other parties can then query said metadata to enable them to make informed decisions on trust and non-repudiation of an object’s lifecycle. For more details visit the sigstore website.

The Rekor project provides a restful API based server for validation and a transparency log for storage. A CLI application is available to make and verify entries, query the transparency log for inclusion proof, integrity verification of the transparency log or retrieval of entries by either public key or artifact. Rekor Git

An important feature of Tekton Chains is that it integrates seamlessly with an application called Rekor. The configuration transparency.enabled: true enables the call to the Rekor API that can be found by default at: https://rekor.sigstore.dev. This will create a transparency log which can be used for verification purposes later.

Of course, it would be possible to install your own server. Learn in the official Docs how this can be achieved.

This means every time our build-Tasks successfully completes a URL to the transparency logs will be attached to the Task.

After our PipelineRuns were successful, we can verify if the image has an entry in the Rekor transparency log. This is important to ensure that the image went through a valid signing process. At a later step (before we create the pull request for the production update) we will verify if the log exists.

The following annotations should have been added to the TaskRun or PipelineRun secure-supply-chain-XXX-build-sign-image automatically:

metadata:
  annotations:
    chains.tekton.dev/signed: 'true'
    chains.tekton.dev/transparency: 'https://rekor.sigstore.dev/api/v1/log/entries?logIndex=25593495'

This created the following transparency logs:

{
  "24296fb24b8ad77a7ae84f4b950336d1872a066aeb9463dfbc231a7d3b73dd040dd5faefb3c013a7": {
    "body": "eyJhcGlWZXJzaW9uIjoiMC4wLjEiLCJraW5kIjoiaGFzaGVkcmVrb3JkIiwic3BlYyI6eyJkYXRhIjp7Imhhc2giOnsiYWxnb3JpdGhtIjoic2hhMjU2IiwidmFsdWUiOiJmYzE2ODM3Mzc1ODUyNjc0MTQ4MjIyMDJhMWU2Y2RkZDkxNWI4NWUzYzhhNDVhZmI0YzEyYmE2YmNhMGNmNDQyIn19LCJzaWduYXR1cmUiOnsiY29udGVudCI6Ik1FWUNJUUNRMnlnU2ZGTllHS2pzbXRIYjVielAwdlRNMFgyNHVlNXlKbjJxdWFWSyt3SWhBSmhoSWpweEFybDJERjFsVmpMT0ZzVnRhMzJmbXJXRmxXWkdRQ015b2xrayIsInB1YmxpY0tleSI6eyJjb250ZW50IjoiTFMwdExTMUNSVWRKVGlCUVZVSk1TVU1nUzBWWkxTMHRMUzBLVFVacmQwVjNXVWhMYjFwSmVtb3dRMEZSV1VsTGIxcEplbW93UkVGUlkwUlJaMEZGVEc0clduaENNMHNyS3pWSk1pOWlNWFJqVFVKcVZXODRjWGx6YVFwNlQzUlpVbGRQUTBwTVRWQlNWVGR4WTFJeVMyWlZZazAzYlVwUlNtbHZjbXR6VW1KUmNHMTBTekV4WjJNM1pIVkZObGg0WmxOdlpWUkJQVDBLTFMwdExTMUZUa1FnVUZWQ1RFbERJRXRGV1MwdExTMHRDZz09In19fX0=",
    "integratedTime": 1688060588,
    "logID": "c0d23d6ad406973f9559f3ba2d1ca01f84147d8ffc5b8445c224f98b9591801d",
    "logIndex": 25593495,
    "verification": {
      "inclusionProof": {
        "checkpoint": "rekor.sigstore.dev - 2605736670972794746\n21430853\n10sx5mwekx2c8jltWbt0WeJu+tNM9FOBNQoyPiC3CPQ=\nTimestamp: 1688061275289390813\n\n— rekor.sigstore.dev wNI9ajBFAiEAyzFYKKghB+RQsbMIIzjywdA9vFjsusQoMWMUJPRjw3ECIDVeYSGKn7zwXWxGLhQlIGAI1Ca0Gu7ZuLJ64xR3SrY0\n",
        "hashes": [
          "5cc5dba1f5f420acc9ed049c77b04c4fdd66cf6ef1ebd46b66e26039bb9ba06c",
          "3041520c9fec6177225063053503f6d20b09e86b72968b737e3211a233dab6ce",
          "ddb57132bba122df920d4816af7ac02bef03aec533cc7b45f8552ce19c023d10",
          "8a510da356706a0a822daf3c3a935176d3634c6e0cce6830fe90a8771a3bc83d",
          "7d254234192620827306cc5024ffb8ae699bfe2725c8e6aa13757f5471d416d4",
          "648a4e45da960f4ad286dfa22ae93721c9ba82ee05edd5a7134d9b19e35735e0",
          "9dcf0f21690ec420bffbc1d10c383b7d3dc568d26bd8fd9e8771abb01f5976dc",
          "86575ffec011d78ad393e7af267693c0c59360e2299b308369951d8362032733",
          "0088ef1f6ec1e992fa6c91837287f2bd7eea238e5af7467e21e6df0dc8efe516",
          "b149d95903a4e554ac1c381f1afff31bb62b6e12b6b2fc95f36b8e198e1a42cd",
          "de73a694862cebd660ef1cecdd1bb66e273ab0651ca939bf7fa6f5f567bafe93",
          "a3c84734ddae3102952584443dea70e3dec2cc085af7cf0ba3530751966861ec",
          "0be5c7bbcf481d1efcfc63a27fce447cf6345f7bb3155cf5de39de592c16d52d",
          "f597f4bae8df3d6fc6eebfe3eabd7d393e08781f6f16b42398eca8512398fff1",
          "4e35fcb3c0a59e7f329994002b38db35f5d511f499ba009e10b31f5d27563607",
          "47044b7ac3aab820e44f0010538d7de71e17a11f4140cbbe9eeb37f78b77cc7d",
          "eee63677e2591eefe06ab537d6dd1b4060770682744c8287879b5dcd3365a5b2",
          "ff41aa21106dbe03996b4335dd158c7ffafd144e45022193de19b2b9136c3e42",
          "e6ebdeef2e23335d8d7049ba5a0049a90593efdfe9c1b4548946b44a19d7214f",
          "dd51e840e892d70093ad7e1db1e2dea3d50334c7345d360e444d22fc49ed9f5e",
          "ad712c98424de0f1284d4f144b8a95b5d22c181d4c0a246518e7a9a220bdf643"
        ],
        "logIndex": 21430064,
        "rootHash": "d74b31e66c1e931d9cf2396d59bb7459e26efad34cf45381350a323e20b708f4",
        "treeSize": 21430853
      },
      "signedEntryTimestamp": "MEYCIQDqmqb4k95FjiBNogOAmjTskkIPaGslgvrSND4pQdUALwIhAMt85yLsa5Ei+3NsmJ906/T9Hx1YZDDHQfHlTEYqqcaE"
    }
  }
}

Summary

Finally, we have a signed image uploaded to Quay. In addition, a transparency log has been created. Now let us add some security checks in the next steps.

See Also:

  • Introducing AdminNetworkPolicies –

    Classic Kubernetes/OpenShift offer a feature called NetworkPolicy that allows users to control the traffic to and from their assigned Namespace. NetworkPolicies are designed to give project owners or tenants the ability to protect their own namespace. Sometimes, however, I worked with customers where the cluster administrators or a dedicated (network) team need to enforce these policies. Since the NetworkPolicy API is namespace-scoped, it is not possible to enforce policies across namespaces. The only solution was to create custom (project) admin and edit roles, and remove the ability of creating, modifying or deleting NetworkPolicy objects. Technically, this is possible and easily done. But shifts the whole network security to cluster administrators. Luckily, this is where AdminNetworkPolicy (ANP) and BaselineAdminNetworkPolicy (BANP) comes into play.

  • Using Kustomize to post render a Helm Chart –

    Lately I came across several issues where a given Helm Chart must be modified after it has been rendered by Argo CD. Argo CD does a helm template to render a Chart. Sometimes, especially when you work with Subcharts or when a specific setting is not yet supported by the Chart, you need to modify it later …​ you need to post-render the Chart. In this very short article, I would like to demonstrate this on a real-live example I had to do. I would like to inject annotations to a Route objects, so that the certificate can be injected. This is done by the cert-utils operator. For the post-rendering the Argo CD repo pod will be extended with a sidecar container, that is watching for the repos and patches them if required.

  • Managing Certificates using GitOps approach –

    The article SSL Certificate Management for OpenShift on AWS explains how to use the Cert-Manager Operator to request and install a new SSL Certificate. This time, I would like to leverage the GitOps approach using the Helm Chart cert-manager I have prepared to deploy the Operator and order new Certificates. I will use an ACME Letsencrypt issuer with a DNS challenge. My domain is hosted at AWS Route 53. However, any other integration can be easily used.

  • Update Cluster Version using GitOps approach –

    During a GitOps journey at one point, the question arises, how to update a cluster? Nowadays it is very easy to update a cluster using CLI or WebUI, so why bother with GitOps in that case? The reason is simple: Using GitOps you can be sure that all clusters are updated to the correct, required version and the version of each cluster is also managed in Git. All you need is the channel you want to use and the desired cluster version. Optionally, you can define the exact image SHA. This might be required when you are operating in a restricted environment.

  • Multiple Sources for Applications in Argo CD –

    Argo CD or OpenShift GitOps uses Applications or ApplicationSets to define the relationship between a source (Git) and a cluster. Typically, this is a 1:1 link, which means one Application is using one source to compare the cluster status. This can be a limitation. For example, if you are working with Helm Charts and a Helm repository, you do not want to re-build (or re-release) the whole chart just because you made a small change in the values file that is packaged into the repository. You want to separate the configuration of the chart with the Helm package. The most common scenarios for multiple sources are (see: Argo CD documentation): Your organization wants to use an external/public Helm chart You want to override the Helm values with your own local values You don’t want to clone the Helm chart locally as well because that would lead to duplication and you would need to monitor it manually for upstream changes. This small article describes three different ways with a working example and tries to cover the advantages and disadvantages of each of them. They might be opinionated but some of them proved to be easier to use and manage.

Copyright © 2020 - 2024 Toni Schmidbauer & Thomas Jungbauer