Step 8 - Updating Kubernetes Manifests

OpenShift Supply Chain

- Thomas Jungbauer Thomas Jungbauer ( Lastmod: 2024-05-08 ) - 4 min read

With the finalization of the build process and the security checks during this phase, it is now time to update the Kubernetes manifests and provide the new tag for the created image. I have forked (and cleaned up) another repository that will store all yaml specifications that are required for OpenShift. You can find this repository at: Kubernetes Manifests

Goals

The goals of this step are:

  • Update the manifest in Git with the new image tag

Preparing the Pipeline

  1. Let’s create the Task object that will take care of the update.

    The Task will use git commands to update the manifests accordingly.

    apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: update-manifest
  namespace: ci
spec:
  description: >-
    This task updates the manifest for the current application to point to the
    image tag created with the short commit.
  params: (1)
    - description: Used to tag the built image.
      name: image
      type: string
    - default: main
      description: Target branch to push to
      name: target-branch
      type: string
    - default: Tekton Pipeline
      description: Git user name for performing the push operation.
      name: git_user_name
      type: string
    - default: tekton@tekton.com
      description: Git user email for performing the push operation.
      name: git_user_email
      type: string
    - description: File in which the image configuration is stored.
      name: configuration_file
      type: string
    - description: Repo in which the image configuration is stored.
      name: repository
      type: string
    - default: 'registry.redhat.io/openshift-pipelines/pipelines-git-init-rhel8:v1.10.4-4'
      name: gitInit
      type: string
    - name: verbose
      description: Verbose output
      type: string
      default: "true"
  steps: (2)
    - image: $(params.gitInit)
      name: git
      env:
        - name: PARAM_VERBOSE
          value: $(params.verbose)
      resources: {}
      script: >
        #!/usr/bin/env sh

        set -eu

        if [ "${PARAM_VERBOSE}" = "true" ] ; then
          set -x
        fi

        # Setting up the git config.

        git config --global user.email "$(params.git_user_email)"

        git config --global user.name "$(params.git_user_name)"

        # Checkout target branch to avoid the detached HEAD state

        TMPDIR=$(mktemp -d)

        cd $TMPDIR

        git clone $(params.repository)

        cd securing-software-supply-chain

        git checkout $(params.target-branch)

        # Set to the short commit value passed as parameter.
        # Notice the enclosing " to keep it as a string in the resulting YAML.

        IMAGE=\"$(params.image)\"

        sed -i "s#\(.*value:\s*\).*#\1 ${IMAGE}#" $(params.configuration_file)

        git add $(params.configuration_file)

        git commit -m "Automatically updated manifest to point to image tag

        $IMAGE"

        git push origin $(params.target-branch)
    1Required default parameters.
    2Script to clone and push the update to Git.

  2. To the Pipeline we have to add the following section

    spec:
  params: (1)
    - name: MANIFEST_FILE
      type: string
    - name: MANIFEST_FILE_PROD
      type: string
    - name: MANIFEST_REPO
      type: string
    - name: MANIFEST_REPO_NAME
      type: string
    - name: MANIFEST_GIT_REF
      type: string
    ...
  tasks:
  ...
    - name: update-dev-manifest
      params:
        - name: image
          value: '$(params.IMAGE_REPO):$(params.IMAGE_TAG)'
        - name: configuration_file
          value: $(params.MANIFEST_FILE)
        - name: repository
          value: $(params.MANIFEST_REPO)
        - name: git_user_name
          value: $(params.COMMIT_AUTHOR)
      runAfter: (2)
        - acs-image-check
        - acs-image-scan
        - generate-sbom
      taskRef:
        kind: Task
        name: update-manifest
    1Parameters that define the settings for the manifest repository
    2This time we are running after these three tasks

  3. Update the TriggerBinding globex-ui

        - name: manifestRepo
      value: git@github.com:tjungbauer/securing-software-supply-chain.git (1)
    - name: manifestRepoRef
      value: main
    - name: manifestFile (2)
      value: application/globex/overlays/dev/kustomization.yaml
    - name: manifestFileProd (3)
      value: application/globex/overlays/prod/kustomization.yaml
    - name: manifestRepoName
      value: tjungbauer/securing-software-supply-chain
    1The SSH url to GitHub
    2The overlay that will be used for the DEV environment
    3The overlay that will be used for the PROD environment

  4. Update the TriggerTemplate and add:

    spec:
  params:
    - description: The file to update to point to newly built image
      name: manifestFile
    - description: The file to update to point to newly built image in prod
      name: manifestFileProd
    - description: The repo to update to point to newly built image
      name: manifestRepo
    - description: The reference to the repo
      name: manifestRepoRef
    - description: The full name of the repo
      name: manifestRepoName
...
  resourcetemplates:
  ...
      spec:
        params:
          - name: MANIFEST_FILE
            value: $(tt.params.manifestFile)
          - name: MANIFEST_FILE_PROD
            value: $(tt.params.manifestFileProd)
          - name: MANIFEST_REPO
            value: $(tt.params.manifestRepo)
          - name: MANIFEST_GIT_REF
            value: $(tt.params.manifestRepoRef)
          - name: MANIFEST_REPO_NAME
            value: $(tt.params.manifestRepoName)
      ...

Git SSH Key

To be able to do changes in Git, which is using SSH keys to authenticate, we need to specify a Secret that knows the SSH private key and Github’s host keys.

Be sure that to create an SSH key (for example using ssh-keygen) and that GitHub knows about this key. You can add your key at "Account > Settings > SSH and GPG keys".

Run the following command to get the host keys of GitHub:

ssh-keyscan -H github.com

# github.com:22 SSH-2.0-babeld-c89ab1f3
# github.com:22 SSH-2.0-babeld-c89ab1f3
|1|cr4dkqIl2SnZqktvRsFnx1lA5Ag=|eie8EHXqQHgCZJq7F/TtRRZcBbc= ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk=
# github.com:22 SSH-2.0-babeld-c89ab1f3
|1|r6z4yeEV9Cog/2I4stZp1A34BAE=|EU8VcdD+KHMJJt9uPL4jh5zK0fI= ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg=
# github.com:22 SSH-2.0-babeld-c89ab1f3
|1|hVnTyBzb/gSR8jpz+NUziUkHy1A=|ENMDVCVsLfz2CWLa1C+BnzZI8Yg= ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl
# github.com:22 SSH-2.0-babeld-c89ab1f3

Finally, we need to create a Secret with the following content:

apiVersion: v1
kind: Secret
metadata:
  annotations:
    tekton.dev/git-0: github.com (1)
  name: git-ssh-key
  namespace: ci
type: kubernetes.io/ssh-auth
data:
  ssh-privatekey: <BASE64 PRIVAT SSH KEY > (2)
  known_hosts: < BASE64 of Githubs hostkey> (3)
1This annotation defines for which host Tekton will inject the Secret.
2A base64 decoded string of your PRIVATE ssh key.
3A base64 decoded string of the host keys

Update the pipeline ServiceAccount and add the new Secret:

secrets:
...
  - name: git-ssh-key

Execute the Pipeline

Time to trigger the Pipeline, which now looks like:

Pipeline Details
Figure 1. Pipeline Details

The new Task will automatically push the new image tag to the Kubernetes manifests Git repository. From there it can later be used to roll out the new version.

In Git you can track the changes:

Git Commit
Figure 2. Git Commit

Summary

We have now updated the Kubernetes manifests with the new image tag. In the next steps, we will verify these manifests (linting) and then try to deploy the new image on the DEV environment.

See Also:

  • Introducing AdminNetworkPolicies –

    Classic Kubernetes/OpenShift offer a feature called NetworkPolicy that allows users to control the traffic to and from their assigned Namespace. NetworkPolicies are designed to give project owners or tenants the ability to protect their own namespace. Sometimes, however, I worked with customers where the cluster administrators or a dedicated (network) team need to enforce these policies. Since the NetworkPolicy API is namespace-scoped, it is not possible to enforce policies across namespaces. The only solution was to create custom (project) admin and edit roles, and remove the ability of creating, modifying or deleting NetworkPolicy objects. Technically, this is possible and easily done. But shifts the whole network security to cluster administrators. Luckily, this is where AdminNetworkPolicy (ANP) and BaselineAdminNetworkPolicy (BANP) comes into play.

  • Using Kustomize to post render a Helm Chart –

    Lately I came across several issues where a given Helm Chart must be modified after it has been rendered by Argo CD. Argo CD does a helm template to render a Chart. Sometimes, especially when you work with Subcharts or when a specific setting is not yet supported by the Chart, you need to modify it later …​ you need to post-render the Chart. In this very short article, I would like to demonstrate this on a real-live example I had to do. I would like to inject annotations to a Route objects, so that the certificate can be injected. This is done by the cert-utils operator. For the post-rendering the Argo CD repo pod will be extended with a sidecar container, that is watching for the repos and patches them if required.

  • Managing Certificates using GitOps approach –

    The article SSL Certificate Management for OpenShift on AWS explains how to use the Cert-Manager Operator to request and install a new SSL Certificate. This time, I would like to leverage the GitOps approach using the Helm Chart cert-manager I have prepared to deploy the Operator and order new Certificates. I will use an ACME Letsencrypt issuer with a DNS challenge. My domain is hosted at AWS Route 53. However, any other integration can be easily used.

  • Update Cluster Version using GitOps approach –

    During a GitOps journey at one point, the question arises, how to update a cluster? Nowadays it is very easy to update a cluster using CLI or WebUI, so why bother with GitOps in that case? The reason is simple: Using GitOps you can be sure that all clusters are updated to the correct, required version and the version of each cluster is also managed in Git. All you need is the channel you want to use and the desired cluster version. Optionally, you can define the exact image SHA. This might be required when you are operating in a restricted environment.

  • Multiple Sources for Applications in Argo CD –

    Argo CD or OpenShift GitOps uses Applications or ApplicationSets to define the relationship between a source (Git) and a cluster. Typically, this is a 1:1 link, which means one Application is using one source to compare the cluster status. This can be a limitation. For example, if you are working with Helm Charts and a Helm repository, you do not want to re-build (or re-release) the whole chart just because you made a small change in the values file that is packaged into the repository. You want to separate the configuration of the chart with the Helm package. The most common scenarios for multiple sources are (see: Argo CD documentation): Your organization wants to use an external/public Helm chart You want to override the Helm values with your own local values You don’t want to clone the Helm chart locally as well because that would lead to duplication and you would need to monitor it manually for upstream changes. This small article describes three different ways with a working example and tries to cover the advantages and disadvantages of each of them. They might be opinionated but some of them proved to be easier to use and manage.

Copyright © 2020 - 2024 Toni Schmidbauer & Thomas Jungbauer