Using Kustomize to post render a Helm Chart

OpenShift GitOps

- Thomas Jungbauer Thomas Jungbauer ( Lastmod: 2024-10-13 ) - 5 min read

Lately I came across several issues where a given Helm Chart must be modified after it has been rendered by Argo CD. Argo CD does a helm template to render a Chart. Sometimes, especially when you work with Subcharts or when a specific setting is not yet supported by the Chart, you need to modify it later …​ you need to post-render the Chart.

In this very short article, I would like to demonstrate this on a real-live example I had to do. I would like to inject annotations to a Route objects, so that the certificate can be injected. This is done by the cert-utils operator. For the post-rendering the Argo CD repo pod will be extended with a sidecar container, that is watching for the repos and patches them if required.

Everything below is using OpenShift Gitops Operator. This is based on Argo CD, but instead of directly modifying the repo Deployment, we will modify the Argo CD Custom Resource.
In the future it will be easier to inject certificates into a Route, by defining a Secret. This as currently a TechPreview feature (OpenShift 4.17). Creating a route with externally managed certificate

The Route Object

Imagine we have the following Route object, rendered via Helm template:

---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: my-route
  namespace: my-namespace
spec:
  host: my.route.apps.cluster.name
  port:
    targetPort: http
  tls:
    insecureEdgeTerminationPolicy: Redirect
    termination: edge
  to:
    kind: Service
    name: my-service
    weight: 100
  wildcardPolicy: None

The cert-manager Operator requested a certificate which can be found in the Secret "my-certificate ". To let the cert-utils Operator inject the data from the certificate automatically, we need to add annotations to that Route object.

This injection is usually a good idea, since we do not want to define certificate and (private) key directly in the Route object using our Chart.

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  annotations: (1)
    cert-manager.io/cluster-issuer: my-issuer
    cert-utils-operator.redhat-cop.io/certs-from-secret: my-certificate
1Two annotations shall be added to the Route object.
In this example certificates have to be ordered. No wildcard certificate is available.

Post-Rendering

To modify the output after it has been rendered by Argo CD we will use Kustomize patch feature. This means, after the template has been rendered, we send it to Kustomize and let it patch it.

Let’s go through the steps one-by-one:

  1. Create a kustomization.yaml Place the following file next to your Chart.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: my-namespace

resources:
  - ./all.yaml (1)

patches:
  - patch: |
      - op: add  (2)
        path: /metadata/annotations
        value:
          cert-manager.io/cluster-issuer: my-issuer
          cert-utils-operator.redhat-cop.io/certs-from-secret: my-certificate
    target:
      kind: Route
      name: my-route (3)
1The all.yaml file will be created by the helm template command.
2Add the annotations to the Route object.
3The name of the Route object.

This will patch the Route object. You can test this locally by execute the command: helm template . > all.yaml && kustomize build && rm all.yaml

  1. Create an empty file called my-cmp-plugin into the folder next to the Chart.yaml I will explain in a bit why I chose to use this approach.

  2. Create the following ConfigMap in the OpenShift GitOps namespace (for example openshift-gitops)

kind: ConfigMap
apiVersion: v1
metadata:
  name: my-cmp-plugin
  namespace: openshift-gitops
data:
  plugin.yaml: |-
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: my-cmp-plugin (1)
    spec:
      version: v1.0
      init: (2)
        command: [sh, -c, 'echo "Initializing my-plugin-cmp..."', 'helm dependency build || true']
      generate: (3)
        command: [sh, -c, "helm template . --name-template $ARGOCD_APP_NAME --namespace $ARGOCD_APP_NAMESPACE --include-crds > all.yaml && kustomize build"]
      discover: (4)
        find:
          glob: "**/my-cmp-plugin"
1The name of the plugin.
2The init command will be executed once, when the plugin is loaded.
3The generate command will be executed every time the plugin is called.
4The discovery command will be executed to find the plugin.

This will execute the command to generate a helm template, pipe the output into all.yaml and let Kustomize patch the output. The "discovery" part is looking for a specific file in the repository. I thought this might be useful to pin down this plugin to specific repositories only. However, there are other ways to implement this. You could omit this part and define the name of the plugin inside the Argo CD Application too for example.

  1. Patching Argo CD Repo server

Now it is time to patch our repo server specification of the Argo CD custom resource. The following should do it:

As image for the sidecar container, I am using Gerald Nunn’s tool image. You can use your own image, as long as Helm and Kustomize are available.
apiVersion: argoproj.io/v1alpha1
kind: ArgoCD
metadata:
  name: openshift-gitops
  namespace: openshift-gitops
spec:
[...]
  repo:
    - configMap: (1)
        name: my-cmp-plugin
      name: my-cmp-plugin
    sidecarContainers: (2)
      - name: my-cmp-plugin
        command: [/var/run/argocd/argocd-cmp-server]
        env:
          - name: APP_ENV
            value: prod
        image: quay.io/gnunn/tools:latest (3)
        imagePullPolicy: Always
        securityContext:
          runAsNonRoot: true
        volumeMounts: (4)
          - mountPath: /var/run/argocd
            name: var-files
          - mountPath: /home/argocd/cmp-server/plugins
            name: plugins
          - mountPath: /tmp
            name: tmp
          - mountPath: /home/argocd/cmp-server/config/plugin.yaml
            subPath: plugin.yaml
            name: my-cmp-plugin
    volumes: (5)
      - configMap:
          name: cloudbees-cmp-plugin
        name: cloudbees-cmp-plugin
1The name of the ConfigMap that was created in step 2.
2The sidecar container specification.
3The image that is used for the sidecar container.
4The volume mounts for the sidecar container.
5The volumes for the sidecar container.

As soon as the repo Pod has been patched a 2nd container inside the Pod will be started as a sidecar. This will take the ConfigMap that was created in step 2 and mount it. As soon as a repo is found where this patch shall be executed, Argo CD will perform the actions defined in the ConfigMap, resulting in the output of the helm template and the patched output of Kustomize.

---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: my-route
  namespace: my-namespace
  annotations: (1)
    cert-manager.io/cluster-issuer: my-issuer
    cert-utils-operator.redhat-cop.io/certs-from-secret: my-certificate
spec:
  host: my.route.apps.cluster.name
  port:
    targetPort: http
  tls:
    insecureEdgeTerminationPolicy: Redirect
    termination: edge
  to:
    kind: Service
    name: my-service
    weight: 100
  wildcardPolicy: None
1The annotations that are added to the Route.

This is it; this will patch our resource. Such post-renderer can be used for other patches as well. For example, to remove certain items from an object.

2nd Example

In my real-live example I had the problem that the path was empty in the Helm Chart and OpenShift automatically removed that, which was shown as out-of-sync in Argo CD.

So I am using the patch to remove the path.

Only do this if you are sure the element is really empty!

I extended the kustomization.yaml with

      - op: remove
        path: /spec/path

so it looks like:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: my-namespace

resources:
  - ./all.yaml

patches:
  - patch: |
      - op: add
        path: /metadata/annotations
        value:
          cert-manager.io/cluster-issuer: my-issuer
          cert-utils-operator.redhat-cop.io/certs-from-secret: my-certificate
      - op: remove (1)
        path: /spec/path
    target:
      kind: Route
      name: my-route
1The patch that removes the path.

This 2nd patch will completely remove the /spec/path from the Route object named my-route.

Further information:

See Also:

  • Managing Certificates using GitOps approach –

    The article SSL Certificate Management for OpenShift on AWS explains how to use the Cert-Manager Operator to request and install a new SSL Certificate. This time, I would like to leverage the GitOps approach using the Helm Chart cert-manager I have prepared to deploy the Operator and order new Certificates. I will use an ACME Letsencrypt issuer with a DNS challenge. My domain is hosted at AWS Route 53. However, any other integration can be easily used.

  • Update Cluster Version using GitOps approach –

    During a GitOps journey at one point, the question arises, how to update a cluster? Nowadays it is very easy to update a cluster using CLI or WebUI, so why bother with GitOps in that case? The reason is simple: Using GitOps you can be sure that all clusters are updated to the correct, required version and the version of each cluster is also managed in Git. All you need is the channel you want to use and the desired cluster version. Optionally, you can define the exact image SHA. This might be required when you are operating in a restricted environment.

  • Multiple Sources for Applications in Argo CD –

    Argo CD or OpenShift GitOps uses Applications or ApplicationSets to define the relationship between a source (Git) and a cluster. Typically, this is a 1:1 link, which means one Application is using one source to compare the cluster status. This can be a limitation. For example, if you are working with Helm Charts and a Helm repository, you do not want to re-build (or re-release) the whole chart just because you made a small change in the values file that is packaged into the repository. You want to separate the configuration of the chart with the Helm package. The most common scenarios for multiple sources are (see: Argo CD documentation): Your organization wants to use an external/public Helm chart You want to override the Helm values with your own local values You don’t want to clone the Helm chart locally as well because that would lead to duplication and you would need to monitor it manually for upstream changes. This small article describes three different ways with a working example and tries to cover the advantages and disadvantages of each of them. They might be opinionated but some of them proved to be easier to use and manage.

  • Installing OpenShift Logging using GitOps –

    OpenShift Logging is one of the more complex things to install and configure on an OpenShift cluster. Not because the service or Operators are so complex to understand, but because of the dependencies logging has. Besides the logging operator itself, the Loki operator is required, the Loki operator requires access to an object storage, that might be configured or is already available. In this article, I would like to demonstrate the configuration of the full stack using an object storage from OpenShift Data Foundation. This means: Installing the logging operator into the namespace openshift-logging Installing the Loki operator into the namespace openshift-operators-redhat Creating a new BackingStore and BucketClass Generating the Secret for Loki to authenticate against the object storage Configuring the LokiStack resource Configuring the ClusterLogging resource All steps will be done automatically. In case you have S3 storage available, or you are not using OpenShift Data Foundation, the setup will be a bit different. For example, you do not need to create a BackingStore or the Loki authentication Secret.

  • Configure Buckets in MinIO using GitOps –

    MinIO is a simple, S3-compatible object storage, built for high-performance and large-scale environments. It can be installed as an Operator to Openshift. In addition, to a command line tool, it provides a WebUI where all settings can be done, especially creating and configuring new buckets. Currently, this is not possible in a declarative GitOps-friendly way. Therefore, I created the Helm chart minio configurator, that will start a Kubernetes Job, which will take care of the configuration. Honestly, when I say I have created it, the truth is, that it is based on an existing MinIO Chart by Bitnami, that does much more than just set up a bucket. I took out the bucket configuration part, streamlined it a bit and added some new features, which I required. This article shall explain how to achieve this.

Copyright © 2020 - 2024 Toni Schmidbauer & Thomas Jungbauer