Managing Certificates using GitOps approach

OpenShift GitOps

- Thomas Jungbauer Thomas Jungbauer ( Lastmod: 2024-07-04 ) - 6 min read

The article SSL Certificate Management for OpenShift on AWS explains how to use the Cert-Manager Operator to request and install a new SSL Certificate. This time, I would like to leverage the GitOps approach using the Helm Chart cert-manager I have prepared to deploy the Operator and order new Certificates.

I will use an ACME Letsencrypt issuer with a DNS challenge. My domain is hosted at AWS Route 53.

However, any other integration can be easily used.

Before we start

Before we start, be sure that Route 53 is configured correctly. The required settings and commands are described at Configure an AWS User for Accessing Route 53

Deploy the Operator

The first step is to deploy the Operator to our cluster. This is done using GitOps and the Helm Chart is located at my Helm repository: https://github.com/tjungbauer/openshift-clusterconfig-gitops/tree/main/clusters/management-cluster/cert-manager

The configuration looks like below. It takes care to:

  1. Deploy the Operator cert-manager-operator

  2. Verify if the Operator has been deployed successfully

  3. Configure Cert-Manager

    1. Create a ClusterIssuer using route53 integration. (You can configure any other configuration too)

    2. Patch the Operator with "overrideArgs". This is required for AWS Route 53 where we need to define which DNS resolvers shall be used.

All these settings are handed over to the appropriate sub-charts. Like helper-operator, helper-status-checker and cert-manager

# Install Operator Compliance Operator
# Deploys Operator --> Subscription and Operatorgroup
# Syncwave: 0
helper-operator: (1)
  operators:
    compliance-operator:
      enabled: true
      syncwave: '0'
      namespace:
        name: cert-manager-operator
        create: true
      subscription:
        channel: stable-v1
        approval: Automatic
        operatorName: openshift-cert-manager-operator
        source: redhat-operators
        sourceNamespace: openshift-marketplace
      operatorgroup:
        create: true
        notownnamespace: false

helper-status-checker: (2)
  enabled: true

  checks:
    - operatorName: cert-manager-operator
      namespace:
        name: cert-manager-operator
      serviceAccount:
        name: "status-checker-cert-manager"

cert-manager: (3)
  certManager:
    enable_patch: true

    overrideArgs: (4)
      - '--dns01-recursive-nameservers-only'
      - --dns01-recursive-nameservers=ns-362.awsdns-45.com:53,ns-930.awsdns-52.net:53

  issuer: (5)
    - name: letsencrypt-prod
      type: ClusterIssuer
      enabled: true
      syncwave: 20

      acme:
        email: tjungbau@redhat.com

        solvers:
          - dns01: (6)
              route53:
                accessKeyIDSecretRef:
                  key: access-key-id
                  name: prod-route53-credentials-secret
                region: us-west-1
                secretAccessKeySecretRef:
                  key: secret-access-key
                  name: prod-route53-credentials-secret
            selector:
              dnsZones:
                - aws.ispworld.at
1Installing the Operator
2Verify if the Operator has been successfully deployed
3Configure the Cert-Manager Operator
4Override the DNS resolver args
5Configure the ClusterIssuer
6Use the solver dns01.route53.
Verify the README of the Helm Chart cert-manager for additional possibilities in the configuration.

One additional piece is missing before we can finally start the deployment.

As you can see in the values file above the accessKey and secretAccessKey are stored in the secret named prod-route53-credentials-secret.

This means, that a secret is required with the keys that have been provided by AWS when you configured the Route 53 access:

kind: Secret
apiVersion: v1
metadata:
  name: prod-route53-credentials-secret
  namespace: cert-manager (1)
data:
  access-key-id: <AccessKey>
  secret-access-key: <Secret Access Key>
type: Opaque
1Namespace of the Secret, here the Operator is managing the Certificate Controller.

I stored this Secret as SealedSecret and put it into the cluster configuration folder. From here, Argo CD will pick it up and deploy it.

Never, never ever store a Secret object directly in Git. Secret objects are not encrypted but encoded. Everybody could decode the data. With Sealed Secrets or any other Secret Management, you are able to prepare these objects and store or retrieve them.

Finally, with these settings, the Operator can be deployed. This is managed by OpenShift GitOps (Argo CD). As soon as the Operator is ready, we can start requesting certificates as we automatically created the ClusterIssuer (letsencrypt-prod)

Deploying and Configuring Cert-Manager Operator
Figure 1. Deploying and Configuring Cert-Manager Operator

Two certificates are of special interest :

  1. Default IngressController of OpenShift

  2. OpenShift’s API

Therefore, let’s request and configure them.

Requesting a Certificate

The chart cert-manager can render a Certificate resource as well. I tried to support any possible setting. However, not everything, especially the non-stable ones, is available yet.

The official Cert-Manager documentation explains how to create such Certificate Resource.

The chart README explains which settings are supported by the chart.

Not every setting is required and some will set default values. The minimum parameters are: name, namespace, secretName, dnsNames and reference to an issuer.

Requesting IngressController Certificate

The default IngressController of OpenShift listens on the wildcard domain *.apps.clustername.basedomain. In my examples, you will see *.apps.ocp.aws.ispworld.at

The IngressController configuration must be modified to reference the Secret object the cert-manager will generate once the Certificate has been successfully requested. The cluster configuration Ingresscontroller defines the required parameters:

---
# -- Define ingressControllers
# Multiple might be defined.
ingresscontrollers: (1)
    # -- Name of the IngressController. OpenShift initial IngressController is called 'default'.
  - name: default

    # -- Enable the configuration
    # @default -- false
    enabled: true

    # -- Number of replicas for this IngressController
    # @default -- 2
    replicas: 3

    # -- The name of the secret that stores the certificate information for the IngressController
    # @default -- N/A
    defaultCertificate: router-certificate (2)

    # -- Bind IngressController to specific nodes
    # Here as example for Infrastructure nodes.
    # @default -- empty
    #nodePlacement:

      # NodeSelector that shall be used.
    #  nodeSelector: (3)
    #    key: node-role.kubernetes.io/infra
    #    value: ''

    #  # -- Tolerations, required if the nodes are tainted.
    #  tolerations:
    #    - effect: NoSchedule
    #      key: node-role.kubernetes.io/infra
    #      operator: Equal
    #      value: reserved
    #    - effect: NoExecute
    #      key: node-role.kubernetes.io/infra
    #      operator: Equal
    #      value: reserved

certificates:
  enabled: true

  # List of certificates
  certificate: (4)
    - name: router-certificate
      enabled: true
      namespace: openshift-ingress
      syncwave: "0"
      secretName: router-certificate (5)

      dnsNames: (5)
        - apps.ocp.aws.ispworld.at
        - '*.apps.ocp.aws.ispworld.at'

      # Reference to the issuer that shall be used.
      issuerRef: (6)
        name: letsencrypt-prod
        kind: ClusterIssuer
1Configuration for the IngressController
2Reference to the Secret that will store the Certificate
3Optional tolerations that can be configured for the IngressController
4List of Certificates to order
5List of domainnames for the IngressController. Here 2 are used, the wildcard domain and the base domain of that wildcard.
6Reference to the issuer (in this case ClusterIssuer)

This will request the Certificate:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: router-certificate
  namespace: openshift-ingress
spec:
  dnsNames:
    - apps.ocp.aws.ispworld.at
    - '*.apps.ocp.aws.ispworld.at'
  duration: 2160h0m0s
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-prod
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    rotationPolicy: Always
  secretName: router-certificate
It may take a while until the Certificate request is approved.

The IngressController will update the reference to the secret and restart the ingress pods:

apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: default
  namespace: openshift-ingress-operator
spec:
[...]
  defaultCertificate:
    name: router-certificate

Once all pods have been successfully restarted, open a new browser, or reload or open a private window to verify the certificate that is provided by the application.

Requesting APIServer Certificate

Requesting the certificate for the OpenShift API follows the same rules as for the IngressController. The example can be found at: Clusterconfig APIServer

The values file may look like the following for example:

In this case, not only the Certificate is requested but the ETCD encryption is also enabled. The reason for that is, that both settings are done in the same Kubernetes resource (apiserver). If we split this up into 2 Argo CD Applications one of them will always show a warning that the same resource is managed by another Argo CD Application.
---
# -- Using subchart generic-cluster-config
generic-cluster-config:
  apiserver: (1)
    enabled: true

    # audit configuration
    audit:
      profile: Default

    # Configure a custom certificate for the API server
    custom_cert:
      enabled: true

      cert_names: (2)
        - api.ocp.aws.ispworld.at

      secretname: api-certificate (3)

    etcd_encryption: (4)
      enabled: true
      encryption_type: aesgcm (5)

      # -- Namespace where Job is executed that verifies the status of the encryption
      namespace: kube-system

      serviceAccount:
        create: true
        name: "etcd-encryption-checker" (6)

cert-manager: (7)
  enabled: true

  certificates:
    enabled: true

    # List of certificates
    certificate:
      - name: api-certificate
        enabled: true
        namespace: openshift-config
        syncwave: "0"
        secretName: api-certificate

        dnsNames:
          - api.ocp.aws.ispworld.at

        # Reference to the issuer that shall be used.
        issuerRef:
          name: letsencrypt-prod
          kind: ClusterIssuer
1Settings for the APIServer object.
2The name of the domain the certificate will be responsible for.
3Reference to the Secret that will store the Certificate
4Enable ETCD encryption
5Tpee of encryption
6Service Account that will be created and used by a Job that will verify when and if the encryption has been finished successfully.
7Settings for the Certificate. Similar to the settings of the IngressController.

The configuration is more or less similar to the IngressController. Again the APIServer will restart and once done, the Certificate is used by the cluster.

Conclusion

With this, very short, article I have tried to easily explain how to deploy the Cert-Manager Operator and request Certificates. Different Helm Charts are used, but the main one is cert-manager.

The cluster configuration repository https://github.com/tjungbauer/openshift-clusterconfig-gitops then use this chart to configure the required resources.

With the support of this Helm Chart anybody in the Cluster can request Certificates which are then managed by the Cert-Manager Operator.

See Also:

  • Introducing AdminNetworkPolicies –

    Classic Kubernetes/OpenShift offer a feature called NetworkPolicy that allows users to control the traffic to and from their assigned Namespace. NetworkPolicies are designed to give project owners or tenants the ability to protect their own namespace. Sometimes, however, I worked with customers where the cluster administrators or a dedicated (network) team need to enforce these policies. Since the NetworkPolicy API is namespace-scoped, it is not possible to enforce policies across namespaces. The only solution was to create custom (project) admin and edit roles, and remove the ability of creating, modifying or deleting NetworkPolicy objects. Technically, this is possible and easily done. But shifts the whole network security to cluster administrators. Luckily, this is where AdminNetworkPolicy (ANP) and BaselineAdminNetworkPolicy (BANP) comes into play.

  • Using Kustomize to post render a Helm Chart –

    Lately I came across several issues where a given Helm Chart must be modified after it has been rendered by Argo CD. Argo CD does a helm template to render a Chart. Sometimes, especially when you work with Subcharts or when a specific setting is not yet supported by the Chart, you need to modify it later …​ you need to post-render the Chart. In this very short article, I would like to demonstrate this on a real-live example I had to do. I would like to inject annotations to a Route objects, so that the certificate can be injected. This is done by the cert-utils operator. For the post-rendering the Argo CD repo pod will be extended with a sidecar container, that is watching for the repos and patches them if required.

  • Update Cluster Version using GitOps approach –

    During a GitOps journey at one point, the question arises, how to update a cluster? Nowadays it is very easy to update a cluster using CLI or WebUI, so why bother with GitOps in that case? The reason is simple: Using GitOps you can be sure that all clusters are updated to the correct, required version and the version of each cluster is also managed in Git. All you need is the channel you want to use and the desired cluster version. Optionally, you can define the exact image SHA. This might be required when you are operating in a restricted environment.

  • Multiple Sources for Applications in Argo CD –

    Argo CD or OpenShift GitOps uses Applications or ApplicationSets to define the relationship between a source (Git) and a cluster. Typically, this is a 1:1 link, which means one Application is using one source to compare the cluster status. This can be a limitation. For example, if you are working with Helm Charts and a Helm repository, you do not want to re-build (or re-release) the whole chart just because you made a small change in the values file that is packaged into the repository. You want to separate the configuration of the chart with the Helm package. The most common scenarios for multiple sources are (see: Argo CD documentation): Your organization wants to use an external/public Helm chart You want to override the Helm values with your own local values You don’t want to clone the Helm chart locally as well because that would lead to duplication and you would need to monitor it manually for upstream changes. This small article describes three different ways with a working example and tries to cover the advantages and disadvantages of each of them. They might be opinionated but some of them proved to be easier to use and manage.

  • Installing OpenShift Logging using GitOps –

    OpenShift Logging is one of the more complex things to install and configure on an OpenShift cluster. Not because the service or Operators are so complex to understand, but because of the dependencies logging has. Besides the logging operator itself, the Loki operator is required, the Loki operator requires access to an object storage, that might be configured or is already available. In this article, I would like to demonstrate the configuration of the full stack using an object storage from OpenShift Data Foundation. This means: Installing the logging operator into the namespace openshift-logging Installing the Loki operator into the namespace openshift-operators-redhat Creating a new BackingStore and BucketClass Generating the Secret for Loki to authenticate against the object storage Configuring the LokiStack resource Configuring the ClusterLogging resource All steps will be done automatically. In case you have S3 storage available, or you are not using OpenShift Data Foundation, the setup will be a bit different. For example, you do not need to create a BackingStore or the Loki authentication Secret.

Copyright © 2020 - 2024 Toni Schmidbauer & Thomas Jungbauer