SSL Certificate Management for OpenShift on AWS

OpenShift

- Thomas Jungbauer Thomas Jungbauer ( Lastmod: 2024-05-08 ) - 6 min read

Finally, after a long time on my backlog, I had some time to look into the Cert-Manager Operator and use this Operator to automatically issue new SSL certificates. This article shall show step-by-step how to create a certificate request and use this certificate for a Route and access a service via your Browser. I will focus on the technical part, using a given domain on AWS Route53.

Introduction

After a new OpenShift Cluster has been deployed, self-signed certificates are used to access the Routes (for example the console) and the API. Typically, an application is exposed to the world using the schema <app-name>-<namespace-name>.apps.<clusterdomain>.

We will try to create a certificate for a specific application with a custom domain name and for the cluster domains: *.apps.clusterdomain and api.clusterdomain.

The domain is already available and delegated to AWS Route53. As certificate authority, I am using Let’s Encrypt.

We will install 2 operators:

  1. Cert Manager: to issue new certificates.

  2. Cert Utils Operator: injects the certificate into a Route object.

The Cert Utils Operator can provide additional information for a certificate and monitors the expiration date. Here we mainly use it to automatically inject Route objects by defining specific annotations.

Prerequisites

  1. OpenShift cluster with a user that has privileges to install Operators.

  2. A domain hosted for example at Route 53

  3. Credentials for your Cloud Provider (AWS)

Deploy an Example Application

Let’s use the super complex demo application bookimport.

oc new-project bookimport
oc apply -f https://raw.githubusercontent.com/tjungbauer/book-import/master-no-pre-post/book-import/deployment.yaml -n bookimport
oc apply -f https://raw.githubusercontent.com/tjungbauer/book-import/master-no-pre-post/book-import/service.yaml -n bookimport
oc expose service book-import -n bookimport
oc get route -n bookimport

The last command will print you an URL which, copied into the browser, will open our application:

Bookimport
Figure 1. Application Book Import using HTTP

As you can see in the address line the connection is not secured (Nicht sicher in German) and my domain is *.apps.ocp.aws.ispworld.at

Configure an AWS user for accessing Route 53

On AWS I have currently 2 Zones, the public aws.ispworld.at and a private zone, created by the OpenShift Installer.

DomainZones
Figure 2. Domain Zones

Before you can manage your domains a user with appropriate privileges must be created.

Store the following in the file policy.json. This will allow a user to perform DNS Upgrades.

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Effect": "Allow",
           "Action": "route53:GetChange",
           "Resource": "arn:aws:route53:::change/*"
       },
       {
           "Effect": "Allow",
           "Action": [
               "route53:ChangeResourceRecordSets",
               "route53:ListResourceRecordSets"
           ],
           "Resource": "arn:aws:route53:::hostedzone/*"
       },
       {
           "Effect": "Allow",
           "Action": [
               "route53:ListHostedZones",
               "route53:ListResourceRecordSets",
               "route53:ListHostedZonesByName"
           ],
           "Resource": "*"
       }
   ]
}

Apply the new policy to AWS and store the ARN into a variable:

aws iam create-policy --policy-name AllowDNSUpdates --policy-document  file://policy.json
export POLICY_ARN=$(aws iam list-policies --query 'Policies[?PolicyName==`AllowDNSUpdates`].Arn' --output text)

Create the user route53-openshift and assign the policy to that user:

aws iam create-user --user-name route53-openshift
aws iam attach-user-policy --policy-arn $POLICY_ARN --user-name route53-openshift

Finally, create the access key and store the AccessKeyId and the SecretAccessKey for later use:

aws iam create-access-key --user-name route53-openshift --output json

{
    "AccessKey": {
        "UserName": "route53-openshift",
        "AccessKeyId": "XXXXXXXXXXXXXX",
        "Status": "Active",
        "SecretAccessKey": "XXXXXXXXXXXXXXXXXXX",
        "CreateDate": "2023-02-15T12:34:06+00:00"
    }
}

Installing Operators to OpenShift

We will install 2 Operators to our cluster:

  1. cert-manager Operator for Red Hat OpenShift

  2. Cert Utils Operator

Simply search both on OLM and install them keeping the default values.

The Cert Utils Operator is a Community Operator.
Operators
Figure 3. Operators

This will install the Cert-Manager into the namespace openshift-cert-manager

Configure the Cert-Manager Operator

Before we can issue a certificate, we need to create a secret with our AWS SecretAccessKey (see above):

oc create secret generic prod-route53-credentials-secret --from-literal secret-access-key="XXXXXXXXXXXXXXXXXXX" -n openshift-cert-manager

As next step, we create a ClusterIssuer that will be available cluster-wide using Let’s Encrypt as certificate authority:

The connection to Let’s Encrypt is using the productive API. If you would like to use the staging environment instead, change the server URL to https://acme-staging-v02.api.letsencrypt.org/directory
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: your@email.com (1)
    preferredChain: ''
    privateKeySecretRef:
      name: letsencrypt-account-key
    server: 'https://acme-v02.api.letsencrypt.org/directory'
    solvers:
      - dns01:
          route53:
            accessKeyID: XXXXXXXXXXXXXX (2)
            region: eu-central-1 (3)
            secretAccessKeySecretRef:
              key: secret-access-key
              name: prod-route53-credentials-secret (4)
        selector:
          dnsZones:
            - your-domain (5)
1Change your email address
2Use the AccessKeyId created above
3Using AWS you need to define a region
4The name of the secret created during the step before
5Your public domain, for example aws.ispworld.at.

Once created the ClusterIssuer should switch to the status "Ready"

oc describe clusterissuer letsencrypt-prod

Status:
...
  Conditions:
    Last Transition Time:  2023-02-16T13:54:49Z
    Message:               The ACME account was registered with the ACME server
    Observed Generation:   1
    Reason:                ACMEAccountRegistered
    Status:                True
    Type:                  Ready

OPTIONAL: When using private Domains or Firewalls

As you can see in one of the images above, I have two domains:

  1. aws.ispworld.at

  2. ocp.aws.ispworld.at

The first one is marked as public, that means everybody can resolve names. The second one is set to private and only define VPCs (in this case the cluster itself) can resolve hostnames.

In case of the following error:

E0216 15:27:29.513080 1 controller.go:163] cert-manager/challenges "msg"="re-queuing item due to error processing" "error"="failed to determine Route 53 hosted zone ID: zone not found in Route 53 for domain _acme-challenge.bookimport.apps.ocp.aws.ispworld.at." "key"="bookimport/bookimport-cert-jbmh6-2173685137-2399596362"

Add the following into ClusterManager

oc edit CertManager.operator.openshift.io/cluster

  unsupportedConfigOverrides:
    controller:
      args:
      - --v=2
      - --cluster-resource-namespace=$(POD_NAMESPACE)
      - --leader-election-namespace=kube-system
      - --dns01-recursive-nameservers-only
      - --dns01-recursive-nameservers=ns-362.awsdns-45.com:53,ns-930.awsdns-52.net:53 (1)
1List of nameserver the PUBLIC domain is hosted on.

The Operator will then try to resolve the names using the specified nameserver only.

Issue a new certificate

At this step, we can create a Certificate:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: bookimport-cert (1)
 namespace: bookimport (2)
spec:
 dnsNames:
   - bookimport.apps.ocp.aws.ispworld.at (3)
 issuerRef:
   kind: ClusterIssuer
   name: letsencrypt-prod (4)
 secretName: bookimport.apps.ocp.aws.ispworld.at-certificate (5)
1Name of the certificate objects
2Application namespace
3List of domain names
4Issuer that shall be used
5Name of the Secret that will be created and hold the certificate information

Create a Route

After a while the certificate will be Ready:

oc get certificate/bookimport-cert -n bookimport

NAME              READY   SECRET                                            AGE
bookimport-cert   True    bookimport.apps.ocp.aws.ispworld.at-certificate   87m

Now we can create a Route object to configure the IngressController. The important part here is the annotation, which will tell the Cert Utils Operator to automatically inject the certificate.

kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: bookimport-tls
  namespace: bookimport
  annotations:
    cert-utils-operator.redhat-cop.io/certs-from-secret: bookimport.apps.ocp.aws.ispworld.at-certificate (1)
spec:
  host: bookimport.apps.ocp.aws.ispworld.at (2)
  to:
    kind: Service
    name: book-import
    weight: 100
  tls:
    termination: edge
  port:
    targetPort: web
  wildcardPolicy: None
1Annotation that points to the Secret which stored the certificate. The values of this Secret will be automatically injected into this Route object.
2The hostname for our Route

As you can see, the Browser will show no warning when opening the URL.

BookimportTLS
Figure 4. Book Import using HTTPS

Cluster Default Certificates

During a cluster deployment, OpenShift will create self-signed certificates for its API and for the default IngressController *.apps.clusterdomain.

Usually, we want to change them as well. So why not use the Cert-Manager to issue the appropriate certificates?

Default IngressController

For the default IngressController I create a certificate request with 2 domain names: the wildcard and the base domain (just to be sure, actually the wildcard should be enough)

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: router-certificate
  namespace: openshift-ingress (1)
spec:
  dnsNames:
   - apps.ocp.aws.ispworld.at (2)
   - '*.apps.ocp.aws.ispworld.at'
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-prod
  secretName: router-certificate (3)
1The default IngressController runs in the namespace openshift-ingress
2List of domains
3Name of the Secret that will be created once the Certificate has been approved.

After a while, the certificate request should be Ready again. In the namespace openshift-ingress a Secret will be available with the name router-certificate

API

For the API URL we do the same. This time it is stored in the namepsace openshift-config

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: api-certificate
  namespace: openshift-config
spec:
  dnsNames:
   - api.ocp.aws.ispworld.at
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-prod
  secretName: api-certificate
It is possible to create one certificate with all required Domainnames. Just be sure that the Secret is available in the appropriate Namespace.

Patching API Server and IngressController

As a final step we need to patch the IngressController and the API server so they will use the correct Secrets with the officially signed certificates.

# IngressController
oc patch ingresscontroller default -n openshift-ingress-operator --type=merge --patch='{"spec": { "defaultCertificate": { "name": "router-certificate" }}}'

# API Server
oc patch apiserver cluster --type=merge -p '{"spec":{"servingCerts": {"namedCertificates": [{"names": ["api.ocp.aws.ispworld.at"], "servingCertificate": {"name": "api-certificate"}}]}}}' (1)
1Be sure to use the correct URL for the API

This will restart a bunch of services. Once everything is up and running again (your can watch using the command watch oc get co), the correct certificate will be shown in the browser:

UI
Figure 5. UI

or via curl:

curl -v https://api.ocp.aws.ispworld.at:6443

* Connected to api.ocp.aws.ispworld.at (13.52.208.31) port 6443
[...]
* Server certificate:
*  subject: CN=api.ocp.aws.ispworld.at
*  start date: Feb 16 15:11:36 2023 GMT
*  expire date: May 17 15:11:35 2023 GMT
*  subjectAltName: host "api.ocp.aws.ispworld.at" matched cert's "api.ocp.aws.ispworld.at"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.

Summary

Now with these steps, it is possible to issue new Certificates. Of course, there I many more options to configure a certificate. I encourage everybody to read the official documentation of the Cert Manager.

Especially, if you are interested in the whole certificate Certificate Lifecycle

See Also:

  • Introducing AdminNetworkPolicies –

    Classic Kubernetes/OpenShift offer a feature called NetworkPolicy that allows users to control the traffic to and from their assigned Namespace. NetworkPolicies are designed to give project owners or tenants the ability to protect their own namespace. Sometimes, however, I worked with customers where the cluster administrators or a dedicated (network) team need to enforce these policies. Since the NetworkPolicy API is namespace-scoped, it is not possible to enforce policies across namespaces. The only solution was to create custom (project) admin and edit roles, and remove the ability of creating, modifying or deleting NetworkPolicy objects. Technically, this is possible and easily done. But shifts the whole network security to cluster administrators. Luckily, this is where AdminNetworkPolicy (ANP) and BaselineAdminNetworkPolicy (BANP) comes into play.

  • Using Kustomize to post render a Helm Chart –

    Lately I came across several issues where a given Helm Chart must be modified after it has been rendered by Argo CD. Argo CD does a helm template to render a Chart. Sometimes, especially when you work with Subcharts or when a specific setting is not yet supported by the Chart, you need to modify it later …​ you need to post-render the Chart. In this very short article, I would like to demonstrate this on a real-live example I had to do. I would like to inject annotations to a Route objects, so that the certificate can be injected. This is done by the cert-utils operator. For the post-rendering the Argo CD repo pod will be extended with a sidecar container, that is watching for the repos and patches them if required.

  • Managing Certificates using GitOps approach –

    The article SSL Certificate Management for OpenShift on AWS explains how to use the Cert-Manager Operator to request and install a new SSL Certificate. This time, I would like to leverage the GitOps approach using the Helm Chart cert-manager I have prepared to deploy the Operator and order new Certificates. I will use an ACME Letsencrypt issuer with a DNS challenge. My domain is hosted at AWS Route 53. However, any other integration can be easily used.

  • Update Cluster Version using GitOps approach –

    During a GitOps journey at one point, the question arises, how to update a cluster? Nowadays it is very easy to update a cluster using CLI or WebUI, so why bother with GitOps in that case? The reason is simple: Using GitOps you can be sure that all clusters are updated to the correct, required version and the version of each cluster is also managed in Git. All you need is the channel you want to use and the desired cluster version. Optionally, you can define the exact image SHA. This might be required when you are operating in a restricted environment.

  • Multiple Sources for Applications in Argo CD –

    Argo CD or OpenShift GitOps uses Applications or ApplicationSets to define the relationship between a source (Git) and a cluster. Typically, this is a 1:1 link, which means one Application is using one source to compare the cluster status. This can be a limitation. For example, if you are working with Helm Charts and a Helm repository, you do not want to re-build (or re-release) the whole chart just because you made a small change in the values file that is packaged into the repository. You want to separate the configuration of the chart with the Helm package. The most common scenarios for multiple sources are (see: Argo CD documentation): Your organization wants to use an external/public Helm chart You want to override the Helm values with your own local values You don’t want to clone the Helm chart locally as well because that would lead to duplication and you would need to monitor it manually for upstream changes. This small article describes three different ways with a working example and tries to cover the advantages and disadvantages of each of them. They might be opinionated but some of them proved to be easier to use and manage.

Copyright © 2020 - 2024 Toni Schmidbauer & Thomas Jungbauer