Operator installation with Argo CD

OpenShift

- Thomas Jungbauer Thomas Jungbauer ( Lastmod: 2024-05-08 ) - 6 min read

GitOps for application deployment and cluster configuration is a must-have I am trying to convince every customer to follow from the very beginning when starting the Kubernetes journey. For me, as more on the infrastructure side of things, I am more focused on the configuration of an environment. Meaning, configuring a cluster, installing an operator etc.

In this article, I would like to share how I deal with cluster configuration when certain Kubernetes objects are dependent on each other and how to use Kubernetes but also Argo CD features to resolve these dependencies.

This article assumes that you have the openshift-gitops Operator, which provides Argo CD, already installed, and configured. If you are new to GitOps check out this article: Argo CD

TL;DR

If you want to jump directly to the technical fun part, go here: Let’s start.

The Idea

Everything should be seen as a code. Everything should be possible to be deployed in a repeatable way. With a GitOps approach, everything is stored naturally in Git and from there, a GitOps agent validates and synchronizes changes to one or more clusters.

When it comes to OpenShift, Red Hat supports Argo CD using the Operator openshift-gitops. This gives you everything you need to deploy an Argo CD instance. The only thing you need to take care of is a Git repository, no matter if it is GitHub, Gitlab, Bitbucket etc.

The Problem

Sometimes Kubernetes objects depend on each other. This is especially true when you would like to install and configure Operators, where the configuration, based on a Customer Resource Definition (CRD), can only happen after the Operator has been installed and is ready.

Why is that? Well, when you want to deploy an Operator, you will store a “Subscription object” in Git. Argo CD will take this object and applies it to the cluster. However, for an Operator, the creation of the Subscription object is just the first step. A lot of other steps are required until the Operator gets ready. Unfortunately, Argo CD cannot verify if the installation is successful. All it sees is that the Subscription object has been created and then it immediately tries to deploy the CRD. The CRD which is not yet available on the system because the Operator is still installing it.

Even if you use Argo CD features like Sync waves it would not wait until the Operator is successfully installed because for Argo CD the “success” is the creation of the Subscription object.

Subsequently, the Argo CD synchronisation process will fail. You could now try to automatically “Retry” the sync or use multiple Argo CD applications that you execute one after each other, but I was not fully happy with that and tried a different approach.

My Solution

Let’s say I would like to deploy and configure the Compliance Operator. The steps would be:

  1. Install the Operator.

  2. Wait until the Operator is ready.

  3. Configure Operator specific CRDs.

This “Wait until the Operator is ready” is the tricky party for Argo CD. What I have done is the following:

  1. Install the Operator, this is the first step and is done during Sync Wave 0.

  2. Create a Kubernetes Job that verifies the status of the Operator. This Job additionally requires a ServiceAccount and a role with a binding. They are configured during Sync Wave is 1. Moreover, I use a Hook (another Argo CD feature) with the deletion policy “HookSucceeded”. This makes sure that the Job, ServiceAccount, Role and RoleBinding are removed after the status has been verified. The verification is successful as soon as the Operator status says “Succeeded”. In fact, all the Job does is to execute some oc commands. For example,

    oc get clusterserviceversion openshift-gitops-operator.v1.8.0 -n openshift-gitops -o jsonpath={.status.phase}

Succeeded

  3. Finally, during the next Sync Wave (2+) the CRD can be deployed. In this case, I deploy the object ScanSettingBinding.

In Argo CD everything is correctly synchronized, and the Operator and its configuration is in place.

If you are new to the compliance operator, I recommend the following article: Compliance Operator

I use this approach for every Operator that I would like to install and configure at the same time. For example, I do the same for Advanced Cluster Security or Advanced Cluster Management where I use the Job to verify if everything is ready before I let Argo CD continue.

More information about Sync Waves and Hooks can be found in the official Argo CD documentation: Sync Phases and Waves

Let’s see this in Action

Prerequisites

  1. OpenShift cluster 4.x

  2. openshift-gitops is installed and ready to be used.

  3. Access to GitHub (or to your own Repository)

I will be using my Helm Chart repository at https://charts.stderr.at/ and from there the charts:

  1. compliance-operator-full-stack

    1. helper-operator (sub chart): Responsible to install the Operators.

    2. helper-status-checker (sub chart): Responsible to check the status of the Operator.

Why do I use Helm charts? There is no specific reason for that. I started with Helm for the cluster configuration and now it has evolved with a separate Chart repository and sub-charts and so on.

Argo CD Application

In Argo CD I have the following Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: in-cluster-install-compliance-scans
  namespace: openshift-gitops
spec:
  destination:
    namespace: default
    server: 'https://kubernetes.default.svc' (1)
  info:
    - name: Description
      value: Deploy and configure the Compliance Scan Operator
  project: in-cluster
  source:
    path: charts/compliance-operator-full-stack (2)
    repoURL: 'https://github.com/tjungbauer/helm-charts'
    targetRevision: main
1Installing on the local cluster where Argo CD is installed.
2Git configuration, including path and revision.
Actually, this Application is created out of an ApplicationSet, but I did not want to make it too complex :)

The Application would like to synchronize the objects:

  1. Subscription

  2. OperatorGroup

  3. Namespace (openshift-compliance)

  4. ScanSettingBinding

Installing Compliance Operator
Figure 1. Argo CD: Installing Compliance Operator
Where are the objects we need for the Job? Since they are only available during the Sync-Hook they will not show up here. In fact, they will only show up during the time they are alive and will disappear again after the status of the operator has been verified.

Helm Chart Configuration

The Helm Chart gets its configuration from a values file. You can verify the whole file on GitHub.

The important pieces here are that some variables are handed over to the appropriate Sub Charts.

Operator Configuration

This part is handed over to the Chart “helper-operator”.

helper-operator:
  operators:
    compliance-operator:
      enabled: true
      syncwave: '0'
      namespace:
        name: openshift-compliance
        create: true
      subscription:
        channel: release-0.1
        approval: Automatic
        operatorName: compliance-operator
        source: redhat-operators
        sourceNamespace: openshift-marketplace
      operatorgroup:
        create: true
        notownnamespace: true

It is executed during Sync Wave 0 and defines if a Namespace (openshift-compliance) shall be created (true) and the specification of the Operator which you need to know upfront:

  1. channel: Defines which channel shall be used. Some operators offer different channels.

  2. approval: Either Automatic or Manual … defines if the Operator shall be updated automatically or requires an approval.

  3. operatorName: the actual name of the Operator (compliance-operator)

  4. source: Where does this Operator come from (redhat-operator)

  5. sourceNamespace: In this case openshift-marketplace

You can fetch these values by looking at the Packagemanifest:

oc get packagemanifest compliance-operator -o yaml

Status Checker Configuration

This part is handed over to the Sub-Chart "helper-status-checker"". The main values here are the operatorName and the namespace where the Operator is installed.

What is not visible here is the Sync Wave, which is per default set to 1 inside the Helm Chart. If you need to overwrite it, it can be configured in this section as well.

helper-status-checker:
  enabled: true (1)

  # use the value of the currentCSV (packagemanifest) but WITHOUT the version !!
  operatorName: compliance-operator (2)

  # where operator is installed
  namespace:
    name: openshift-compliance (3)

  serviceAccount:
    create: true
    name: "sa-compliance" (4)
1Is the status checker enabled or is it not.
2The name of the operator as it is reported by the value currentCSV inside the packageManifest
3The namespace where the Operator has been installed.
4The name of the ServiceAccount that is created temporarily.
The operatorName is sometimes different than the Operator name required for helper-operator chart. Here it seems the value of the currentCSV must be used but without the version number. (The Job will look up the version itself)

Operator CRD configuration

The final section of the values file manages the configuration for the Operator itself. This section does not use a Sub Chart. Instead, the variables are used in the Main-Chart. In this example, the ScanSettingBinding will be configured during Sync Wave 3, which is all we need to basic functionality.

compliance:
  scansettingbinding:
    enabled: true
    syncwave: '3' (1)
    profiles: (2)
      - name: ocp4-cis-node
      - name: ocp4-cis
    scansetting: default
1Define the Sync Wave. This value must be higher than the Sync Wave of the helper-status-checker
2ScanSettingBinding configuration. Two profiles are used in this example.

Synchronizing Argo CD

  1. Basic Application in Argo CD before it is synced:

    argocd operator installation
    Figure 2. Argo CD: Application

  2. Sync Wave 0: Synchronization has started. Namespace and Subscription are deployed.

    argocd starting operator installation
    Figure 3. Argo CD: Synchronization is started (Sync Wave 0)

  3. Sync Wave 1: Status Checker Job has started and tries to verify the Operator.

    argocd starting job
    Figure 4. Argo CD: Status Checker Job started (Sync Wave 1)

  4. The Log output of the Operator. You can see that the status switches from Pending to Installing to Succeeded.

    argocd status checker log
    Figure 5. Argo CD: Log of the Status Checker Pod

  5. After Sync Wave 3, the whole Application has been synchronized and the Checker Job has been removed.

    argocd operator installed
    Figure 6. Argo CD: Compliance Operator is fully deployed

See Also:

  • Introducing AdminNetworkPolicies –

    Classic Kubernetes/OpenShift offer a feature called NetworkPolicy that allows users to control the traffic to and from their assigned Namespace. NetworkPolicies are designed to give project owners or tenants the ability to protect their own namespace. Sometimes, however, I worked with customers where the cluster administrators or a dedicated (network) team need to enforce these policies. Since the NetworkPolicy API is namespace-scoped, it is not possible to enforce policies across namespaces. The only solution was to create custom (project) admin and edit roles, and remove the ability of creating, modifying or deleting NetworkPolicy objects. Technically, this is possible and easily done. But shifts the whole network security to cluster administrators. Luckily, this is where AdminNetworkPolicy (ANP) and BaselineAdminNetworkPolicy (BANP) comes into play.

  • Using Kustomize to post render a Helm Chart –

    Lately I came across several issues where a given Helm Chart must be modified after it has been rendered by Argo CD. Argo CD does a helm template to render a Chart. Sometimes, especially when you work with Subcharts or when a specific setting is not yet supported by the Chart, you need to modify it later …​ you need to post-render the Chart. In this very short article, I would like to demonstrate this on a real-live example I had to do. I would like to inject annotations to a Route objects, so that the certificate can be injected. This is done by the cert-utils operator. For the post-rendering the Argo CD repo pod will be extended with a sidecar container, that is watching for the repos and patches them if required.

  • Managing Certificates using GitOps approach –

    The article SSL Certificate Management for OpenShift on AWS explains how to use the Cert-Manager Operator to request and install a new SSL Certificate. This time, I would like to leverage the GitOps approach using the Helm Chart cert-manager I have prepared to deploy the Operator and order new Certificates. I will use an ACME Letsencrypt issuer with a DNS challenge. My domain is hosted at AWS Route 53. However, any other integration can be easily used.

  • Update Cluster Version using GitOps approach –

    During a GitOps journey at one point, the question arises, how to update a cluster? Nowadays it is very easy to update a cluster using CLI or WebUI, so why bother with GitOps in that case? The reason is simple: Using GitOps you can be sure that all clusters are updated to the correct, required version and the version of each cluster is also managed in Git. All you need is the channel you want to use and the desired cluster version. Optionally, you can define the exact image SHA. This might be required when you are operating in a restricted environment.

  • Multiple Sources for Applications in Argo CD –

    Argo CD or OpenShift GitOps uses Applications or ApplicationSets to define the relationship between a source (Git) and a cluster. Typically, this is a 1:1 link, which means one Application is using one source to compare the cluster status. This can be a limitation. For example, if you are working with Helm Charts and a Helm repository, you do not want to re-build (or re-release) the whole chart just because you made a small change in the values file that is packaged into the repository. You want to separate the configuration of the chart with the Helm package. The most common scenarios for multiple sources are (see: Argo CD documentation): Your organization wants to use an external/public Helm chart You want to override the Helm values with your own local values You don’t want to clone the Helm chart locally as well because that would lead to duplication and you would need to monitor it manually for upstream changes. This small article describes three different ways with a working example and tries to cover the advantages and disadvantages of each of them. They might be opinionated but some of them proved to be easier to use and manage.

Copyright © 2020 - 2024 Toni Schmidbauer & Thomas Jungbauer