Configure App-of-Apps

OpenShift GitOps

- Thomas Jungbauer Thomas Jungbauer ( Lastmod: 2024-05-08 ) - 10 min read

In the article Install GitOps to the cluster OpenShift GitOps is deployed using a shell script. This should be the very first installation and the only deployment that is done manually on a cluster. This procedure automatically installs the so-called App-of-Apps named Argo CD Resources Manager which is responsible for all further Argo CD Applications and ApplicationSets. No other configuration should be done manually if possible.

This article will demonstrate how to configure the App-of-Apps in an easy and declarative way, using ApplicationSet mainly.

Prerequisites

At this stage, the OpenShift cluster with the openshift-gitops operator and the App-of-Apps must be deployed. Your Argo CD should look somehow like this:

Initial Applications
Figure 1. Argo CD: Initial Applications

But how do all these Applications end up in Argo CD and how can you add additional ones?

Understanding the Argo CD Resources Manager

For any further references, I am using the GitHub repository OpenShift Clusterconfig GitOps

The Argo CD Resources Manager is, in fact, the App-of-Apps. Its configuration file can be found in the directory base/argocd-resources-manager. It is simply a values file and uses the Helm Chart helper-argocd to create additional Applications or ApplicationSets for Argo CD.

Analyzing the example values file

The example values file seems to be huge and confusing at first look. But it is quite easy to understand …​ trust me :)

Let’s walk through the file bit by bit:

Defining Header Variables

At the top of the file, some variables are defined as so-called anchors. These definitions might be used multiple times and are defined at the top to allow us to find and change them easily.

I define here the clusters and the information about the GitHub repository for example:

mgmt-cluster: &mgmtcluster https://kubernetes.default.svc (1)
mgmt-cluster-name: &mgmtclustername in-cluster (2)
production-cluster: &prodcluster https://api.ocp.aws.ispworld.at:6443
production-cluster-name: &prodclustername prod

repourl: &repourl 'https://github.com/tjungbauer/openshift-clusterconfig-gitops' (3)
repobranch: &branch main (4)
1Define the API URL for the cluster as configured in Argo CD. (The local cluster where Argo CD is running might be called kubernetes.default.svc)
2Define the short name of the cluster as configured in Argo CD. (The local cluster where Argo CD is running might be called in-cluster)
3Define the URL to the GitHub repository that is used in this file.
4Define the git branch that will be used.

Any additional anchor can be used to define values, that should show up at the very top and/or are used multiple times and you do not want to write them each time.

Whenever you see a value defined as *repourl for example, such an anchor is used.

Understanding Naming Conventions

Each Application or ApplicationSet must have a unique name inside Argo CD. Whenever Applications are generated by ApplicationSet a prefix with the name of the cluster is usually added.

In the values file multiple Applications or ApplicationSets can be defined. They are all bypassed to the Helm Chart which takes care of everything. The name that will be used for an ApplicationSet for example will be the (yaml) key of the definition.

For example, ApplicationSets are defined as:

applicationsets:
  mgmt-cluster:
   ...

  enable-etcd-encryption:
    ...

The keys, for example, mgmt-cluster or enable-etcd-encryption are used as names of the ApplicationSets. This way you do not need to take care of unique names, as YAML will already complain if some kwy is used twice.

Enable/Disable

Like with all of my Helm Charts, I added a switch to enable (or disable) a certain configuration. This way, you can easily remove Applications without actually deleting the specification.

The default value is false, so you actively need to set it to true
  mgmt-cluster:
    # Is the ApplicationSet enabled or not
    enabled: true

Supported Generators

The helm chart helper-argocd supports the following generators currently:

  • Matrix Generator

  • List Generator

  • Git Generator (for files)

  • Cluster Generator

Additional generators might be added in the future (ping me or create a pull request), but I found these the most useful ones.

But what is the difference between these generators from the configuration point of view? The different generators require different configurations and therefore provide different placeholders for variables. While the Git generator might use variables that are defined in a file that it finds {{environment}} the List (or Cluster) generator is using {{url}} to define the target cluster.

This might make the specification of an ApplicationSet quite complex …​ and that’s the whole reason for creating the helper-argocd

Example ApplicationSet - Matrix Generator

The first ApplicationSet I would like to show is probably the most important one. As described in GitOps Repository Structure I am using a folder structure like clusters/management-cluster/ and in this folder I am defining any configuration that is applicable for that specific cluster. If I want to add a new cluster, I simply create a new folder (and a new App-of-Apps configuration). With this, you will always see which settings a specific cluster has without much hassle.

The idea is to walk over this folder and automatically create a new Argo CD Application for any sub-folder that is found. This has the advantage, that whenever I want to create an additional configuration for a cluster, I simply add another sub-folder and the ApplicationSet will automatically create a new Argo CD application.

To achieve this the so-called Matrix Generator is used. This generator combines two (currently two are possible only) generators. In our case, it combines:

  1. git generator: to walk over the folder and get and sub-folder

  2. list generator: to define the target cluster

The snippet of the configuration will look like the following:

    # Definition of Matrix Generator. Only 2 generators are supported at the moment
    generatormatrix: (1)
          # Git: Walking through the specific folder and take whatever is there.
          - git: (2)
              directories:
                - path: clusters/management-cluster/*
                - path: clusters/management-cluster/waves
                  exclude: true
              repoURL: *repourl
              revision: *branch
          # List: simply define the targetCluster. The name of the cluster must be known by Argo CD
          - list: (3)
              elements:
                  # targetCluster is important, this will define on which cluster it will be rolled out.
                  # The cluster name must be known in Argo CD
                - targetCluster: *mgmtclustername
1Using matrix generator
2The first generator is Git: It will observe any changes in the folder clusters/management-cluster and will create a new Argo CD Application if a new sub-folder is found. However, it excludes the folder clusters/management-cluster/waves/
3The second generator is List: It simply defines the target cluster where the Application that is created by the ApplicationSet shall be deployed.

Now let us bring the whole example together:

  mgmt-cluster: (1)
    # Is the ApplicationSet enabled or not
    enabled: true (2)

    # Description - always useful
    description: "ApplicationSet that Deploys on Management Cluster Configuration (using Matrix Generator)" (3)

    # Any labels you would like to add to the Application. Good to filter it in the Argo CD UI.
    labels: (4)
      category: configuration
      env: mgmt-cluster

    # Using go text template. See: https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/GoTemplate/
    goTemplate: true (5)
    argocd_project: *mgmtclustername (6)

    # preserve all resources when the application get deleted. This is useful to keep that workload even if Argo CD is removed or severely changed.
    preserveResourcesOnDeletion: true (7)

    # Definition of Matrix Generator. Only 2 generators are supported at the moment
    generatormatrix: (8)
          # Git: Walking through the specific folder and take whatever is there.
          - git:
              directories:
                - path: clusters/management-cluster/*
                - path: clusters/management-cluster/waves
                  exclude: true
              repoURL: *repourl
              revision: *branch
          # List: simply define the targetCluster. The name of the cluster must be known by Argo CD
          - list:
              elements:
                  # targetCluster is important, this will define on which cluster it will be rolled out.
                  # The cluster name must be known in Argo CD
                - targetCluster: *mgmtclustername
    syncPolicy: (9)
      autosync_enabled: false
1Key of the ApplicationSet inside the yaml specification, that will be used as object name
2Is the ApplicationSet enabled or not (Default: false)
3A useful description
4Labels that can be used to filter
5Enable the usage of Go Template for this ApplicationSet
6The Argo CD project (not OpenShift project) the ApplicationSet belongs to
7Be sure that resources are not deleted when deleting the ApplicationSet. I found this quite useful …​ otherwise, all Applications the ApplicationSet created will be removed INCLUDING the resources they have created.
8The specification of the matrix generator
9Any kind of syncPolicy …​ in this case automatic synchronization of the Applications that are created is disabled.

Based on these settings the helper-argocd helm chart will render an ApplicationSet object automatically. As mentioned above it will be called mgmt-cluster and creates an Application for any sub-folder it finds in clusters/management-cluster.

Any new folder that is added will automatically create a new Application. You do not need to configure anything else.

The full objects will look like this:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: mgmt-cluster (1)
  namespace: openshift-gitops
  labels:
    app.kubernetes.io/instance: argocd-resources-manager
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: helper-argocd
    category: configuration
    env: mgmt-cluster
    helm.sh/chart: helper-argocd-2.0.28
spec:
  generators: (2)
    - matrix:
        generators:
          - git:
              directories:
                - path: clusters/management-cluster/*
                - exclude: true
                  path: clusters/management-cluster/waves
              repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops'
              revision: main
          - list:
              elements:
                - targetCluster: in-cluster
  goTemplate: true
  goTemplateOptions:
    - missingkey=error
  syncPolicy:
    preserveResourcesOnDeletion: true
  template:
    metadata:
      name: '{{ .targetCluster }}-{{ .path.basenameNormalized }}'  (3)
    spec:
      destination: (4)
        name: '{{ .targetCluster }}'
        namespace: default
      info:
        - name: Description
          value: ApplicationSet that Deploys on Management Cluster Configuration (using Matrix Generator)
      project: in-cluster
      source: (5)
        path: '{{ .path.path }}'
        repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops'
        targetRevision: main
1Name of the object == name of the key in the values file definition
2Configuration of the matrix generator
3Name of the Applications that this ApplicationSet will generate. In this case, it will concat the name of the target cluster and the name of the path.
4Target cluster
5Definition of the source for the Application

Example ApplicationSet - Git Generator

Now let us take a look at a second example using the git generator. The basic idea is quite similar and just a few minor changes must be made to our configuration.

In the following object, a git file generator is used to observe a specific folder and look for the file named values.yaml. For each file that it found an Application is created.

This example is also explained in my article at Project onboarding using GitOps and Helm
  # Tenant Onboarding (using Git Generator)
  onboarding-tenant-workload: (1)
    # Is the ApplicationSet enabled or not
    enabled: true

    # Description - always useful
    description: "Onboarding Workload to the cluster"

    # Any labels you would like to add to the Application. Good to filter it in the Argo CD UI.
    labels:
      catagory: tenant-onboarding

    # Path to the Git repository. The default URL and revision are defined as anchors at the beginning of the file, but could be overwritten here.
    path: clusters/all/project-onboarding (2)
    repourl: *repourl
    targetrevision: *branch

    # Using go text template. See: https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/GoTemplate/
    goTemplate: true

    # Helm configuration. A list of helm values files
    helm: (3)
      per_cluster_helm_values: false
      value_files:
        - '/{{ .path.path }}/values.yaml'
        - /tenants/values-global.yaml

    # Generator: currently list, git and cluster are possible.
    # either "generatorlist", "generatorgit" or "generatorclusters"
    # Define the repository that shall be checked for configuration file
    generatorgit: (4)
      - repourl: *repourl
        targetrevision: *branch
        files:
          - tenants/**/values.yaml (5)

    # preserve all resources when the application gets deleted. This is useful to keep that workload even if Argo CD is removed or severely changed.
    preserveResourcesOnDeletion: true
1Name of the ApplicationSet
2The repo URL and path which shall be read for the ApplicationSet
3A list of values files, that shall be used.
4The specification of the Git generator
5The path that shall be observed by this ApplicationSet. ** will return all files and directories recursively.

Again, the Helm chart helper-argocd will render an ApplicationSet for us.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: onboarding-tenant-workload
  namespace: openshift-gitops
  labels:
    app.kubernetes.io/instance: argocd-resources-manager
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: helper-argocd
    catagory: tenant-onboarding
    helm.sh/chart: helper-argocd-2.0.28
spec:
  generators:
    - git:
        files:
          - path: tenants/**/values.yaml
        repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops'
        revision: main
  goTemplate: true
  goTemplateOptions:
    - missingkey=error
  syncPolicy:
    preserveResourcesOnDeletion: true
  template:
    metadata:
      name: '{{ index .path.segments 1 | normalize }}-{{ .path.basename }}'
    spec:
      destination:
        name: '{{ .environment }}'
        namespace: default
      info:
        - name: Description
          value: Onboarding Workload to the cluster
      project: default
      source:
        helm:
          valueFiles:
            - '/{{ .path.path }}/values.yaml'
            - /tenants/values-global.yaml
        path: clusters/all/project-onboarding
        repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops'
        targetRevision: main

Example ApplicationSet - List Generator

At this point, we have seen two examples of ApplicationSets defined for helper-argocd. The List generator will be very easy to understand as it simply uses a list of target clusters to render the ApplicationSet.

The following snippet demonstrates that all you need to set are the clustername and clusterurl.

Please also verify the article Argo CD and Release Management with Helm Charts and ApplicationSets to understand the usage of the setting chart_version.
    # List of clusters
    # "clustername" (string): Is the name of the cluster a defined in Argo CD
    # "clusterurl" (string): Is the URL of the cluster API
    # "chart_version" (string, optional): Defines which chart version shall be deployed on each cluster.
    generatorlist:
      - clustername: *mgmtclustername
        clusterurl: *mgmtcluster

This is all the magic. The rendered ApplicationSet will look like:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: install-sonarqube
  namespace: openshift-gitops
  labels:
    app.kubernetes.io/instance: argocd-resources-manager
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: helper-argocd
    category: project
    helm.sh/chart: helper-argocd-2.0.28
spec:
  generators:
    - list:
        elements:
          - cluster: in-cluster
            url: 'https://kubernetes.default.svc'
  template:
    metadata:
      name: '{{ cluster }}-install-sonarqube'
    spec:
      destination:
        namespace: sonarqube
        server: '{{ url }}'
      info:
        - name: Description
          value: Install Sonarqube
      project: '{{ cluster }}'
      source:
        chart: sonarqube
        helm:
          releaseName: sonarqube
        repoURL: 'https://charts.stderr.at/'
        targetRevision: 1.0.1
The list generator can be used to deploy on ALL clusters too. Simply define generatorlist: []
The example above also demonstrates how to use a Helm chart instead of a git repository.

What about Applications?

The examples above show the usage of ApplicationSet and recently I migrated any specification of an Application to ApplicationSets as I believe this is easier to use, especially when the Chart is rendering it for you. However, it is still possible to define Applications as well.

The following example defines such an Application. The configuration differs compared to the ApplicationSet, however, the main idea stays the same:

applications: (1)
  node-labelling: (2)
    enabled: true
    description: "Deploy Node Labels"
    labels:
      category: configuration
    namespace:
      name: default
      create: false
    server: *mgmtcluster (3)
    project: default
    syncOptions: (4)
      - name: ServerSideApply
        value: true
      - name: Validate
        value: false
    source: (5)
      path: clusters/management-cluster/node-labels
      helm:
        valuesfiles:
          - name: values.yaml
      repourl: *repourl
      targetrevision: *branch
1Defining Applications
2Name of the Application. Be sure that it is unique since this time no prefix will be added
3The target cluster for this Application
4Different options for the synchronization
5The specification of the source. This contains the path, URL and branch of the repository and (in this case) the definition of a Helm values file.

Summary

I hope I was able to explain the usage of my chart helper-argocd and how I configure it. You can also verify the README to find additional possible settings and the example values.file that I use for all my clusters when I to a demo.

See Also:

  • Introducing AdminNetworkPolicies –

    Classic Kubernetes/OpenShift offer a feature called NetworkPolicy that allows users to control the traffic to and from their assigned Namespace. NetworkPolicies are designed to give project owners or tenants the ability to protect their own namespace. Sometimes, however, I worked with customers where the cluster administrators or a dedicated (network) team need to enforce these policies. Since the NetworkPolicy API is namespace-scoped, it is not possible to enforce policies across namespaces. The only solution was to create custom (project) admin and edit roles, and remove the ability of creating, modifying or deleting NetworkPolicy objects. Technically, this is possible and easily done. But shifts the whole network security to cluster administrators. Luckily, this is where AdminNetworkPolicy (ANP) and BaselineAdminNetworkPolicy (BANP) comes into play.

  • Using Kustomize to post render a Helm Chart –

    Lately I came across several issues where a given Helm Chart must be modified after it has been rendered by Argo CD. Argo CD does a helm template to render a Chart. Sometimes, especially when you work with Subcharts or when a specific setting is not yet supported by the Chart, you need to modify it later …​ you need to post-render the Chart. In this very short article, I would like to demonstrate this on a real-live example I had to do. I would like to inject annotations to a Route objects, so that the certificate can be injected. This is done by the cert-utils operator. For the post-rendering the Argo CD repo pod will be extended with a sidecar container, that is watching for the repos and patches them if required.

  • Managing Certificates using GitOps approach –

    The article SSL Certificate Management for OpenShift on AWS explains how to use the Cert-Manager Operator to request and install a new SSL Certificate. This time, I would like to leverage the GitOps approach using the Helm Chart cert-manager I have prepared to deploy the Operator and order new Certificates. I will use an ACME Letsencrypt issuer with a DNS challenge. My domain is hosted at AWS Route 53. However, any other integration can be easily used.

  • Update Cluster Version using GitOps approach –

    During a GitOps journey at one point, the question arises, how to update a cluster? Nowadays it is very easy to update a cluster using CLI or WebUI, so why bother with GitOps in that case? The reason is simple: Using GitOps you can be sure that all clusters are updated to the correct, required version and the version of each cluster is also managed in Git. All you need is the channel you want to use and the desired cluster version. Optionally, you can define the exact image SHA. This might be required when you are operating in a restricted environment.

  • Multiple Sources for Applications in Argo CD –

    Argo CD or OpenShift GitOps uses Applications or ApplicationSets to define the relationship between a source (Git) and a cluster. Typically, this is a 1:1 link, which means one Application is using one source to compare the cluster status. This can be a limitation. For example, if you are working with Helm Charts and a Helm repository, you do not want to re-build (or re-release) the whole chart just because you made a small change in the values file that is packaged into the repository. You want to separate the configuration of the chart with the Helm package. The most common scenarios for multiple sources are (see: Argo CD documentation): Your organization wants to use an external/public Helm chart You want to override the Helm values with your own local values You don’t want to clone the Helm chart locally as well because that would lead to duplication and you would need to monitor it manually for upstream changes. This small article describes three different ways with a working example and tries to cover the advantages and disadvantages of each of them. They might be opinionated but some of them proved to be easier to use and manage.

Copyright © 2020 - 2024 Toni Schmidbauer & Thomas Jungbauer