Configure App-of-Apps

April 2, 2024 - By: Thomas Jungbauer ( Lastmod: 2024-04-25 ) - 10 min read

In the article Install GitOps to the cluster OpenShift GitOps is deployed using a shell script. This should be the very first installation and the only deployment that is done manually on a cluster. This procedure automatically installs the so-called App-of-Apps named Argo CD Resources Manager which is responsible for all further Argo CD Applications and ApplicationSets. No other configuration should be done manually if possible.

This article will demonstrate how to configure the App-of-Apps in an easy and declarative way, using ApplicationSet mainly.

Prerequisites

At this stage, the OpenShift cluster with the openshift-gitops operator and the App-of-Apps must be deployed. Your Argo CD should look somehow like this:

Figure 1. Argo CD: Initial Applications

But how do all these Applications end up in Argo CD and how can you add additional ones?

Understanding the Argo CD Resources Manager

For any further references, I am using the GitHub repository OpenShift Clusterconfig GitOps

The Argo CD Resources Manager is, in fact, the App-of-Apps. Its configuration file can be found in the directory base/argocd-resources-manager. It is simply a values file and uses the Helm Chart helper-argocd to create additional Applications or ApplicationSets for Argo CD.

Analyzing the example values file

The example values file seems to be huge and confusing at first look. But it is quite easy to understand … trust me :)

Let’s walk through the file bit by bit:

Defining Header Variables

At the top of the file, some variables are defined as so-called anchors. These definitions might be used multiple times and are defined at the top to allow us to find and change them easily.

I define here the clusters and the information about the GitHub repository for example:

mgmt-cluster: &mgmtcluster https://kubernetes.default.svc (1)
mgmt-cluster-name: &mgmtclustername in-cluster (2)
production-cluster: &prodcluster https://api.ocp.aws.ispworld.at:6443
production-cluster-name: &prodclustername prod

repourl: &repourl 'https://github.com/tjungbauer/openshift-clusterconfig-gitops' (3)
repobranch: &branch main (4)

1	Define the API URL for the cluster as configured in Argo CD. (The local cluster where Argo CD is running might be called kubernetes.default.svc)
2	Define the short name of the cluster as configured in Argo CD. (The local cluster where Argo CD is running might be called in-cluster)
3	Define the URL to the GitHub repository that is used in this file.
4	Define the git branch that will be used.

Any additional anchor can be used to define values, that should show up at the very top and/or are used multiple times and you do not want to write them each time.

Whenever you see a value defined as *repourl for example, such an anchor is used.

Understanding Naming Conventions

Each Application or ApplicationSet must have a unique name inside Argo CD. Whenever Applications are generated by ApplicationSet a prefix with the name of the cluster is usually added.

In the values file multiple Applications or ApplicationSets can be defined. They are all bypassed to the Helm Chart which takes care of everything. The name that will be used for an ApplicationSet for example will be the (yaml) key of the definition.

For example, ApplicationSets are defined as:

applicationsets:
  mgmt-cluster:
   ...

  enable-etcd-encryption:
    ...

The keys, for example, mgmt-cluster or enable-etcd-encryption are used as names of the ApplicationSets. This way you do not need to take care of unique names, as YAML will already complain if some kwy is used twice.

Enable/Disable

Like with all of my Helm Charts, I added a switch to enable (or disable) a certain configuration. This way, you can easily remove Applications without actually deleting the specification.

The default value is false, so you actively need to set it to true

  mgmt-cluster:
    # Is the ApplicationSet enabled or not
    enabled: true

Supported Generators

The helm chart helper-argocd supports the following generators currently:

Matrix Generator
List Generator
Git Generator (for files)
Cluster Generator

Additional generators might be added in the future (ping me or create a pull request), but I found these the most useful ones.

But what is the difference between these generators from the configuration point of view? The different generators require different configurations and therefore provide different placeholders for variables. While the Git generator might use variables that are defined in a file that it finds {{environment}} the List (or Cluster) generator is using {{url}} to define the target cluster.

This might make the specification of an ApplicationSet quite complex … and that’s the whole reason for creating the helper-argocd

Example ApplicationSet - Matrix Generator

The first ApplicationSet I would like to show is probably the most important one. As described in GitOps Repository Structure I am using a folder structure like clusters/management-cluster/ and in this folder I am defining any configuration that is applicable for that specific cluster. If I want to add a new cluster, I simply create a new folder (and a new App-of-Apps configuration). With this, you will always see which settings a specific cluster has without much hassle.

The idea is to walk over this folder and automatically create a new Argo CD Application for any sub-folder that is found. This has the advantage, that whenever I want to create an additional configuration for a cluster, I simply add another sub-folder and the ApplicationSet will automatically create a new Argo CD application.

To achieve this the so-called Matrix Generator is used. This generator combines two (currently two are possible only) generators. In our case, it combines:

git generator: to walk over the folder and get and sub-folder
list generator: to define the target cluster

The snippet of the configuration will look like the following:

    # Definition of Matrix Generator. Only 2 generators are supported at the moment
    generatormatrix: (1)
          # Git: Walking through the specific folder and take whatever is there.
          - git: (2)
              directories:
                - path: clusters/management-cluster/*
                - path: clusters/management-cluster/waves
                  exclude: true
              repoURL: *repourl
              revision: *branch
          # List: simply define the targetCluster. The name of the cluster must be known by Argo CD
          - list: (3)
              elements:
                  # targetCluster is important, this will define on which cluster it will be rolled out.
                  # The cluster name must be known in Argo CD
                - targetCluster: *mgmtclustername

1	Using matrix generator
2	The first generator is Git: It will observe any changes in the folder clusters/management-cluster and will create a new Argo CD Application if a new sub-folder is found. However, it excludes the folder clusters/management-cluster/waves/
3	The second generator is List: It simply defines the target cluster where the Application that is created by the ApplicationSet shall be deployed.

Now let us bring the whole example together:

  mgmt-cluster: (1)
    # Is the ApplicationSet enabled or not
    enabled: true (2)

    # Description - always useful
    description: "ApplicationSet that Deploys on Management Cluster Configuration (using Matrix Generator)" (3)

    # Any labels you would like to add to the Application. Good to filter it in the Argo CD UI.
    labels: (4)
      category: configuration
      env: mgmt-cluster

    # Using go text template. See: https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/GoTemplate/
    goTemplate: true (5)
    argocd_project: *mgmtclustername (6)

    # preserve all resources when the application get deleted. This is useful to keep that workload even if Argo CD is removed or severely changed.
    preserveResourcesOnDeletion: true (7)

    # Definition of Matrix Generator. Only 2 generators are supported at the moment
    generatormatrix: (8)
          # Git: Walking through the specific folder and take whatever is there.
          - git:
              directories:
                - path: clusters/management-cluster/*
                - path: clusters/management-cluster/waves
                  exclude: true
              repoURL: *repourl
              revision: *branch
          # List: simply define the targetCluster. The name of the cluster must be known by Argo CD
          - list:
              elements:
                  # targetCluster is important, this will define on which cluster it will be rolled out.
                  # The cluster name must be known in Argo CD
                - targetCluster: *mgmtclustername
    syncPolicy: (9)
      autosync_enabled: false

1	Key of the ApplicationSet inside the yaml specification, that will be used as object name
2	Is the ApplicationSet enabled or not (Default: false)
3	A useful description
4	Labels that can be used to filter
5	Enable the usage of Go Template for this ApplicationSet
6	The Argo CD project (not OpenShift project) the ApplicationSet belongs to
7	Be sure that resources are not deleted when deleting the ApplicationSet. I found this quite useful … otherwise, all Applications the ApplicationSet created will be removed INCLUDING the resources they have created.
8	The specification of the matrix generator
9	Any kind of syncPolicy … in this case automatic synchronization of the Applications that are created is disabled.

Based on these settings the helper-argocd helm chart will render an ApplicationSet object automatically. As mentioned above it will be called mgmt-cluster and creates an Application for any sub-folder it finds in clusters/management-cluster.

Any new folder that is added will automatically create a new Application. You do not need to configure anything else.

The full objects will look like this:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: mgmt-cluster (1)
  namespace: openshift-gitops
  labels:
    app.kubernetes.io/instance: argocd-resources-manager
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: helper-argocd
    category: configuration
    env: mgmt-cluster
    helm.sh/chart: helper-argocd-2.0.28
spec:
  generators: (2)
    - matrix:
        generators:
          - git:
              directories:
                - path: clusters/management-cluster/*
                - exclude: true
                  path: clusters/management-cluster/waves
              repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops'
              revision: main
          - list:
              elements:
                - targetCluster: in-cluster
  goTemplate: true
  goTemplateOptions:
    - missingkey=error
  syncPolicy:
    preserveResourcesOnDeletion: true
  template:
    metadata:
      name: '{{ .targetCluster }}-{{ .path.basenameNormalized }}'  (3)
    spec:
      destination: (4)
        name: '{{ .targetCluster }}'
        namespace: default
      info:
        - name: Description
          value: ApplicationSet that Deploys on Management Cluster Configuration (using Matrix Generator)
      project: in-cluster
      source: (5)
        path: '{{ .path.path }}'
        repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops'
        targetRevision: main

1	Name of the object == name of the key in the values file definition
2	Configuration of the matrix generator
3	Name of the Applications that this ApplicationSet will generate. In this case, it will concat the name of the target cluster and the name of the path.
4	Target cluster
5	Definition of the source for the Application

Example ApplicationSet - Git Generator

Now let us take a look at a second example using the git generator. The basic idea is quite similar and just a few minor changes must be made to our configuration.

In the following object, a git file generator is used to observe a specific folder and look for the file named values.yaml. For each file that it found an Application is created.

This example is also explained in my article at Project onboarding using GitOps and Helm

  # Tenant Onboarding (using Git Generator)
  onboarding-tenant-workload: (1)
    # Is the ApplicationSet enabled or not
    enabled: true

    # Description - always useful
    description: "Onboarding Workload to the cluster"

    # Any labels you would like to add to the Application. Good to filter it in the Argo CD UI.
    labels:
      catagory: tenant-onboarding

    # Path to the Git repository. The default URL and revision are defined as anchors at the beginning of the file, but could be overwritten here.
    path: clusters/all/project-onboarding (2)
    repourl: *repourl
    targetrevision: *branch

    # Using go text template. See: https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/GoTemplate/
    goTemplate: true

    # Helm configuration. A list of helm values files
    helm: (3)
      per_cluster_helm_values: false
      value_files:
        - '/{{ .path.path }}/values.yaml'
        - /tenants/values-global.yaml

    # Generator: currently list, git and cluster are possible.
    # either "generatorlist", "generatorgit" or "generatorclusters"
    # Define the repository that shall be checked for configuration file
    generatorgit: (4)
      - repourl: *repourl
        targetrevision: *branch
        files:
          - tenants/**/values.yaml (5)

    # preserve all resources when the application gets deleted. This is useful to keep that workload even if Argo CD is removed or severely changed.
    preserveResourcesOnDeletion: true

1	Name of the ApplicationSet
2	The repo URL and path which shall be read for the ApplicationSet
3	A list of values files, that shall be used.
4	The specification of the Git generator
5	The path that shall be observed by this ApplicationSet. ** will return all files and directories recursively.

Again, the Helm chart helper-argocd will render an ApplicationSet for us.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: onboarding-tenant-workload
  namespace: openshift-gitops
  labels:
    app.kubernetes.io/instance: argocd-resources-manager
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: helper-argocd
    catagory: tenant-onboarding
    helm.sh/chart: helper-argocd-2.0.28
spec:
  generators:
    - git:
        files:
          - path: tenants/**/values.yaml
        repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops'
        revision: main
  goTemplate: true
  goTemplateOptions:
    - missingkey=error
  syncPolicy:
    preserveResourcesOnDeletion: true
  template:
    metadata:
      name: '{{ index .path.segments 1 | normalize }}-{{ .path.basename }}'
    spec:
      destination:
        name: '{{ .environment }}'
        namespace: default
      info:
        - name: Description
          value: Onboarding Workload to the cluster
      project: default
      source:
        helm:
          valueFiles:
            - '/{{ .path.path }}/values.yaml'
            - /tenants/values-global.yaml
        path: clusters/all/project-onboarding
        repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops'
        targetRevision: main

Example ApplicationSet - List Generator

At this point, we have seen two examples of ApplicationSets defined for helper-argocd. The List generator will be very easy to understand as it simply uses a list of target clusters to render the ApplicationSet.

The following snippet demonstrates that all you need to set are the clustername and clusterurl.

Please also verify the article Argo CD and Release Management with Helm Charts and ApplicationSets to understand the usage of the setting chart_version.

    # List of clusters
    # "clustername" (string): Is the name of the cluster a defined in Argo CD
    # "clusterurl" (string): Is the URL of the cluster API
    # "chart_version" (string, optional): Defines which chart version shall be deployed on each cluster.
    generatorlist:
      - clustername: *mgmtclustername
        clusterurl: *mgmtcluster

This is all the magic. The rendered ApplicationSet will look like:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: install-sonarqube
  namespace: openshift-gitops
  labels:
    app.kubernetes.io/instance: argocd-resources-manager
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: helper-argocd
    category: project
    helm.sh/chart: helper-argocd-2.0.28
spec:
  generators:
    - list:
        elements:
          - cluster: in-cluster
            url: 'https://kubernetes.default.svc'
  template:
    metadata:
      name: '{{ cluster }}-install-sonarqube'
    spec:
      destination:
        namespace: sonarqube
        server: '{{ url }}'
      info:
        - name: Description
          value: Install Sonarqube
      project: '{{ cluster }}'
      source:
        chart: sonarqube
        helm:
          releaseName: sonarqube
        repoURL: 'https://charts.stderr.at/'
        targetRevision: 1.0.1

The list generator can be used to deploy on ALL clusters too. Simply define generatorlist: []

The example above also demonstrates how to use a Helm chart instead of a git repository.

What about Applications?

The examples above show the usage of ApplicationSet and recently I migrated any specification of an Application to ApplicationSets as I believe this is easier to use, especially when the Chart is rendering it for you. However, it is still possible to define Applications as well.

The following example defines such an Application. The configuration differs compared to the ApplicationSet, however, the main idea stays the same:

applications: (1)
  node-labelling: (2)
    enabled: true
    description: "Deploy Node Labels"
    labels:
      category: configuration
    namespace:
      name: default
      create: false
    server: *mgmtcluster (3)
    project: default
    syncOptions: (4)
      - name: ServerSideApply
        value: true
      - name: Validate
        value: false
    source: (5)
      path: clusters/management-cluster/node-labels
      helm:
        valuesfiles:
          - name: values.yaml
      repourl: *repourl
      targetrevision: *branch

1	Defining Applications
2	Name of the Application. Be sure that it is unique since this time no prefix will be added
3	The target cluster for this Application
4	Different options for the synchronization
5	The specification of the source. This contains the path, URL and branch of the repository and (in this case) the definition of a Helm values file.

Summary

I hope I was able to explain the usage of my chart helper-argocd and how I configure it. You can also verify the README to find additional possible settings and the example values.file that I use for all my clusters when I to a demo.

Built with Hugo Learn Theme and Hugo