In Ep.19, the default instance openshift-gitops became the principal of the Argo CD Agent. Its application-controller is disabled, so it cannot sync anything to the hub itself, and the manifests of agent Applications are rendered on the managed clusters anyway. The policies, however, must be created on the hub.

This article therefore uses a second, small Argo CD instance on the hub that only takes care of the hub configuration: policy-gitops in the namespace of the same name. If your default instance still has its application-controller (push or pull model), you can use it instead. Replace policy-gitops with openshift-gitops in the examples below.

The application-controller of the instance cannot create policies and Placements. Create a ClusterRole and bind it to the controller’s service account:

The ClusterRole in the ACM documentation also contains placementrules. I left them out on purpose: PlacementRules are deprecated.

Creating the policy YAML by hand is tough and error-prone. Every manifest has to be wrapped in a ConfigurationPolicy, which is wrapped in a Policy, which needs a Placement and a PlacementBinding. The PolicyGenerator solves this. It is a Kustomize generator plugin that takes plain manifests and wraps them for you.

The plugin is not part of the OpenShift GitOps images. An init container copies it from an ACM image into the repo-server, and Kustomize must be allowed to run alpha plugins.

The setup is described in the ACM documentation: ACM 2.17: Accessing the Policy Generator from OpenShift GitOps.

If your policy folders contain Helm charts that should be rendered before the policies are generated, add the environment variable POLICY_GEN_ENABLE_HELM with the value "true" to spec.repo.env.

By default, Argo CD considers a policy healthy as soon as it exists. That is not very useful: you want to see in Argo CD whether the policy is compliant. The policy collection of the Open Cluster Management project contains a policy that adds the matching health checks (resourceHealthChecks) to the ArgoCD resource: argocd-policy-healthchecks.yaml. The policy adds the health checks to every ArgoCD instance it finds on the cluster, so policy-gitops is covered as well. It is shipped with remediationAction: inform: set it to enforce, otherwise it only reports the missing health checks. After that, Argo CD shows a non-compliant policy as Degraded.

The files in input/ are plain Kubernetes manifests, exactly as you would apply them with oc apply. The Operator is installed in the namespace openshift-operators, which already has an OperatorGroup for all namespaces, so neither a namespace nor an OperatorGroup is needed for it:

The control plane consists of two cluster-scoped resources, IstioCNI and Istio, each pointing to its own namespace (istio-cni and istio-system):

And the kubeadmin secret that must not exist:

The kustomization.yaml only references the generator:

Render the folder locally before you push. If it does not render on your machine, it will not render in Argo CD either. With the PolicyGenerator binary in your Kustomize plugin path:

The output contains a Policy, a Placement and a PlacementBinding per policy.

The file policy-generator.yaml is the heart of the setup. It does not contain any policy itself, it only describes which policies the generator shall create from the manifests in input/. It has two parts:

For every entry in policies, the generator creates a Policy, a Placement and a PlacementBinding. The manifests are wrapped into a ConfigurationPolicy inside the Policy: by default all manifests of a policy into one, with consolidateManifests: false one per manifest. Our two policies use this as follows: service-mesh overrides the defaults to enforce and to apply its manifests in order, while no-kubeadmin keeps the defaults and only changes the compliance type.

A Placement only sees the clusters of cluster sets that are bound to its own namespace. The bindings from Ep.19 live in openshift-gitops, so the Placements in acm-policies would find no cluster at all. ACM then shows the warning "ClusterSets failed to load. Verify that there is at least one ClusterSet bound to your selected namespace."

Bind the cluster set production to acm-policies, and global as well if you use the policy that keeps the PolicyGenerator image in sync (its Placement selects the hub from global):

The generated Placements select sno and bm by name, from the cluster sets bound to acm-policies. Check the result:

On the hub, one Application syncs the folder:

When ACM propagates a policy, it creates a copy in the namespace of every managed cluster on the hub, the replicated policy. Argo CD could treat these copies as extraneous resources of the Application and show it as OutOfSync or even try to delete them. ACM prevents that: it automatically sets the annotation argocd.argoproj.io/compare-options: IgnoreExtraneous on every replicated policy.

The Placement lives in the Argo CD namespace, next to the ManagedClusterSetBinding from the prerequisites. It selects the OpenShift clusters of the cluster set production.

The result of a Placement is a PlacementDecision. Check which clusters were selected:

With bm and sno in the set, the output is:

The Placement and its PlacementDecision complete the picture on the hub:

At this point nothing has happened in Argo CD yet: Settings > Clusters still shows only in-cluster. The Placement only decides what clusters could be selected. The GitOpsCluster in the next section turns the decision into Argo CD clusters.

The push model is "classic" Argo CD. One Argo CD instance on the hub renders the manifests and applies them to the API server of every managed cluster, by default through the cluster proxy of the multicluster engine. ACM contributes the cluster registration, the credentials and the connection path.

What happens under the hood (the numbers refer to the diagram above):

The consequences:

The GitOpsCluster connects the Placement with the Argo CD instance. This now finally creates the Argo CD clusters on the hub. In the Argo CD UI, Settings > Clusters should show the new clusters:

Verify that ACM created the cluster secrets:

You should see one <cluster-name>-application-manager-cluster-secret per selected cluster.

The clusters also appear in the Argo CD UI under Settings > Clusters. The API URL is the cluster proxy URL:

For the ApplicationSet we use the clusterDecisionResource generator. It reads the PlacementDecision directly, so the list of clusters always reflects the current state of the fleet. The generator requires a ConfigMap that tells it how to read PlacementDecisions. The GitOpsCluster controller creates it automatically under the name acm-placement:

With this information we can create the ApplicationSet using our Compliance Operator as example:

In Argo CD we can see the ApplicationSet which automatically created two Applications, one for each (selected) cluster:

The Applications have been automatically synced by the hub Argo CD (because we configured it to do so).

Compare this with manually maintained cluster secrets: no tokens, no rotation scripts. Label a new cluster with vendor=OpenShift, add it to the cluster set production, and within a few minutes it has its Application. Detach a cluster from ACM and the Application disappears. You do not need to maintain the cluster secrets manually.

For platform configuration, the ServiceAccount application-manager with cluster-admin is often acceptable: the Argo CD instance is the cluster administrator. For application teams it is too much. ACM lets you replace it with your own ManagedServiceAccount and grant it exactly the permissions you want with a ClusterPermission. Both resources are created in the namespace of the managed cluster on the hub:

Then reference the service account in the GitOpsCluster:

The ManagedServiceAccount and the ClusterPermission must exist for every cluster of the Placement. Creating them by hand does not scale and is not GitOps-like.

The pull model turns the direction around. The Application is still defined on the hub, but the hub does not reconcile it. Instead, ACM transports the Application object itself to the managed cluster, where a local Argo CD instance does the actual work.

What happens under the hood:

All three controllers run as containers in the multicluster-integrations pod in the open-cluster-management namespace on the hub. The resource sync and aggregation controllers run every 10 seconds.

The consequences:

The ACM documentation lists the requirements for the pull model:

There are two ways to get a local Argo CD onto every spoke.

Now with Argo CD deployed on the managed cluster, the local application-controller must be allowed to create whatever the Applications contain. Without that, syncs fail with forbidden errors for the service account openshift-gitops-argocd-application-controller. The ACM documentation offers two ways:

For option 1, add the ClusterRoleBinding to the policy that manages the spokes (the <gitopscluster-name>-argocd-policy from option A or your own policy from option B):

The ApplicationSet uses the same generator as the push model: clusterDecisionResource. The template gets a label and three annotations, and the destination changes.

Follow the Application through the chain. First, the Applications on the hub, which are not reconciled there:

Then the ManifestWork the propagation controller created in the namespace of one managed cluster:

On the managed cluster, the Application now exists and is reconciled by the local Argo CD:

Back on the hub, the aggregated status per ApplicationSet:

The report lists the sync and health status per cluster and a summary. A shortened example with one failing cluster:

The forbidden message in the example is the most common error with the pull model: the local controller is missing permissions (step 2). The report is the first place to look. For the details, go to the Argo CD of the spoke: it has the full conditions and events.

The pull model has one weak spot: visibility. The hub only gets an aggregated and delayed status. The hub cluster never connects directly to the spoke clusters and therefore never sees the full resource tree.

The result is the following:

While the ApplicationSet in the hub shows a healthy sync status:

The Application itself shows also a healthy sync status BUT …​ it does not show any child-resources.

The model is decided per ApplicationSet, by the label and annotations in the template. One hub Argo CD can therefore use push and pull ApplicationSets side by side. A pattern I like: platform components via push (one UI, full visibility for a small platform team), application workloads via pull.

The one exception is the GitOps add-on: once it is enabled through a GitOpsCluster, the push model is switched off. If you want to mix, install OpenShift GitOps on the spokes with your own policy (option B).

The pull model has one weak spot: visibility. The hub only gets an aggregated, delayed status. The Argo CD Agent (see Ep.15) solves exactly that. A lightweight agent on the spoke reconciles locally and synchronises the full Application status back to the principal on the hub. The connection is initiated by the spoke (gRPC with mutual TLS), the hub holds no Kubernetes credentials for the spokes, and the Argo CD UI on the hub shows the full resource tree.

Setting up the agent by hand means a lot of plumbing: a CA, certificates for the principal and for every agent, registering every agent, installing the agent on every spoke. This is where ACM helps. The GitOps add-on in Agent mode does all of that for the clusters of a Placement: it installs OpenShift GitOps and the agent on the spokes, manages the PKI and creates the cluster secrets in agent mode.

What happens under the hood:

First we need to prepare Argo CD. We need to modify the Operator Subscription and the Argo CD resource. This step has an important consequence: the Argo CD instance on the hub becomes an agent principal. Its own application-controller is disabled, so this instance no longer pushes to any cluster.

OpenShift GitOps 1.22 itself also supports a hybrid architecture (GA), where the application-controller stays enabled and push and agent Applications run side by side in one instance. The ACM 2.17 documentation for the add-on does not cover that combination yet.

First, add the required environment variables to the Subscription of the OpenShift GitOps Operator on the hub:

Then configure the ArgoCD resource on the hub for agent mode:

Finally, the principal only propagates AppProjects whose destinations and sourceNamespaces match the agents. The ACM documentation therefore configures the default AppProject in openshift-gitops with matching (or wildcard) values.

The controller discovers the principal address from the Route.

You can check the discovered address with the following command:

When the GitOpsCluster is created, ACM also creates the policy <gitopscluster-name>-argocd-policy on the hub. It is created once and then belongs to you, so it is the right place to add the permissions. Two service accounts on the spokes need them:

With destination-based mapping, the ApplicationSet looks almost like a push ApplicationSet. The only difference is the destination: name instead of server.

The big advantage of the Argo CD Agent is the visibility in the UI. While the ApplicationSet in the hub shows a healthy sync status:

The Application itself now also shows the full resource tree.

Without ACM, Argo CD needs to know the target clusters. That means a cluster secret per cluster, carrying the labels we want to select on and the token to access the cluster:

The ApplicationSet then uses the cluster generator to create one Application per matching cluster:

This works well and it is exactly what I do in my own demo environment. The catch is in the details:

With ACM, the same requirement is a policy. A Placement selects the clusters, a PlacementBinding connects the policy with the Placement:

The ManagedClusterSetBinding mentioned in callout 5 looks like the following. It binds the cluster set production to the namespace acm-policies.

Even after working with ACM for a while, I still find it difficult to understand the object model or to explain it to others. Therefore, I tried to create a visual representation of the objects and their relationships here, to help you to understand and to remind myself when I forget. :)

How do these objects play together? The easiest way to understand it is to follow a few clusters through the chain. In the following example, four clusters are imported into ACM. Each step narrows the selection down:

The PlacementDecision itself does not deploy anything. It is just a list and the point where it is handed over to the consumers (Argo CD). Other objects in the same namespace read it and act on it:

Clusters that are not part of the decision receive nothing: no Application, no Policy.

So after all these different objects and relationships, which look rather complex at the beginning, what is the advantage of ACM?

One thing I have seen more than once: Argo CD manages an object with selfHeal: true, and an ACM policy enforces the same object with a slightly different value. Each controller "fixes" the drift caused by the other one. The object flips back and forth.

Example: a single production cluster plus a test cluster.

OpenShift GitOps only. ACM would add a hub, an additional component to operate and additional cost, without solving a real problem. Configure the clusters with Argo CD as described in the earlier episodes of this series, with either one Argo CD instance per cluster or a central Argo CD on a management cluster (recommended). If you need compliance reports, the Compliance Operator gives you scan results per cluster without ACM.

Example: dev, test, prod in two data centres, installed by the infrastructure team.

OpenShift GitOps is usually sufficient. A central Argo CD with an ApplicationSet cluster generator, or decentralised Argo CD instances per cluster. Start thinking about ACM when one of these questions comes up:

Example: clusters per business unit, factory or per tenant, created on demand.

ACM plus OpenShift GitOps. Push for smaller fleets, pull or agent for larger fleets. ACM creates the clusters, labels them and adds them to cluster sets. The GitOpsCluster registers them in Argo CD automatically. Applications and complex platform components come via ApplicationSets driven by ACM Placements. A new cluster receives its full configuration without anybody touching Argo CD. Around 20 clusters is also where a single push hub usually needs sharding, and where the "hub holds all the credentials" discussion with the security team starts. The pull model or the Argo CD Agent become serious options here.

Example: banking or public sector, regular audits, a defined security baseline.

ACM Policies for the baseline, Argo CD for the rest. The baseline lives as policies in Git, deployed by Argo CD to the hub, enforced and reported by ACM. Start in inform mode on existing clusters, check the compliance dashboard, then switch to enforce.

Example: hundreds of small clusters (SNO, compact clusters) in stores, factories or vehicles with unstable connectivity.

ACM plus OpenShift GitOps in pull or agent mode. The push model fails here: the hub cannot reliably reach the spokes. Policies are evaluated locally and continue to work when the connection is down. Applications are reconciled by a local Argo CD or the Argo CD Agent. Two things to keep in mind: every spoke still needs to reach its sources (Git or an OCI registry, often a local mirror), and every cluster that runs the Argo CD Agent needs an OpenShift Platform Plus subscription.