GitOps Episodes > [Ep.19] ACM and OpenShift GitOps: push, pull and the Argo CD Agent

[Ep.19] ACM and OpenShift GitOps: push, pull and the Argo CD Agent

OpenShift GitOps October 23, 2026

Thomas Jungbauer 36 min read (7152 words)

image from [Ep.19] ACM and OpenShift GitOps: push, pull and the Argo CD Agent

In [Ep.18] I compared OpenShift GitOps and Advanced Cluster Management and ended with the statement that they are layers, not competitors. ACM knows the fleet and Argo CD delivers the content. This part is about the glue between the two: how ACM registers its clusters in Argo CD, and the three ways an Argo CD Application can reach a managed cluster: the push model, the pull model and the Argo CD Agent.

For each of them I will explain what it actually is, how the data flows, and how to set it up step by step.

The topic turned out to be too big for a single article, so I have split it into multiple parts and maybe I should have split into even more. This is the second part:

[Ep.18] OpenShift GitOps or Advanced Cluster Management? Choosing the right tool What each tool does, where they overlap and a decision guide.
[Ep.19] ACM and OpenShift GitOps: push, pull and the Argo CD Agent (this article): how ACM registers clusters in Argo CD, and how to set up each of the three delivery models.
[Ep.20] Managing ACM Policies with OpenShift GitOps: policies as code, the PolicyGenerator, hub templates and bootstrapping Argo CD with a policy.

Everything in this series was written against OpenShift GitOps 1.22 (Argo CD 3.5) and Advanced Cluster Management 2.17 (Open Cluster Management). Technology Preview and GA states move quickly in both products, so check the release notes of your versions before you commit to a design.

While I am using the term "Advanced Cluster Management" in this article, I am referring to the product formerly known as Open Cluster Management. Same counts for Argo CD and OpenShift GitOps.

The setup

Before we dig into the models, let’s talk about my lab setup.

There are 3 clusters:
- hub is the ACM hub with openshift-gitops installed.
- bm is a bare metal cluster with a single node.
- sno a single node cluster.

The hub cluster is the starting point. Argo CD on the hub currently knows one cluster: in-cluster. This entity is automatically added during installation of Argo CD and it is the cluster Argo CD is hosted on.

Figure 1. Argo CD settings clusters

Prerequisites for all models

An ACM hub with the OpenShift GitOps Operator installed (the default instance in openshift-gitops).

The ManagedServiceAccount add-on enabled. ACM uses it to create and rotate the tokens for the push and the pull model. It is enabled by default with current versions of the multicluster engine, but check it if you upgraded from an old release: oc get managedclusteraddon managed-serviceaccount -n <cluster-name>.

oc get managedclusteraddon -A | grep managed-serviceaccount                                                                                                                               ─╯
bm              managed-serviceaccount        True                   False
local-cluster   managed-serviceaccount        True                   False
sno             managed-serviceaccount        True                   False

The managed clusters sno and bm are in a ManagedClusterSet. In this article I called it production.

ManagedClusterSet production
apiVersion: cluster.open-cluster-management.io/v1beta2
kind: ManagedClusterSet
metadata:
  name: production (1)
spec:
  clusterSelector:
    selectorType: ExclusiveClusterSetLabel (2)

1	Cluster-scoped, so there is no namespace.
2	The default. A cluster joins the set through the label `cluster.open-cluster-management.io/clusterset=production`. A cluster can only be in one such set at a time.

Add the clusters to the set by labelling them.

oc label managedcluster bm cluster.open-cluster-management.io/clusterset=production --overwrite
oc label managedcluster sno cluster.open-cluster-management.io/clusterset=production --overwrite

Check the result:

oc get managedclusterset production
oc get managedclusters -l cluster.open-cluster-management.io/clusterset=production

NAME         EMPTY   AGE
production   False   53s
NAME   HUB ACCEPTED   MANAGED CLUSTER URLS                   JOINED   AVAILABLE   AGE
bm     true           https://api.bm.stdin.at:6443           True     True        172m
sno    true           https://api.sno.aws.ispworld.at:6443   True     True        51m

A ManagedClusterSetBinding for the cluster set production in the openshift-gitops namespace.

It binds a ManagedClusterSet resource to a namespace. This is easy to forget: the Placement for Argo CD lives in the Argo CD namespace, so the cluster set must be bound there. Without this binding, the Placement does not find a single cluster.

ManagedClusterSetBinding in the Argo CD namespace
apiVersion: cluster.open-cluster-management.io/v1beta2
kind: ManagedClusterSetBinding
metadata:
  name: production
  namespace: openshift-gitops
spec:
  clusterSet: production

Verify that your managed clusters are in the cluster set:

oc get managedclusters -l cluster.open-cluster-management.io/clusterset=production

The following image shows the currently created objects in the hub cluster:

Currently created objects in the hub cluster

The bridge: Placement, GitOpsCluster and ApplicationSet

So far, we only created the prerequisites for all three models. When we now look at the models all three of them start with the same chain of objects on the hub:

a Placement decides which managed clusters are used,
a GitOpsCluster makes these clusters known to an Argo CD instance, and
an ApplicationSet creates one Application per selected cluster.

The GitOpsCluster is the actual bridge between ACM and Argo CD. ACM knows the clusters, Argo CD knows the Applications, and the GitOpsCluster translates one into the other.

What differs between the models is what happens after this chain: who talks to the managed cluster, in which direction the network connection goes, and where the reconciliation happens. That is why the GitOpsCluster and the ApplicationSet look different in each model and are created in the model sections below.

The Placement is identical for all three, so we create it now.

The Placement

The Placement lives in the Argo CD namespace, next to the ManagedClusterSetBinding from the prerequisites. It selects the OpenShift clusters of the cluster set production.

Placement in the Argo CD namespace
apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: production-clusters
  namespace: openshift-gitops (1)
spec:
  clusterSets:
    - production (2)
  predicates:
    - requiredClusterSelector: (3)
        labelSelector:
          matchExpressions:
            - key: vendor
              operator: In
              values:
                - OpenShift
  tolerations: (4)
    - key: cluster.open-cluster-management.io/unreachable
      operator: Exists
    - key: cluster.open-cluster-management.io/unavailable
      operator: Exists

1	The same namespace as the GitOpsCluster that will reference it.
2	Represents the ManagedClusterSets from which the ManagedClusters are selected.
3	The predicate to select only OpenShift clusters.
4	By default a Placement drops clusters that are unreachable. For GitOps we want to keep them, so that a short network hiccup does not remove the cluster secret and, worst case, the generated Applications.

The result of a Placement is a PlacementDecision. Check which clusters were selected:

oc get placementdecisions -n openshift-gitops \
  -l cluster.open-cluster-management.io/placement=production-clusters \
  -o jsonpath='{.items[*].status.decisions[*].clusterName}{"\n"}'

With bm and sno in the set, the output is:

bm sno

The Placement and its PlacementDecision complete the picture on the hub:

Objects in the hub cluster after creating the Placement

At this point nothing has happened in Argo CD yet: Settings > Clusters still shows only in-cluster. The Placement only decides what clusters could be selected. The GitOpsCluster in the next section turns the decision into Argo CD clusters.

The push model

What it actually is

The push model is "classic" Argo CD. One Argo CD instance on the hub renders the manifests and applies them to the API server of every managed cluster, by default through the cluster proxy of the multicluster engine. ACM contributes the cluster registration, the credentials and the connection path.

Push model: the hub Argo CD applies the resources to every managed cluster

What happens under the hood (the numbers refer to the diagram above):

The Placement selects managed clusters, the result is stored in a PlacementDecision.
The GitOpsCluster controller reads the PlacementDecision and creates one Argo CD cluster secret per cluster in the Argo CD namespace, named <cluster-name>-application-manager-cluster-secret.
The token in the secret belongs to the ManagedServiceAccount application-manager. ACM rotates it automatically for you, and the cluster secret on the hub is updated when the token changes. No long-lived tokens.
The server URL in the cluster secret points to the cluster proxy of the multicluster engine. This is the default, because the cluster-proxy add-on is always enabled. Only if the cluster proxy is unavailable does ACM fall back to the API server URL of the managed cluster (OpenShift clusters only).
The ApplicationSet with the clusterDecisionResource generator creates one Application per cluster. The application-controller of the hub Argo CD reconciles all of them.

By default, the application-manager service account has cluster administrator permissions on the managed cluster. Everybody who can read secrets in the Argo CD namespace on the hub therefore effectively has cluster-admin on every registered managed cluster. Do not hand out the admin role for openshift-gitops lightly, and consider a customised service account (see below).

The consequences:

One Argo CD to rule them all. One UI, one place to look, the full resource tree and live diff for every cluster.
The hub carries the load. Every cluster adds watches and cache to the hub’s application-controller. With many clusters, the performance tuning from Ep.16 becomes mandatory.
No hub, no reconciliation. If the hub is down, nothing drifts back into the desired state on the spokes.
Network: with the cluster proxy (default), the managed cluster opens the tunnel to the hub, so the hub does not need to reach the spoke API server directly. Only with the API-URL fallback must the hub reach the spoke’s API server.

Step 1: The GitOpsCluster

The GitOpsCluster connects the Placement with the Argo CD instance. This now finally creates the Argo CD clusters on the hub. In the Argo CD UI, Settings > Clusters should show the new clusters:

GitOpsCluster for the push model
apiVersion: apps.open-cluster-management.io/v1beta1
kind: GitOpsCluster
metadata:
  name: gitops-production
  namespace: openshift-gitops
spec:
  argoServer:
    cluster: local-cluster (1)
    argoNamespace: openshift-gitops (2)
  placementRef:
    kind: Placement
    apiVersion: cluster.open-cluster-management.io/v1beta1
    name: production-clusters (3)

1	The name of the local cluster, where Argo CD runs. On a default hub this is `local-cluster`.
2	The namespace of the Argo CD instance.
3	The Placement from the prerequisites.

Verify that ACM created the cluster secrets:

oc get gitopscluster gitops-production -n openshift-gitops -o yaml
oc get secrets -n openshift-gitops -l argocd.argoproj.io/secret-type=cluster

You should see one <cluster-name>-application-manager-cluster-secret per selected cluster.

NAME                                        TYPE     DATA   AGE
bm-application-manager-cluster-secret       Opaque   3      32s
cluster-kubernetes.default.svc-3396314289   Opaque   3      10h
sno-application-manager-cluster-secret      Opaque   3      32s

The clusters also appear in the Argo CD UI under Settings > Clusters. The API URL is the cluster proxy URL:

Figure 2. GitOpsCluster cluster secrets

The connection status is unknown because the cluster has no Argo CD Applications yet.

Step 2: The ApplicationSet

For the ApplicationSet we use the clusterDecisionResource generator. It reads the PlacementDecision directly, so the list of clusters always reflects the current state of the fleet. The generator requires a ConfigMap that tells it how to read PlacementDecisions. The GitOpsCluster controller creates it automatically under the name acm-placement:

oc get configmap acm-placement -n openshift-gitops -o yaml

data:
  apiVersion: cluster.open-cluster-management.io/v1beta1 (1)
  kind: placementdecisions
  statusListKey: decisions (2)
  matchKey: clusterName (3)

1	The PlacementDecision API.
2	The list in the PlacementDecision status that holds the selected clusters.
3	The field in each list entry that is matched against the `name` of the Argo CD cluster secret.

With this information we can create the ApplicationSet using our Compliance Operator as example:

ApplicationSet for the push model
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: compliance-operator
  namespace: openshift-gitops
spec:
  generators:
    - clusterDecisionResource:
        configMapRef: acm-placement (1)
        labelSelector:
          matchLabels:
            cluster.open-cluster-management.io/placement: production-clusters (2)
        requeueAfterSeconds: 180 (3)
  template:
    metadata:
      name: 'compliance-operator-{{name}}' (4)
    spec:
      project: default
      source:
        repoURL: https://github.com/tjungbauer/openshift-clusterconfig-gitops.git
        targetRevision: main
        path: 'clusters/{{name}}/setup-compliance-operator' (5)
      destination:
        server: '{{server}}' (6)
      syncPolicy:
        automated: (7)
          prune: true
          selfHeal: true
  syncPolicy:
    preserveResourcesOnDeletion: true (8)

1	The ConfigMap created by the GitOpsCluster controller.
2	The name of our Placement.
3	How often the generator re-reads the PlacementDecision (in seconds).
4	Create the name for the Application based on the cluster name.
5	The path to the Compliance Operator setup in the Git repository. Each cluster has its own path in the Git repository.
6	The server URL from the cluster secret, usually the cluster proxy URL.
7	Automated sync policy for the Application.
8	If a cluster leaves the Placement, the generated Application is removed. With this setting, the deployed resources on that cluster are kept.

In Argo CD we can see the ApplicationSet which automatically created two Applications, one for each (selected) cluster:

Figure 3. ApplicationSet Applications in Argo CD

The Applications have been automatically synced by the hub Argo CD (because we configured it to do so).

Figure 4. ApplicationSet Applications synced in Argo CD

The ApplicationSet created the Applications which were immediately synced by the hub Argo CD. They installed and configured the Compliance Operator on the two managed clusters: bm and sno.

Why is this nice?

Compare this with manually maintained cluster secrets: no tokens, no rotation scripts. Label a new cluster with vendor=OpenShift, add it to the cluster set production, and within a few minutes it has its Application. Detach a cluster from ACM and the Application disappears. You do not need to maintain the cluster secrets manually.

What to watch out for

Think about prune: true together with dynamic Placements. Without preserveResourcesOnDeletion, removing a cluster from the Placement (for example by changing a label) can delete the deployed resources on that cluster. Use it for everything that holds data or that the cluster cannot live without.

Least privilege: a customised service account

For platform configuration, the ServiceAccount application-manager with cluster-admin is often acceptable: the Argo CD instance is the cluster administrator. For application teams it is too much. ACM lets you replace it with your own ManagedServiceAccount and grant it exactly the permissions you want with a ClusterPermission. Both resources are created in the namespace of the managed cluster on the hub:

ManagedServiceAccount and ClusterPermission for one managed cluster
apiVersion: authentication.open-cluster-management.io/v1alpha1
kind: ManagedServiceAccount
metadata:
  name: argocd-deployer
  namespace: sno (1)
spec:
  rotation: {} (2)
---
apiVersion: rbac.open-cluster-management.io/v1alpha1
kind: ClusterPermission
metadata:
  name: argocd-deployer
  namespace: sno
spec:
  roles: (3)
    - namespace: shop
      rules:
        - apiGroups: ["apps"]
          resources: ["deployments"]
          verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
        - apiGroups: [""]
          resources: ["services", "configmaps"]
          verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  roleBindings:
    - namespace: shop
      roleRef:
        kind: Role
      subject:
        apiGroup: authentication.open-cluster-management.io
        kind: ManagedServiceAccount
        name: argocd-deployer
  clusterRole: (4)
    rules:
      - apiGroups: ["*"]
        resources: ["*"]
        verbs: ["get", "list", "watch"]
  clusterRoleBinding:
    subject:
      apiGroup: authentication.open-cluster-management.io
      kind: ManagedServiceAccount
      name: argocd-deployer

1	The namespace of the managed cluster on the hub. The service account is created on that cluster.
2	Let ACM rotate the token.
3	Write permissions only in the namespace `shop`.
4	Read-only access to the rest of the cluster, so that Argo CD can discover API resources and show health.

Then reference the service account in the GitOpsCluster:

apiVersion: apps.open-cluster-management.io/v1beta1
kind: GitOpsCluster
metadata:
  name: gitops-team-shop
  namespace: openshift-gitops
spec:
  managedServiceAccountRef: argocd-deployer (1)
  argoServer:
    cluster: local-cluster
    argoNamespace: openshift-gitops
  placementRef:
    kind: Placement
    apiVersion: cluster.open-cluster-management.io/v1beta1
    name: production-clusters

1	Use this ManagedServiceAccount instead of `application-manager`. The cluster secrets are now named `<cluster-name>-argocd-deployer-cluster-secret`.

The ManagedServiceAccount and the ClusterPermission must exist for every cluster of the Placement. Creating them by hand does not scale and is not GitOps-like.

The pull model

What it actually is

The pull model turns the direction around. The Application is still defined on the hub, but the hub does not reconcile it. Instead, ACM transports the Application object itself to the managed cluster, where a local Argo CD instance does the actual work.

Pull model: the hub ships the Application, the local Argo CD reconciles it

What happens under the hood:

The ApplicationSet on the hub generates an Application per cluster, just as in the push model.
The generated Application carries the label apps.open-cluster-management.io/pull-to-ocm-managed-cluster: "true" and the annotation argocd.argoproj.io/skip-reconcile: "true". The latter tells the hub Argo CD to ignore it.
The propagation controller on the hub picks up every Application with the annotation apps.open-cluster-management.io/ocm-managed-cluster. It wraps the Application into a ManifestWork in the namespace of the target cluster. We are still on the hub cluster at this point.
The work agent on the managed cluster (part of the regular ACM klusterlet) pulls the ManifestWork and creates the Application in the openshift-gitops namespace of the managed cluster.
The local Argo CD on the managed cluster pulls Git and reconciles the Application against its own API server.
The status travels back in two ways: the ManifestWork reports the status of the Application object, and the resource sync controller queries the ACM search component on each managed cluster for the resource list and error messages. The aggregation controller combines both into a MulticlusterApplicationSetReport per ApplicationSet.

All three controllers run as containers in the multicluster-integrations pod in the open-cluster-management namespace on the hub. The resource sync and aggregation controllers run every 10 seconds.

The consequences:

The spokes do the work. The hub only creates ManifestWorks, which scales much better than one controller watching every cluster.
The hub never connects to the spoke APIs. All connections are initiated by the managed cluster (the klusterlet of ACM does that anyway). Perfect for firewalls that only allow outbound traffic from the spoke.
No hub, still reconciliation. If the hub goes away, the local Argo CD keeps self-healing with the last known Application.
Limited visibility. The status on the hub is not real-time and not complete. Resources that failed to deploy are not part of the resource list, only the error messages show them. For details, you have to look at the Argo CD of the spoke.
An Argo CD per cluster. Every managed cluster runs a full OpenShift GitOps instance, needs access to Git and its own repository credentials, and must be installed, updated and given the right permissions.

The ACM documentation lists the requirements for the pull model:

OpenShift GitOps installed on the hub and on every target managed cluster, in the namespace openshift-gitops.
OpenShift GitOps on the hub in version 1.9.0 or later, and equal to or newer than the version on the managed clusters.
A cluster secret per managed cluster on the hub, created by a GitOpsCluster. The ApplicationSet generator matches the PlacementDecision against these secrets, even if the hub never uses them to deploy.

So the GitOpsCluster and the Placement from the push model are needed here as well.

Step 1: OpenShift GitOps on the managed clusters

There are two ways to get a local Argo CD onto every spoke.

Option A: the GitOps add-on in Basic mode

The GitOpsCluster can install OpenShift GitOps automatically for you. Simply enable the add-on:

GitOpsCluster with the GitOps add-on in Basic mode
apiVersion: apps.open-cluster-management.io/v1beta1
kind: GitOpsCluster
metadata:
  name: gitops-production
  namespace: openshift-gitops
spec:
  argoServer:
    argoNamespace: openshift-gitops
  placementRef:
    kind: Placement
    apiVersion: cluster.open-cluster-management.io/v1beta1
    name: production-clusters
    namespace: openshift-gitops
  gitopsAddon:
    enabled: true (1)
    overrideExistingConfigs: false (2)

1	Basic mode: deploys the OpenShift GitOps Operator into `openshift-gitops-operator` and an Argo CD instance into `openshift-gitops` on every selected cluster.
2	Safety feature: do not overwrite existing Argo CD configurations on the managed clusters.

When we update the GitOpsCluster resource, that we have created in the previous chapter, the managed clusters will start to install OpenShift GitOps. The Argo CD instance acm-openshift-gitops will be created in the namespace openshift-gitops.

This is by far the easiest way to get a local Argo CD onto every spoke as you need to configure the GitOpsCluster resource anyway. This automatically creates an ACM policy gitops-production-argocd-policy that manages the Argo CD resource on the spokes.

Enabling the GitOps add-on through a GitOpsCluster disables the push model for all applications. This is a fleet-wide decision. If you need push and pull side by side, use option B.
The Basic mode is supported for OpenShift managed clusters only.
To customise the ArgoCD resource on a single spoke without the add-on reverting it, annotate it with gitops-addon.open-cluster-management.io/skip: "true".

Verify the add-on:

# on a managed cluster
oc get pods -n open-cluster-management-agent-addon
oc get pods -n openshift-gitops-operator
oc get pods -n openshift-gitops

Option B: your own policy

If you want better control over the operator version, or you need the push model as well, install OpenShift GitOps with your own policy. This is the same OperatorPolicy pattern as in Ep.18:

Policy that installs OpenShift GitOps on the managed clusters
apiVersion: policy.open-cluster-management.io/v1
kind: Policy
metadata:
  name: openshift-gitops-spoke
  namespace: acm-policies
spec:
  remediationAction: enforce (1)
  disabled: false
  policy-templates:
    - objectDefinition: (2)
        apiVersion: policy.open-cluster-management.io/v1
        kind: ConfigurationPolicy
        metadata:
          name: openshift-gitops-operator-namespace
        spec:
          severity: high
          object-templates:
            - complianceType: musthave
              objectDefinition:
                apiVersion: v1
                kind: Namespace
                metadata:
                  name: openshift-gitops-operator
    - objectDefinition: (3)
        apiVersion: policy.open-cluster-management.io/v1beta1
        kind: OperatorPolicy
        metadata:
          name: openshift-gitops-operator
        spec:
          severity: high
          complianceType: musthave
          subscription:
            name: openshift-gitops-operator
            namespace: openshift-gitops-operator
            channel: gitops-1.22 (4)
            source: redhat-operators
            sourceNamespace: openshift-marketplace
          upgradeApproval: Automatic

1	Enforce the policy.
2	Create the namespace `openshift-gitops-operator` on the managed clusters.
3	Install the OpenShift GitOps Operator.
4	Pin the channel. The hub must run the same or a newer OpenShift GitOps version than the spokes, so upgrade the hub first.

This installs the Namespace and the Operator, the Argo CD instance is created automatically by the Operator.

Bind the policy with a Placement and a PlacementBinding in acm-policies, using the same label selector as the GitOps Placement, so that every cluster that should receive pull Applications also receives a local Argo CD. See above for examples.

Step 2: Permissions for the local Argo CD

Now with Argo CD deployed on the managed cluster, the local application-controller must be allowed to create whatever the Applications contain. Without that, syncs fail with forbidden errors for the service account openshift-gitops-argocd-application-controller. The ACM documentation offers two ways:

Cluster administrator: a ClusterRoleBinding on every managed cluster. Simple, and for platform configuration usually what you want. For example, to install a new Operator.
Least privilege: create every destination namespace on the managed cluster with the label argocd.argoproj.io/managed-by: openshift-gitops. The OpenShift GitOps Operator then grants the local instance admin rights in exactly these namespaces. The namespaces must be declared in Git, as part of the Application.

For option 1, add the ClusterRoleBinding to the policy that manages the spokes (the <gitopscluster-name>-argocd-policy from option A or your own policy from option B):

Policy template granting the local controller cluster-admin
    - objectDefinition:
        apiVersion: policy.open-cluster-management.io/v1
        kind: ConfigurationPolicy
        metadata:
          name: openshift-gitops-controller-rbac
        spec:
          severity: high
          object-templates:
            - complianceType: musthave
              objectDefinition:
                apiVersion: rbac.authorization.k8s.io/v1
                kind: ClusterRoleBinding
                metadata:
                  name: argo-admin
                subjects:
                  - kind: ServiceAccount
                    name: openshift-gitops-argocd-application-controller
                    namespace: openshift-gitops
                roleRef:
                  apiGroup: rbac.authorization.k8s.io
                  kind: ClusterRole
                  name: cluster-admin

Step 3: The ApplicationSet for the pull model

The ApplicationSet uses the same generator as the push model: clusterDecisionResource. The template gets a label and three annotations, and the destination changes.

The Compliance Operator is already installed on sno and bm by the push model. To keep the two models from fighting over the same objects, the pull example deploys something else: the console branding from the folder clusters/<cluster>/branding of my GitOps repository.

ApplicationSet for the pull model
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: branding-pull
  namespace: openshift-gitops
spec:
  generators:
    - clusterDecisionResource:
        configMapRef: acm-placement
        labelSelector:
          matchLabels:
            cluster.open-cluster-management.io/placement: production-clusters
        requeueAfterSeconds: 180
  template:
    metadata:
      name: 'branding-{{name}}'
      labels:
        apps.open-cluster-management.io/pull-to-ocm-managed-cluster: 'true' (1)
      annotations:
        argocd.argoproj.io/skip-reconcile: 'true' (2)
        apps.open-cluster-management.io/ocm-managed-cluster: '{{name}}' (3)
        apps.open-cluster-management.io/ocm-managed-cluster-app-namespace: openshift-gitops (4)
    spec:
      project: default (5)
      source:
        repoURL: https://github.com/tjungbauer/openshift-clusterconfig-gitops.git
        targetRevision: main
        path: 'clusters/{{name}}/branding' (6)
      destination:
        server: https://kubernetes.default.svc (7)
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

1	Marks the Application as a pull model Application.
2	Prevents the Argo CD instance on the hub from reconciling the Application itself.
3	The target managed cluster. The propagation controller only processes Applications with this annotation.
4	The namespace on the managed cluster where the Application is created, i.e. the namespace of the local Argo CD.
5	The AppProject must exist on the managed cluster. The default instance always has `default`. Custom projects, and the repository credentials for private repositories, have to be created on the spokes, for example by another policy.
6	One folder per cluster: `clusters/sno/branding` and `clusters/bm/branding`.
7	From the perspective of the local Argo CD on the spoke, the destination is the cluster it runs on.

Do not forget the skip-reconcile annotation. Without it, the hub Argo CD tries to sync the Application to https://kubernetes.default.svc, which is the hub itself. You would deploy the configuration of every managed cluster onto the hub.

Step 4: Verify the flow

Follow the Application through the chain. First, the Applications on the hub, which are not reconciled there:

oc get applications -n openshift-gitops -l apps.open-cluster-management.io/pull-to-ocm-managed-cluster=true

Then the ManifestWork the propagation controller created in the namespace of one managed cluster:

oc get manifestwork -n sno

On the managed cluster, the Application now exists and is reconciled by the local Argo CD:

oc get applications -n openshift-gitops

Back on the hub, the aggregated status per ApplicationSet:

oc get multiclusterapplicationsetreports -n openshift-gitops
oc get multiclusterapplicationsetreport branding-pull -n openshift-gitops -o yaml

The report lists the sync and health status per cluster and a summary. A shortened example with one failing cluster:

apiVersion: apps.open-cluster-management.io/v1alpha1
kind: MulticlusterApplicationSetReport
metadata:
  name: branding-pull
  namespace: openshift-gitops
statuses:
  clusterConditions:
    - cluster: sno
      healthStatus: Healthy
      syncStatus: Synced
    - cluster: bm
      healthStatus: Missing
      syncStatus: OutOfSync
      conditions:
        - type: SyncError
          message: 'Failed sync attempt: ... is forbidden: User "system:serviceaccount:openshift-gitops:openshift-gitops-argocd-application-controller" cannot create resource ...'
  summary:
    clusters: "2"
    healthy: "1"
    notHealthy: "1"
    synced: "1"
    notSynced: "1"

The forbidden message in the example is the most common error with the pull model: the local controller is missing permissions (step 2). The report is the first place to look. For the details, go to the Argo CD of the spoke: it has the full conditions and events.

The visibility trade-off

The pull model has one weak spot: visibility. The hub only gets an aggregated and delayed status. The hub cluster never connects directly to the spoke clusters and therefore never sees the full resource tree.

The result is the following:

While the ApplicationSet in the hub shows a healthy sync status:

Figure 5. ApplicationSet in the hub shows a healthy sync status

The Application itself shows also a healthy sync status BUT … it does not show any child-resources.

Figure 6. Pull Model UI Trade-off

The whole resource tree is missing in the UI. You are basically blind and you cannot see a Diff. You can check oc get multiclusterapplicationsetreport branding-pull -n openshift-gitops -o yaml to see the full status, but that’s not as nice as seeing it in the UI.

Mixing push and pull

The model is decided per ApplicationSet, by the label and annotations in the template. One hub Argo CD can therefore use push and pull ApplicationSets side by side. A pattern I like: platform components via push (one UI, full visibility for a small platform team), application workloads via pull.

The one exception is the GitOps add-on: once it is enabled through a GitOpsCluster, the push model is switched off. If you want to mix, install OpenShift GitOps on the spokes with your own policy (option B).

The Argo CD Agent via the GitOps add-on

What it actually is

The pull model has one weak spot: visibility. The hub only gets an aggregated, delayed status. The Argo CD Agent (see Ep.15) solves exactly that. A lightweight agent on the spoke reconciles locally and synchronises the full Application status back to the principal on the hub. The connection is initiated by the spoke (gRPC with mutual TLS), the hub holds no Kubernetes credentials for the spokes, and the Argo CD UI on the hub shows the full resource tree.

Setting up the agent by hand means a lot of plumbing: a CA, certificates for the principal and for every agent, registering every agent, installing the agent on every spoke. This is where ACM helps. The GitOps add-on in Agent mode does all of that for the clusters of a Placement: it installs OpenShift GitOps and the agent on the spokes, manages the PKI and creates the cluster secrets in agent mode.

Argo CD Agent: the agent opens the connection, receives the Applications and sends the full status back

What happens under the hood:

The Argo CD instance on the hub runs as principal. In the ACM configuration its application-controller is disabled, and a Route exposes the gRPC endpoint for the agents.
The GitOpsCluster with the add-on and the agent enabled creates the certificates, the cluster secrets in agent mode and the ManifestWorks for every cluster of the Placement.
The gitops-addon on the managed cluster installs OpenShift GitOps and the agent, including the client certificate of the agent.
The agent opens the connection to the principal: a gRPC stream with mutual TLS, initiated by the spoke.
The ApplicationSet generates the Applications on the hub. With destination-based mapping, the principal sends each Application to the agent whose name matches spec.destination.name.
The local components on the managed cluster pull Git and reconcile. The agent sends the full status back, so the Argo CD UI on the hub shows the resource tree, health and events of every cluster.

The Agent mode of the GitOps add-on is Technology Preview in ACM 2.17. With ACM 5.0 which is planned for November 2026 this will become GA.
Every cluster that runs the Argo CD Agent requires an OpenShift Platform Plus subscription.

Step 1: Prepare the hub Argo CD

First we need to prepare Argo CD. We need to modify the Operator Subscription and the Argo CD resource. This step has an important consequence: the Argo CD instance on the hub becomes an agent principal. Its own application-controller is disabled, so this instance no longer pushes to any cluster.

OpenShift GitOps 1.22 itself also supports a hybrid architecture (GA), where the application-controller stays enabled and push and agent Applications run side by side in one instance. The ACM 2.17 documentation for the add-on does not cover that combination yet.

First, add the required environment variables to the Subscription of the OpenShift GitOps Operator on the hub:

Subscription of the OpenShift GitOps Operator on the hub (excerpt)
spec:
  config:
    env:
      - name: ARGOCD_CLUSTER_CONFIG_NAMESPACES (1)
        value: openshift-gitops,local-cluster
      - name: ARGOCD_PRINCIPAL_TLS_SERVER_ALLOW_GENERATE (2)
        value: "false"
      - name: ARGOCD_PRINCIPAL_REDIS_SERVER_ADDRESS (3)
        value: openshift-gitops-redis:6379

1	Namespaces in which the OpenShift GitOps Operator runs Argo CD instances as cluster-scoped instances, with permissions for the whole cluster. The ACM documentation lists the default instance in `openshift-gitops` and `local-cluster`, the cluster namespace of the hub itself.
2	Do not let the principal generate a temporary, self-signed TLS certificate. That option is meant for development only. The server certificate of the principal is provided instead (the GitOps add-on manages the PKI).
3	The Redis instance the principal uses. The default of the principal is `argocd-redis:6379`, which does not exist with OpenShift GitOps. The Redis service of the default instance is called `openshift-gitops-redis`.

Then configure the ArgoCD resource on the hub for agent mode:

ArgoCD resource on the hub in agent mode
apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
  name: openshift-gitops
  namespace: openshift-gitops
spec:
  controller:
    enabled: false (1)
  sourceNamespaces:
    - '*'
  argoCDAgent:
    principal:
      enabled: true (2)
      destinationBasedMapping: true (3)
      auth: "mtls:CN=system:open-cluster-management:cluster:([^:]+):addon:gitops-addon:agent:gitops-addon-agent" (4)
      namespace:
        allowedNamespaces:
          - '*'
      server:
        route:
          enabled: true (5)

1	The hub instance no longer reconciles Applications itself.
2	Turns the instance into an agent principal.
3	Applications are routed to an agent by `spec.destination.name` instead of by namespace.
4	The agent identity is taken from the client certificate that the add-on issues.
5	Exposes the principal with a Route. The agents connect to it, and the GitOpsCluster controller discovers its address from there.

Finally, the principal only propagates AppProjects whose destinations and sourceNamespaces match the agents. The ACM documentation therefore configures the default AppProject in openshift-gitops with matching (or wildcard) values.

If your hub Argo CD currently deploys anything via the push model, do not convert it. Install a second, dedicated Argo CD instance for the agent principal, or keep the agent for a separate hub. OpenShift GitOps 1.22 itself supports a hybrid architecture (push and agent Applications in one control plane, GA in 1.22), but the ACM 2.17 documentation for the add-on configures the hub instance with its controller disabled.

Step 2: The GitOpsCluster in agent mode

GitOpsCluster with the GitOps add-on and the Argo CD Agent
apiVersion: apps.open-cluster-management.io/v1beta1
kind: GitOpsCluster
metadata:
  name: gitops-agent-clusters
  namespace: openshift-gitops
spec:
  argoServer:
    argoNamespace: openshift-gitops
  placementRef:
    kind: Placement
    apiVersion: cluster.open-cluster-management.io/v1beta1
    name: production-clusters
    namespace: openshift-gitops
  gitopsAddon:
    enabled: true (1)
    argoCDAgent:
      enabled: true (2)
      mode: managed (3)

1	Installs OpenShift GitOps on the spokes, as in Basic mode.
2	Additionally deploys the agent, creates the PKI and the cluster secrets in agent mode.
3	`managed`: the hub is the source of truth for Applications. `autonomous`: the spoke owns its Applications and the hub only observes them. The default is `managed`.

The controller discovers the principal address from the Route.

You can check the discovered address with the following command:

It will take a while until it appears.

oc get gitopscluster gitops-agent-clusters -n openshift-gitops \
  -o jsonpath='{.spec.gitopsAddon.argoCDAgent.serverAddress}{"\n"}'

Step 3: Permissions on the managed clusters

When the GitOpsCluster is created, ACM also creates the policy <gitopscluster-name>-argocd-policy on the hub. It is created once and then belongs to you, so it is the right place to add the permissions. Two service accounts on the spokes need them:

the agent itself, which reports the status and the live resources to the principal, needs read access,
the local application-controller, which actually applies the manifests, needs write access to everything the Applications deploy.

Additional object templates in the gitops-agent-clusters-argocd-policy
            - complianceType: musthave
              objectDefinition:
                apiVersion: rbac.authorization.k8s.io/v1
                kind: ClusterRoleBinding
                metadata:
                  name: acm-openshift-gitops-argocd-agent-cluster-reader
                subjects:
                  - kind: ServiceAccount
                    name: acm-openshift-gitops-agent-agent (1)
                    namespace: openshift-gitops
                roleRef:
                  apiGroup: rbac.authorization.k8s.io
                  kind: ClusterRole
                  name: view
            - complianceType: musthave
              objectDefinition:
                apiVersion: rbac.authorization.k8s.io/v1
                kind: ClusterRoleBinding
                metadata:
                  name: argo-admin
                subjects:
                  - kind: ServiceAccount
                    name: acm-openshift-gitops-argocd-application-controller (2)
                    namespace: openshift-gitops
                roleRef:
                  apiGroup: rbac.authorization.k8s.io
                  kind: ClusterRole
                  name: cluster-admin (3)

1	The agent: read access is enough.
2	The local application-controller of the instance the add-on creates.
3	Simple and usually fine for platform configuration. For application teams, use the least-privilege approach from the pull model instead: the label `argocd.argoproj.io/managed-by` on the destination namespaces.

Step 4: The ApplicationSet

With destination-based mapping, the ApplicationSet looks almost like a push ApplicationSet. The only difference is the destination: name instead of server.

The push example already installed the Compliance Operator and the pull example the console branding. The agent example therefore uses a third folder, clusters/<cluster>/branding-bottom, so that the three models do not manage the same objects.

ApplicationSet for agent mode
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: branding-bottom-agent
  namespace: openshift-gitops
spec:
  generators:
    - clusterDecisionResource:
        configMapRef: acm-placement
        labelSelector:
          matchLabels:
            cluster.open-cluster-management.io/placement: production-clusters
        requeueAfterSeconds: 30
  template:
    metadata:
      name: 'branding-bottom-{{name}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/tjungbauer/openshift-clusterconfig-gitops.git
        targetRevision: main
        path: 'clusters/{{name}}/branding-bottom' (1)
      destination:
        name: '{{name}}' (2)
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

1	One folder per cluster: `clusters/sno/branding-bottom` and `clusters/bm/branding-bottom`.
2	The principal routes the Application to the agent with this name, which is the name of the managed cluster.

Step 5: Verify

# hub: overall state, the conditions tell you which step failed
oc get gitopscluster gitops-agent-clusters -n openshift-gitops \
  -o jsonpath='{.status.conditions}' | jq

# hub: the principal is running
oc get pods -n openshift-gitops -l app.kubernetes.io/name=openshift-gitops-agent-principal

# managed cluster: the agent is running
oc get pods -n openshift-gitops -l app.kubernetes.io/part-of=argocd-agent

# hub: Application status, synchronised by the agent
oc get applications.argoproj.io -n openshift-gitops

The visibility in the UI

The big advantage of the Argo CD Agent is the visibility in the UI. While the ApplicationSet in the hub shows a healthy sync status:

Figure 7. ApplicationSet in the hub shows a healthy sync status

The Application itself now also shows the full resource tree.

Figure 8. Agent Model UI Visibility

Side by side

Aspect	Push	Pull	Argo CD Agent (add-on)
Status in ACM 2.17	GA, default	GA	Technology Preview (GA in ACM 5.0)
Where Argo CD reconciles	Hub	Managed cluster (full Argo CD instance)	Managed cluster (agent with local Argo CD components)
Network direction	Hub → spoke (cluster proxy or API)	Spoke → hub (klusterlet)	Spoke → hub (agent → principal)
Spoke credentials on the hub	Token per cluster	Cluster secret exists, not used for delivery	mTLS certificates, managed by the add-on
Status visible on the hub	Full (resource tree, live diff)	Aggregated report, not real-time	Full, synchronised by the agent
Hub down	No reconciliation	Reconciliation continues	Reconciliation continues
Footprint on the spoke	None	Full OpenShift GitOps instance	OpenShift GitOps plus agent
Git access needed from	Hub only	Every spoke	Every spoke
Can be mixed with push on the same hub instance	–	Yes (unless the add-on is enabled)	Not in the ACM 2.17 documented setup (hub controller disabled)

Aspect

Push

Pull

Argo CD Agent (add-on)

Status in ACM 2.17

GA, default

Technology Preview (GA in ACM 5.0)

Where Argo CD reconciles

Hub

Managed cluster (full Argo CD instance)

Managed cluster (agent with local Argo CD components)

Network direction

Hub → spoke (cluster proxy or API)

Spoke → hub (klusterlet)

Spoke → hub (agent → principal)

Spoke credentials on the hub

Token per cluster

Cluster secret exists, not used for delivery

mTLS certificates, managed by the add-on

Status visible on the hub

Full (resource tree, live diff)

Aggregated report, not real-time

Full, synchronised by the agent

Hub down

No reconciliation

Reconciliation continues

Footprint on the spoke

None

Full OpenShift GitOps instance

OpenShift GitOps plus agent

Git access needed from

Hub only

Every spoke

Can be mixed with push on the same hub instance

–

Yes (unless the add-on is enabled)

Not in the ACM 2.17 documented setup (hub controller disabled)

Summary

I have to admit that this article is a bit long. But I hope it gives you a good overview of the different models and how to set them up.

Always keep in mind:

The GitOpsCluster is the bridge: it turns placed managed clusters into Argo CD cluster secrets and keeps the tokens rotated.
Push is the default and the easiest start: one Argo CD on the hub, full visibility, but the hub holds cluster-admin tokens for every spoke unless you use your own ManagedServiceAccount. With a huge fleet, usually greater than 20 clusters, it becomes unmanageable.
Pull moves the reconciliation to the spokes: the hub ships Applications as ManifestWorks, a local Argo CD does the work. More scalable and outbound-only, at the price of an Argo CD per cluster and a delayed, aggregated status as well as a trade-off in visibility.
The Argo CD Agent add-on combines pull-style distribution with a full UI on the hub, and ACM takes care of the PKI. It is Technology Preview in ACM 2.17 and in the documented setup it turns the hub Argo CD into a principal without its own controller.

In the last part, we turn the integration around: Argo CD deploys ACM policies.

And as always: If it is not in Git, it does not exist.

The diagrams in this article have been manually drawn on a whiteboard. AI then generated the SVGs from photos and made them look equal in all articles.

References

GitOps Episodes Part 19 of 20

16 [Ep.16] Tuning OpenShift GitOps for performance at scale 17 [Ep.17] Closing the gap between CI and CD with the Argo CD Image … 18 [Ep.18] OpenShift GitOps or Advanced Cluster Management? Choosing the … 19 [Ep.19] ACM and OpenShift GitOps: push, pull and the Argo CD Agent 20 [Ep.20] Managing ACM Policies with OpenShift GitOps

Show all 20 parts

1 [Ep.1] Introducing the GitOps Approach 2 [Ep.2] Choosing the right Git repository structure 3 [Ep.3] Setup OpenShift GitOps/Argo CD 4 [Ep.4] Configure App-of-Apps 5 [Ep.5] Setup & Configure Compliance Operator 6 [Ep.6] Setup & Configure Advanced Cluster Security 7 [Ep.7] Configure Buckets in MinIO 8 [Ep.8] Installing OpenShift Logging 9 [Ep.9] Multiple Sources for Applications 10 [Ep.10] Update Cluster Version 11 [Ep.11] Managing Certificates 12 [Ep.12] Using Kustomize Post-Renderer 13 [Ep.13] ApplicationSet with Matrix Generator 14 [Ep.14] Reusable Argo CD Application Template 15 [Ep.15] OpenShift GitOps - Argo CD Agent 16 [Ep.16] Tuning OpenShift GitOps for performance at scale 17 [Ep.17] Closing the gap between CI and CD with the Argo CD Image … 18 [Ep.18] OpenShift GitOps or Advanced Cluster Management? Choosing the … 19 [Ep.19] ACM and OpenShift GitOps: push, pull and the Argo CD Agent 20 [Ep.20] Managing ACM Policies with OpenShift GitOps

Previous: [Ep.18] OpenShift GitOps or … All 20 parts Next: [Ep.20] Managing ACM Policies …

[Ep.19] ACM and OpenShift GitOps: push, pull and the Argo CD Agent

The setup

Prerequisites for all models

The bridge: Placement, GitOpsCluster and ApplicationSet

The Placement

The push model

What it actually is

Step 1: The GitOpsCluster

Step 2: The ApplicationSet

Why is this nice?

What to watch out for

Least privilege: a customised service account

The pull model

What it actually is

Step 1: OpenShift GitOps on the managed clusters

Option A: the GitOps add-on in Basic mode

Option B: your own policy

Step 2: Permissions for the local Argo CD

Step 3: The ApplicationSet for the pull model

Step 4: Verify the flow

The visibility trade-off

Mixing push and pull

The Argo CD Agent via the GitOps add-on

What it actually is

Step 1: Prepare the hub Argo CD

Step 2: The GitOpsCluster in agent mode

Step 3: Permissions on the managed clusters

Step 4: The ApplicationSet

Step 5: Verify

The visibility in the UI

Side by side

Summary

References

Related Articles

Discussion