Skip to main content

[Ep.18] OpenShift GitOps or Advanced Cluster Management? Choosing the right tool

At customer engagements where I support the implementation of a GitOps process, I often hear the question: "Should we use OpenShift GitOps (Argo CD) or Advanced Cluster Management (Open Cluster Management)?" The honest answer is: it is the wrong question. Both tools solve different problems, they overlap in one specific area, and in larger environments with numerous clusters they are designed to work together.

In this article I will try to explain what each tool is (and also what it is not), where they overlap, how they integrate, and I will walk through an example, so that you can decide for yourself which tool fits which job.

The topic turned out to be too big for a single article, so I have split it into multiple parts and maybe I should have split into even more. This is the first:

  1. [Ep.18] OpenShift GitOps or Advanced Cluster Management? (this article): what each tool does, where they overlap and a decision guide.

  2. [Ep.19] ACM and OpenShift GitOps: push, pull and the Argo CD Agent: how ACM registers clusters in Argo CD, and how to set up each of the three delivery models.

  3. [Ep.20] Managing ACM Policies with OpenShift GitOps: policies as code, the PolicyGenerator, hub templates and bootstrapping Argo CD with a policy.

Everything in this series was written against OpenShift GitOps 1.22 (Argo CD 3.5) and Advanced Cluster Management 2.17 (Open Cluster Management). Technology Preview and GA states move quickly in both products, so check the release notes of your versions before you commit to a design.
While I am using the term "Advanced Cluster Management" in this article, I am referring to the product formerly known as Open Cluster Management. Same counts for Argo CD and OpenShift GitOps.

The short answer

  • OpenShift GitOps (Argo CD) answers the question: "Is the state in Git the state on my cluster(s)?" It is a delivery engine.

  • Advanced Cluster Management (ACM) answers the question: "Which clusters do I have, how are they doing, and do they comply with my rules?" It is a fleet management platform.

  • ACM does not replace Argo CD. For application delivery, ACM uses OpenShift GitOps. What ACM adds is the knowledge about the fleet: which clusters exist, which labels they carry, which ones are healthy.

  • The real overlap is cluster configuration: you can configure clusters with Argo CD Applications or with ACM Policies. Choose what fits best, and never let both tools manage the same object.

Or, in one sentence I like to use with customers: ACM decides where and checks compliance. Argo CD decides what and keeps it in sync.

Other features ACM offers are an automated cluster deployment and centralized observability. But these are not the focus of this article.

What OpenShift GitOps is

OpenShift GitOps is Red Hat’s distribution of Argo CD, delivered and lifecycle-managed by the OpenShift GitOps Operator. It is included in the OpenShift subscription. At the time of writing, the current version is 1.22, which ships Argo CD 3.5. New releases and features come quickly, always check the release notes of your version first.

The core job of Argo CD is reconciliation: it renders manifests (plain YAML, Kustomize, Helm, etc.), compares them with the live state of a cluster and syncs differences. Everything else (ApplicationSets, sync waves, hooks, RBAC via AppProjects, the UI) is built around that job.

If you want to dig into the details of the GitOps process, Ep.1 is a good introduction to the topic.

What OpenShift GitOps does not do:

  • It does not create, upgrade, or destroy clusters. A cluster has to exist, and someone has to register it in Argo CD.

  • It does not report compliance. It tells you that an (Argo CD) Application is Synced and Healthy. It does not tell you that three out of 40 clusters violate your NIST 800-53 baseline.

  • It has no fleet-wide view of anything that is not an Argo CD Application: no search across clusters, no metrics, no inventory.

That is perfectly fine. Argo CD was never meant to do any of these things.

What Advanced Cluster Management is

Red Hat Advanced Cluster Management for Kubernetes is a hub-and-spoke platform. You install it on a hub cluster and import or create managed clusters. It is a separate subscription, or part of OpenShift Platform Plus. At the time of writing, the current version is 2.17. The open source equivalent is called Open Cluster Management.

ACM is built around five pillars:

PillarWhat it does

Cluster lifecycle (multicluster engine)

Create, import, upgrade, and destroy clusters: Installer Provided Infrastructure (IPI) on cloud providers, Hosted Control Planes, Assisted/Agent-based installs on bare metal, ROSA/ARO, import clusters and so on.

Governance

Policies that check (inform) or enforce configuration on managed clusters, with compliance status reported back to the hub. Integrates with Gatekeeper, the Compliance Operator and certificate checks.

Application lifecycle

Deploy applications to many clusters. It uses Argo CD in the background for this task.

Observability

Centralized metrics and Grafana dashboards across the fleet.

Search and fleet views

Search any resource across all managed clusters, including a dedicated view for virtual machines from OpenShift Virtualization.

The building blocks you will meet everywhere in ACM are:

  • ManagedCluster – the cluster that is managed, the representation of a spoke on the hub, carrying labels (for example environment=production, region=eu-central).

  • ManagedClusterSet – a group of clusters, used for RBAC and scoping.

  • ManagedClusterSetBinding – makes a cluster set usable from a namespace on the hub.

  • Placement – a dynamic selection of clusters based on labels, etc. It defines the scope of a policy or application.

Below is a more detailed description of the different objects

The Placement is the important concept for this series. Argo CD selects clusters by looking at labels on cluster secrets that someone created. ACM selects clusters by looking at the actual fleet, including whether a cluster is available right now.

ACM also has its own, older application model based on Channels and Subscriptions objects. That model is deprecated and will be removed in a future release.

Side-by-side comparison

Let’s do a side-by-side comparison of the two tools:

TopicOpenShift GitOpsAdvanced Cluster Management

Primary purpose

Continuous delivery from Git

Fleet management: lifecycle, governance, visibility

Scope

One or more clusters, registered manually

Hundreds or thousands of managed clusters

Cluster creation and upgrades

No (cluster upgrades only via a ClusterVersion manifest, see Ep.10)

Yes (including upgrades with pre/post hooks)

Source of truth

Git (or Helm/OCI repository)

The hub cluster. The hub objects themselves can (and should) come from Git, so maybe it is still Git after all.

Configuration management

Applications, ApplicationSets

Policies (ConfigurationPolicy, OperatorPolicy, …​)

"Inform only" / audit mode

Partly, shows OutOfSync of managed resources

Yes (remediationAction: inform)

Compliance reporting

No

Yes, per policy, per cluster, with history

Per-cluster templating

ApplicationSet generators, Helm values

Hub templates and managed-cluster templates in policies

Behaviour when the hub is unreachable

Push model: nothing is reconciled. Agent/pull model: local reconciliation continues

Policies keep being evaluated and enforced on the managed cluster

Licensing

Included in OpenShift. Clusters running the Argo CD Agent need OpenShift Platform Plus

Separate subscription or OpenShift Platform Plus

Who does what in a combined setup

When both tools are in place, the question is no longer "which one", but "which one for which task". This is the split I would like to recommend:

AreaACMArgo CD

Cluster lifecycle

Create, import, upgrade, hibernate

–

Fleet inventory and selection

ManagedCluster labels, ManagedClusterSets, Placements

Consumes them (GitOpsCluster, cluster decision generator)

Governance / Compliance

Policies (inform/enforce), compliance history

Deploys the policies from Git to the hub. Can show health status.

Cluster configuration

Possible with ConfigurationPolicies

Preferred: richer diff, sync waves, health

Application delivery

Placement of Applications, pull transport

The delivery engine (push, pull, agent)

Observability

Multicluster observability, search

Sync and health status only

Bootstrapping Argo CD on new clusters

Policy or GitOps add-on installs it everywhere

–

Where they overlap: cluster configuration

While application deployment is clearly the responsibility of Argo CD, creating new clusters or upgrading existing ones is clearly ACM territory. The grey zone is cluster configuration: Operators, OAuth, RBAC, network policies, the compliance baseline. Both tools can do it, but you should choose what fits best.

To make the difference visible, let’s implement the same requirement twice:

"Every production cluster must run the Compliance Operator from the stable channel."

Option 1: Argo CD with an ApplicationSet

Without ACM, Argo CD needs to know the target clusters. That means a cluster secret per cluster, carrying the labels we want to select on and the token to access the cluster:

Cluster secret for a production cluster
apiVersion: v1
kind: Secret
metadata:
  name: cluster-prod-eu-01
  namespace: openshift-gitops
  labels:
    argocd.argoproj.io/secret-type: cluster (1)
    environment: production (2)
type: Opaque
stringData:
  name: prod-eu-01
  server: https://api.prod-eu-01.example.com:6443
  config: | (3)
    {
      "bearerToken": "<token-of-a-service-account-on-the-spoke>",
      "tlsClientConfig": {
        "insecure": false,
        "caData": "<base64-encoded-ca>"
      }
    }
1Marks this secret as a cluster definition for Argo CD.
2Our own label. The ApplicationSet uses it for the selection.
3Access configuration for the cluster.

The ApplicationSet then uses the cluster generator to create one Application per matching cluster:

Other generators are available too, but for simplicity only the cluster generator is used here.
ApplicationSet with cluster generator
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: compliance-operator
  namespace: openshift-gitops
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - clusters:
        selector:
          matchLabels:
            environment: production (1)
  template:
    metadata:
      name: 'compliance-operator-{{ .name }}'
    spec:
      project: default
      source:
        repoURL: https://github.com/tjungbauer/openshift-clusterconfig-gitops.git
        targetRevision: main
        path: clusters/all/setup-compliance-operator
      destination:
        server: '{{ .server }}' (2)
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
1Selects all cluster secrets with this label.
2Argo CD on the hub pushes the manifests directly to the API of each spoke.

This works well and it is exactly what I do in my own demo environment. The catch is in the details:

  • The token in the cluster secret must be created, rotated and stored securely. For 5 clusters that is fine, for 200 clusters it is a project of its own.

  • The hub needs network access to the API of every spoke.

  • Whoever controls this Argo CD instance effectively has the permissions of every token it holds.

  • The result is "Synced/Healthy". There is no compliance report you can hand to an auditor.

Option 2: ACM with an OperatorPolicy

With ACM, the same requirement is a policy. A Placement selects the clusters, a PlacementBinding connects the policy with the Placement:

Policy, Placement and PlacementBinding
apiVersion: policy.open-cluster-management.io/v1
kind: Policy
metadata:
  name: compliance-operator
  namespace: acm-policies
  annotations:
    policy.open-cluster-management.io/standards: NIST SP 800-53
    policy.open-cluster-management.io/categories: CM Configuration Management
    policy.open-cluster-management.io/controls: CM-2 Baseline Configuration
spec:
  disabled: false
  remediationAction: enforce (1)
  policy-templates:
    - objectDefinition:
        apiVersion: policy.open-cluster-management.io/v1
        kind: ConfigurationPolicy (2)
        metadata:
          name: compliance-operator-namespace
        spec:
          severity: high
          object-templates:
            - complianceType: musthave
              objectDefinition:
                apiVersion: v1
                kind: Namespace
                metadata:
                  name: openshift-compliance
    - objectDefinition:
        apiVersion: policy.open-cluster-management.io/v1beta1
        kind: OperatorPolicy (3)
        metadata:
          name: compliance-operator
        spec:
          severity: high
          complianceType: musthave
          subscription:
            name: compliance-operator
            namespace: openshift-compliance
            channel: stable
            source: redhat-operators
            sourceNamespace: openshift-marketplace
          upgradeApproval: Automatic (4)
---
apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: production-clusters
  namespace: acm-policies
spec:
  clusterSets:
    - production (5)
  predicates:
    - requiredClusterSelector:
        labelSelector:
          matchLabels:
            environment: production
---
apiVersion: policy.open-cluster-management.io/v1
kind: PlacementBinding (6)
metadata:
  name: compliance-operator
  namespace: acm-policies
placementRef:
  name: production-clusters
  apiGroup: cluster.open-cluster-management.io
  kind: Placement
subjects:
  - name: compliance-operator
    apiGroup: policy.open-cluster-management.io
    kind: Policy
1Set this to inform and ACM only reports whether the clusters comply, without changing anything.
2A ConfigurationPolicy ensures that an object exists (musthave), matches exactly (mustonlyhave) or does not exist (mustnothave).
3The OperatorPolicy handles the OLM details: OperatorGroup, Subscription, install plan approval and the health of the CSV.
4Controls whether new versions are approved automatically. With None, ACM does not approve upgrades. With spec.complianceConfig, the policy can additionally report available upgrades, since ACM 2.17 even minor version upgrades on other channels.
5The Placement can only see clusters of cluster sets that are bound to the namespace acm-policies via a ManagedClusterSetBinding (see below).
6The PlacementBinding connects the policy with the Placement.
One important difference to Argo CD I maybe should have mentioned earlier: Argo CD can manage objects it knows. It has no concept of "delete an object completely that cannot compare with Git". For example, delete the secret "kubeadmin". ACM is able to do that, you can create a policy that ensures the secret is deleted.

The ManagedClusterSetBinding mentioned in callout 5 looks like the following. It binds the cluster set production to the namespace acm-policies.

apiVersion: cluster.open-cluster-management.io/v1beta2
kind: ManagedClusterSetBinding (1)
metadata:
  name: production
  namespace: acm-policies
spec:
  clusterSet: production
1The ManagedClusterSetBinding binds the cluster set production to the namespace acm-policies.

Object model of ACM together with Argo CD

Even after working with ACM for a while, I still find it difficult to understand the object model or to explain it to others. Therefore, I tried to create a visual representation of the objects and their relationships here, to help you to understand and to remind myself when I forget. :)

How do these objects play together? The easiest way to understand it is to follow a few clusters through the chain. In the following example, four clusters are imported into ACM. Each step narrows the selection down:

  1. Every imported cluster is a ManagedCluster on the hub. The labels are what everything else selects on.

  2. The ManagedClusterSet production groups clusters. dev-eu-01 belongs to another cluster set and is ignored.

  3. The ManagedClusterSetBinding makes the cluster set usable in a namespace, here openshift-gitops. Without this binding, a Placement in that namespace does not find a single cluster, which is the most common reason for an empty selection.

  4. The Placement selects from the bound cluster sets only. It asks for environment=prod, so prod-eu-02 (a test cluster) is ignored as well.

  5. The PlacementDecision is the result: prod-eu-01 and prod-us-01. ACM keeps it up to date when clusters join, leave or change their labels.

Four example clusters are narrowed down step by step: the cluster set drops dev-eu-01
Figure 1. How ACM selects clusters: from ManagedCluster to PlacementDecision

The PlacementDecision itself does not deploy anything. It is just a list and the point where it is handed over to the consumers (Argo CD). Other objects in the same namespace read it and act on it:

  • The GitOpsCluster registers the selected managed clusters automatically in Argo CD by creating a cluster secret for each of them. ACM rotates the tokens automatically, therefore you do not need to take care of that.

  • An ApplicationSet creates one Application per selected cluster and uses these cluster secrets as destinations.

  • For governance, a PlacementBinding connects a Policy with a Placement. ACM then puts a copy of the Policy into the hub namespace of each selected cluster, where it gets picked up.

Clusters that are not part of the decision receive nothing: no Application, no Policy.

GitOpsCluster creates Argo CD cluster secrets
Figure 2. What uses the PlacementDecision
Consumers can only reference a Placement in their own namespace. Policies usually live in a different namespace (for example acm-policies), so they need their own ManagedClusterSetBinding and Placement there, typically with the same selection.

What is the advantage now?

So after all these different objects and relationships, which look rather complex at the beginning, what is the advantage of ACM?

  • No extra credentials per cluster: the managed clusters are already registered with the hub. The policy is pulled by the policy addon on the spoke and evaluated locally. ACM takes care of the tokens automatically.

  • The hub does not need to reach the API of the spoke.

  • If the hub is unreachable, the spoke keeps enforcing the last known policy because it is stored locally. Good for edge clusters with unstable connectivity.

  • A compliance view: "compliance-operator: 37 compliant, 3 non-compliant", with the reason per cluster and a history.

The rule of thumb

  • Applications and everything owned by application teams: → Argo CD.

  • Baseline configuration that must be audited (security settings, mandatory Operators, certificates, "this must never exist"): → ACM Policies.

  • Platform configuration that is complex to render (a full logging stack, a service mesh control plane): → Argo CD, optionally wrapped by an ACM Policy that only checks (inform) the result.

The anti-pattern: two controllers, one object

One thing I have seen more than once: Argo CD manages an object with selfHeal: true, and an ACM policy enforces the same object with a slightly different value. Each controller "fixes" the drift caused by the other one. The object flips back and forth.

Every object must have exactly one owner. If an ACM Policy and Argo CD should both care about an object, let Argo CD own it and use the policy of ACM in inform mode only.

The bootstrap question: who installs Argo CD?

GitOps needs one first imperative step: somebody has to install the OpenShift GitOps Operator and apply the first (root) Application. Everything after that comes from Git. This is also a question where ACM and Argo CD meet:

ApproachGood forWatch out

Script or oc apply

PoC, a handful of clusters, workshops

A manual step per cluster. Version and document the script.

Automation (Ansible, Terraform, …​)

Customers with Automation environment

A second tool to maintain. Keep it to the bootstrap only.

Install-time manifests

Repeatable on-premises installations (agent-based installer)

Must tolerate CRD timing until the operator is up.

ACM Policy

Fleets: every imported cluster is bootstrapped automatically

Policy and Argo CD must not fight over the same objects.

ACM GitOpsCluster

Hub Argo CD (push) or GitOps add-on (pull/agent) targets the cluster

The hub becomes part of the critical path.

My recommendation: fewer than five clusters, use a script or Ansible. For a fleet that is managed by ACM, use a policy or the GitOpsCluster. Never: "we click it in the console once". The bootstrap policy is shown in Ep.20.

Working together: ACM as the brain for Argo CD

As mentioned above, ACM can register its managed clusters in OpenShift GitOps automatically. You no longer maintain cluster secrets by hand: ACM creates them, based on a Placement, rotates the credentials automatically and removes the secrets when a cluster is removed from the Placement.

The bridge between the two worlds is the GitOpsCluster custom resource. On top of it, ACM offers three options to get an Argo CD Application onto a managed cluster:

ModelHow it worksUse it when

Push

Argo CD on the hub pushes manifests to the spoke API, like option 1 above. ACM creates and rotates the cluster secrets.

A moderate number of clusters, good connectivity from hub to spokes, and you want one UI with full visibility.

Pull

Applications are created on the hub but not reconciled there. Instead the target cluster pulls it and a local Argo CD instance on the target cluster reconciles it.

Many clusters, outbound-only networks, or the hub must not hold credentials for the spokes.

Argo CD Agent

ACM deploys the Argo CD Agent to the spokes via the GitOps add-on. Applications are defined on the hub, the agent reconciles locally and synchronises the full status back.

The direction for large fleets and edge sites.

The following diagram shows how the three models fit together.

Push reconciles on the hub
Figure 3. Three ways an Argo CD Application reaches a managed cluster
Yes, you read that right, the version will jump from 2.17 to 5.0 for ACM in the next release, to harmonize with OpenShift versioning.

What each model actually is, how the data flows in which direction and how to set it up step by step is the topic of the next part: [Ep.19] ACM and OpenShift GitOps: push, pull and the Argo CD Agent.


Decision guide

The following are the scenarios I saw in real projects and what I personally would recommend. Verify if these fit your situation.

A quick summary of the scenarios is shown in the following slide I was using for a workshop (more details below):

Three questions lead to the scenarios A to E
Figure 4. Decision guide: OpenShift GitOps alone or together with ACM

Scenario A: one to three clusters, one platform team

Example: a single production cluster plus a test cluster.

OpenShift GitOps only. ACM would add a hub, an additional component to operate and additional cost, without solving a real problem. Configure the clusters with Argo CD as described in the earlier episodes of this series, with either one Argo CD instance per cluster or a central Argo CD on a management cluster (recommended). If you need compliance reports, the Compliance Operator gives you scan results per cluster without ACM.

Scenario B: 3 to 20 clusters, all reachable, created by another team or process

Example: dev, test, prod in two data centres, installed by the infrastructure team.

OpenShift GitOps is usually sufficient. A central Argo CD with an ApplicationSet cluster generator, or decentralised Argo CD instances per cluster. Start thinking about ACM when one of these questions comes up:

  • "Who rotates the cluster tokens in the Argo CD secrets?"

  • "Can you show us which clusters are not compliant?"

  • "Can we have one search, one observability view across all clusters?"

Scenario C: 20 to several hundred clusters

Example: clusters per business unit, factory or per tenant, created on demand.

ACM plus OpenShift GitOps. Push for smaller fleets, pull or agent for larger fleets. ACM creates the clusters, labels them and adds them to cluster sets. The GitOpsCluster registers them in Argo CD automatically. Applications and complex platform components come via ApplicationSets driven by ACM Placements. A new cluster receives its full configuration without anybody touching Argo CD. Around 20 clusters is also where a single push hub usually needs sharding, and where the "hub holds all the credentials" discussion with the security team starts. The pull model or the Argo CD Agent become serious options here.

Scenario D: regulated environment

Example: banking or public sector, regular audits, a defined security baseline.

ACM Policies for the baseline, Argo CD for the rest. The baseline lives as policies in Git, deployed by Argo CD to the hub, enforced and reported by ACM. Start in inform mode on existing clusters, check the compliance dashboard, then switch to enforce.

Scenario E: edge and disconnected sites

Example: hundreds of small clusters (SNO, compact clusters) in stores, factories or vehicles with unstable connectivity.

ACM plus OpenShift GitOps in pull or agent mode. The push model fails here: the hub cannot reliably reach the spokes. Policies are evaluated locally and continue to work when the connection is down. Applications are reconciled by a local Argo CD or the Argo CD Agent. Two things to keep in mind: every spoke still needs to reach its sources (Git or an OCI registry, often a local mirror), and every cluster that runs the Argo CD Agent needs an OpenShift Platform Plus subscription.

Frequently asked questions

Just some questions I heard from customers during the last years:

Can ACM replace Argo CD? For applications, no. ACM’s application lifecycle is built on OpenShift GitOps. The older Channel and Subscription model still exists, but it is deprecated. I would not start a new project with Channels and Subscriptions.

Can Argo CD replace ACM? For delivery to existing, reachable clusters, yes. For cluster lifecycle, compliance reporting, fleet search and observability, no.

Do we need ACM to do multi-cluster GitOps? No. Argo CD handles multiple clusters on its own, and with the Argo CD Agent even in a hub-and-spoke model with local reconciliation (see Ep.15). ACM makes it easier to operate at scale, because the cluster registration, the token rotation and the agent installation are automated.

Should the ACM hub itself be managed by Argo CD? Yes. The MultiClusterHub, the policies, Placements, cluster sets, GitOpsCluster resources: all of it belongs in Git. The hub is just another cluster.

Should I configure the cluster with Argo CD or with policies? See the rule of thumb above. And whatever you choose always keep in mind: one owner per object.

Summary

So, which one is better? Neither. They are not competitors, they are layers.

  • OpenShift GitOps is the delivery engine. It makes sure that what is in Git is running on a cluster. It is included in OpenShift and should be your default for applications and complex platform components, no matter how many clusters you have.

  • Advanced Cluster Management is the fleet layer on top. It knows which clusters exist, creates and upgrades them, enforces and reports compliance, and feeds Argo CD with the list of clusters via Placements.

  • The real decision is not "GitOps or ACM", but "do I need fleet management?" For a handful of clusters, usually not. For many clusters, dynamic cluster creation, regulated environments or edge sites, almost certainly yes.

  • Where both tools can do the same job, cluster configuration, use policies for the audited baseline and Argo CD for everything that needs rendering, and keep exactly one owner per object.

In the next part, we look under the hood of the integration: push, pull and the Argo CD Agent.

And as always: If it is not in Git, it does not exist.

The diagrams in this article have been manually drawn on a whiteboard. AI then generated the SVGs from photos and made them look equal in all articles.

Discussion

← Previous
Use arrow keys to navigate
Next →
←
→