[Ep.15] OpenShift GitOps - Argo CD Agent

In a centralized deployment, all changes are applied to a single, central Argo CD instance, often installed on a management cluster. This is the traditional way of deploying Argo CD, at least in my opinion. With this model, you have a single pane of glass to manage all your clusters. You have a single UI and will see all your clusters in one place, which makes this model very convenient when you have multiple clusters. However, the scalability of this model is limited. Organizations with a huge number of clusters or Argo CD applications would hit some boundaries at some point. A sharding configuration would help, but only to a certain extent. The performance would degrade significantly. In addition, this model creates a Single Point of Failure. If this instance is down, the company loses the ability to manage their clusters through Argo CD.

In a decentralized deployment, multiple instances of Argo CD, often one for each cluster, are installed. With this approach, the issue with scalability is solved. Moreover, the Single Point of Failure is eliminated as well, since a broken instance will not affect the other instances. However, the disadvantages of this model are that the complexity for the operational teams will increase, since they need to manage multiple instances now. Also, the single pane of glass for management is lost, as there are multiple UIs to manage.

The Argo CD Agent supports two modes of operation: Managed and Autonomous. The mode determines where the authoritative source of truth for the Application .spec field resides.

The Argo CD Agent uses mTLS certificates to communicate between the hub and the spoke clusters.

To create some kind of real customer scenario, I have created two clusters:

The first one I have installed on AWS, and the second one is a Bare Metal Single node cluster. Both can reach each other via the Internet.

Before we start with the installation of the Principal or Agent component, we need to ensure that the following prerequisites are met:

The OpenShift GitOps Operator installs by default an Argo CD instance. In this test we will disable this, as we do not need that instance. Moreover and even more important, we need to tell the Operator for which namespaces it should feel responsible for. In this case, we will tell the Operator to be responsible for all namespaces on the cluster.

We need to modify the Subscription openshift-gitops-operator and add the following environment variables:

To activate the Principal components we first need a cluster (the Hub) where OpenShift GitOps is installed. On this cluster, there might be a running instance of Argo CD already. However, at this time an existing Argo CD cannot be used. Instead, a new Argo CD instance must be created. (This is because the controller must not be activated)

In my case, I will install the Principal component in the namespace argocd-principal.

To create an Argo CD instance we need to create the following configuration:

This will start a Pod called hub-gitops-agent-principal in the namespace argocd-principal. However, this pod will fail at this moment and that is fine.

At this point the Operator created the Route object already:

If you configured the AppProject with sourceNamespaces, you need to add the following to the AppProject (for example to the default AppProject). This must match exactly the namespaces you have created for the Agent.

You can also use this patch command:

Restart the Argo CD Pods to apply the changes.

To create the required secrets, we need to download the argocd-agentctl tool.

This can be found at: https://developers.redhat.com/content-gateway/rest/browse/pub/cgw/openshift-gitops/

Download and install it for your platform.

The following steps will create the required secrets for the Principal component. In this example, we will create our own CA and certificates. This is suitable for development and testing purposes. For production environments, you should use certificates issued by your organization’s PKI or a trusted certificate authority.

Now the principal pod should be running successfully.

In the logs you should see the following:

This concludes the configuration of the Principal component. There are a lot of steps to create the required secrets, but this is only done once. A GitOps-friendly way to achieve this might be done using a Kubernetes Job (if you consider this as GitOps-friendly…​ which I do).

We first need to create an agent on the Principal cluster.

This will create the secret cluster-argocd-agent-bm01 with the label argocd.argoproj.io/secret-type: cluster in the Argo CD namespace.

Be sure that the target namespace on the agent or workload cluster exists. If not, create it first.

To copy the CA certificate from the principal to the agent cluster the following command is used:

This will copy the CA certificate from the Principal cluster to the Agent cluster into the namespace and secret argocd-agent-bm01/argocd-agent-ca.

Now we need to create, based on the imported CA, a client certificate for the Agent. This is done by the following command:

This will create the secret argocd-agent-client-tls on the workload cluster, containing a certificate and a key, signed by the CA certificate imported from the Principal cluster.

The OpenShift GitOps Operator installs by default an Argo CD instance. In this test we will disable this, as we do not need that instance. Moreover and even more important, we need to tell the Operator for which namespaces it should feel responsible for. In this case, we will tell the Operator to be responsible for all namespaces on the cluster.

We need to modify the Subscription openshift-gitops-operator and add the following environment variables:

To create a minimalistic Argo CD instance on the Agent cluster, we can use the following Argo CD configuration:

This will create a lightweight Argo CD instance in the namespace argocd-agent-bm01.

Before we start with the actual installation of the Agent, we need to ensure that the Redis instance on the spoke cluster is accessible for the Agent. We need to create a NetworkPolicy accordingly:

Apply the NetworkPolicy to the spoke cluster:

Add the Helm repository:

Install the agent in the managed mode using the Helm Chart. The following parameters are used:

With this chart a pod on the spoke cluster will be created and will start to synchronize the Argo CD applications between the hub and the spoke cluster.

To verify the Agent in managed mode we need to create an Argo CD Application on the hub cluster. We can try the following Application. The Application is taken from the openshift-clusterconfig-gitops repository and simply adds a banner to the top of the OpenShift UI. I typically use this as a quick test if GitOps is working.

Apply the Application to the hub cluster:

Since the application will try to automatically synchronize the configuration, the status will change to Synced after a few seconds:

and the (top) banner will be visible on the UI:

On the hub cluster, the Argo CD Application will be Synced:

Let’s cleanup the first installation of the Chart (managed agent) in order to not have any conflicts.

Install the agent in the autonomous mode using the Helm Chart. The following parameters are used:

The only difference to the managed mode is the agentMode parameter.

With this chart a pod on the spoke cluster will be created and will start to synchronize the Argo CD applications between the hub and the spoke cluster.

To verify the Agent in autonomous mode we need to create an Argo CD Application on the spoke cluster.

We can try the following Application. It is basically the same as we used for the test for the managed mode, except that this time we will add a banner on the bottom of the UI.

Apply the Application to the spoke cluster:

Like the managed mode, the status will change to Synced after a few seconds and the (this time) bottom banner will be visible on the UI:

Moreover, the Argo CD Application will appear on the hub cluster:

If the Principal pod fails to start with errors about missing secrets, verify that all required secrets have been created:

You should see the following secrets:

If any of these are missing, re-run the corresponding argocd-agentctl command to create them.

When you see errors like the following in the logs of the Principal pod, ensure that the Argo CD instance does not have the controller enabled in that namespace (set spec.controller.enabled: false). Hopefully, this will change in the future.

Currently, the Principal component cannot be installed alongside an existing Argo CD instance where the application controller is enabled. You must create a separate Argo CD instance with the controller disabled (spec.controller.enabled: false). To me, this is one of the biggest limitations. However, this will be addressed in the future and is tracked in the issue: https://github.com/argoproj-labs/argocd-agent/issues/708

The mTLS certificates must be created and managed manually by the user. There is no automatic certificate rotation or renewal. For production environments, you should integrate with your organization’s PKI infrastructure and implement a certificate rotation strategy.