[Ep.20] Managing ACM Policies with OpenShift GitOps

In Ep.19, the default instance openshift-gitops became the principal of the Argo CD Agent. Its application-controller is disabled, so it cannot sync anything to the hub itself, and the manifests of agent Applications are rendered on the managed clusters anyway. The policies, however, must be created on the hub.

This article therefore uses a second, small Argo CD instance on the hub that only takes care of the hub configuration: policy-gitops in the namespace of the same name. If your default instance still has its application-controller (push or pull model), you can use it instead. Replace policy-gitops with openshift-gitops in the examples below.

The application-controller of the instance cannot create policies and Placements. Create a ClusterRole and bind it to the controller’s service account:

The ClusterRole in the ACM documentation also contains placementrules. I left them out on purpose: PlacementRules are deprecated.

Creating the policy YAML by hand is tough and error-prone. Every manifest has to be wrapped in a ConfigurationPolicy, which is wrapped in a Policy, which needs a Placement and a PlacementBinding. The PolicyGenerator solves this. It is a Kustomize generator plugin that takes plain manifests and wraps them for you.

The plugin is not part of the OpenShift GitOps images. An init container copies it from an ACM image into the repo-server, and Kustomize must be allowed to run alpha plugins.

The setup is described in the ACM documentation: ACM 2.17: Accessing the Policy Generator from OpenShift GitOps.

If your policy folders contain Helm charts that should be rendered before the policies are generated, add the environment variable POLICY_GEN_ENABLE_HELM with the value "true" to spec.repo.env.

By default, Argo CD considers a policy healthy as soon as it exists. That is not very useful: you want to see in Argo CD whether the policy is compliant. The policy collection of the Open Cluster Management project contains a policy that adds the matching health checks (resourceHealthChecks) to the ArgoCD resource: argocd-policy-healthchecks.yaml. The policy adds the health checks to every ArgoCD instance it finds on the cluster, so policy-gitops is covered as well. It is shipped with remediationAction: inform: set it to enforce, otherwise it only reports the missing health checks. After that, Argo CD shows a non-compliant policy as Degraded.

The files in input/ are plain Kubernetes manifests, exactly as you would apply them with oc apply. The Operator is installed in the namespace openshift-operators, which already has an OperatorGroup for all namespaces, so neither a namespace nor an OperatorGroup is needed for it:

The control plane consists of two cluster-scoped resources, IstioCNI and Istio, each pointing to its own namespace (istio-cni and istio-system):

And the kubeadmin secret that must not exist:

The kustomization.yaml only references the generator:

Render the folder locally before you push. If it does not render on your machine, it will not render in Argo CD either. With the PolicyGenerator binary in your Kustomize plugin path:

The output contains a Policy, a Placement and a PlacementBinding per policy.

The file policy-generator.yaml is the heart of the setup. It does not contain any policy itself, it only describes which policies the generator shall create from the manifests in input/. It has two parts:

For every entry in policies, the generator creates a Policy, a Placement and a PlacementBinding. The manifests are wrapped into a ConfigurationPolicy inside the Policy: by default all manifests of a policy into one, with consolidateManifests: false one per manifest. Our two policies use this as follows: service-mesh overrides the defaults to enforce and to apply its manifests in order, while no-kubeadmin keeps the defaults and only changes the compliance type.

A Placement only sees the clusters of cluster sets that are bound to its own namespace. The bindings from Ep.19 live in openshift-gitops, so the Placements in acm-policies would find no cluster at all. ACM then shows the warning "ClusterSets failed to load. Verify that there is at least one ClusterSet bound to your selected namespace."

Bind the cluster set production to acm-policies, and global as well if you use the policy that keeps the PolicyGenerator image in sync (its Placement selects the hub from global):

The generated Placements select sno and bm by name, from the cluster sets bound to acm-policies. Check the result:

On the hub, one Application syncs the folder:

When ACM propagates a policy, it creates a copy in the namespace of every managed cluster on the hub, the replicated policy. Argo CD could treat these copies as extraneous resources of the Application and show it as OutOfSync or even try to delete them. ACM prevents that: it automatically sets the annotation argocd.argoproj.io/compare-options: IgnoreExtraneous on every replicated policy.