Quay Deployment and Configuration using GitOps

OpenShift GitOps

- Thomas Jungbauer Thomas Jungbauer ( Lastmod: 2024-05-08 ) - 4 min read

Installing and configuring Quay Enterprise using a GitOps approach is not as easy as it sounds. On the one hand, the operator is deployed easily, on the other hand, the configuration of Quay is quite tough to do in a declarative way and syntax rules must be strictly followed.

In this article, I am trying to explain how I solved this issue by using a Kubernetes Job and a Helm Chart.

What about Quay configuration?

Quay Enterprise is using a (quite big) Secrets object that defines tons of settings for the registry. The syntax must strictly be followed. For example, a Boolean must be true or false. Quay ignores if a string like "true" or "false" is provided.

This alone is already a hassle since working with Booleans in Helm is not as easy as you might think.

The Secret combines non-sensitive data (like DEFAULT_TAG_EXPIRATION) with sensitive data (like settings for the Object Store)

If there is any error in the configuration file, or if the Operator is configured to manage a specific component, but finds settings for this component in the Secret, the deployment will fail.

The solution?

I thought for a long time about how to solve this issue. One solution might be to create the whole Secret upfront and simply provide it during the deployment. This works and I have done this previously, but I wanted to generate the Secret during the deployment.

Therefore, I am now trying to create a ConfigMap that holds a complete skeleton of the required Secret. This ConfigMap is used by a Kubernetes Job, which reads the required sensitive information out of other existing Secrets (such as S3 information) and generates a quay-secret by replacing the required fields.

Is this the perfect and optimal way to do that? Probably not, however, it works :)

Let us see that in action

Prerequisites

First, we have some prerequisites.

  1. Quay is very …​ very hungry for resources. Quay application pods require 8 CPU and 16GB Memory per pod and per default…​ and it tries to spin up 2 pods. The same goes for Clair and so on. Therefore, I will configure Quay to only use 1 replica for these services.

  2. Quay requires node roles infra. I am not sure if this is new or if I never saw that, but the nodeSelector, which it seems you cannot configure, is looking for the label infra. Therefore, the nodes that should host Quay must have the label: node-role.kubernetes.io/infra: ''

  3. Currently, I am using a BucketClaim object to create an S3-bucket and once created I read the required information of the bucket to replace the settings in the generated Quay configuration accordingly. The BucketClaim object comes from the OpenShift Data Foundation. I installed the Multicloud Object Gateway only, which allows me to provide Object Storage (S3). (Very useful to test other solutions too, like OpenShift Logging or Network Observability)

Meeting these requirements, allows us to deploy Quay. But first some theory.

Deployment Workflow

The workflow of the deployment is as the following image demonstrates

Argo CD Syncwaves
Figure 1. Argo CD: Syncwaves

  1. Everything starts with the Helm Chart setup-quay. This Chart itself provides the logic to create an S3-bucket, a Job to generate the configuration and the creation of a Secret, that provides the initial administrator credential.

  2. setup-quay has multiple dependencies to other Helm charts, that can be found at my Helm repository:

    1. quay-registry-setup

    2. helper-operator

    3. helper-status-checker

Configuration

All values for the Helm Charts can be found in the values file

Since sub-charts are used the file is divided into 4 blocks:

  1. helper-operator: Here the sub-chart helper-operator is configured. All settings here are bypassed to the subchart. It defines the required settings to deploy the operator.

  2. helper-status-checker: Here settings for the status-checker Job are defined.

  3. quay-registry-setup: This defines the actual configuration of the QuayEnterprise object. It will spin up the Quay instance. Some components are set to "false" or "replica == 1" to minimize the required resources in my lab. For a production environment, additional replicas or components might be required.

  4. quay: These are the actual values for the configuration. It defines the bucketClaim as well as settings for the Quay configuration that might be overwritten.

Deploy it

Using for example the following Argo CD Application we can deploy everything.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: in-cluster-setup-quay
  namespace: openshift-gitops
spec:
  destination:
    name: in-cluster (1)
    namespace: default
  ignoreDifferences: (2)
    - jsonPointers:
        - /data/password
      kind: Secret
      name: init-user
      namespace: quay-enterprise
  info:
    - name: Description
      value: ApplicationSet that Deploys on Management Cluster (Matrix Generator)
  project: in-cluster
  source:
    path: clusters/management-cluster/setup-quay (3)
    repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops'
    targetRevision: main
1The destination cluster. Here the "in-cluster" means the local cluster of Argo CD.
2The initial credential for Quay is being generated and would change if the Argo CD application gets refreshed and therefore it would be out of sync. So, we are ignoring differences in the password field.
3The source of the Helm Chart.

In Argo CD this Application will look like

Quay in Argo CD
Figure 2. Quay in Argo CD

Deployment means in GitOps approach: synchronizing the Argo CD Application.

This will install the Operator and spin up all required Pods and Jobs. It will take several minutes until everything is up and running. During the deployment, some Pods may fail and will get restarted automatically. This happens because they are dependent on the Postgres DB which must be started first.

Quay is Alive

Congratulations, you have now a Quay instance. Use the auto-generated credentials, that are stored in the Secret init-user to authenticate.

Quay Login
Figure 3. Quay Login

Is that All - Kind of Summary?

Several configurations are done here now. However, there are tons to follow. For example, log forwarding or additional certificates. Some settings will contain sensitive information some will not. All these settings can be added to the ConfigMap skeleton and be replaced accordingly with "little" effort. For me, it is simply not possible to test every setting and possibility. Maybe I will extend the Helm Chart during the journey. If you find this useful, feel free to re-use it and of course, if you find any issues feel free to create a GitHub issue.

See Also:

  • Introducing AdminNetworkPolicies –

    Classic Kubernetes/OpenShift offer a feature called NetworkPolicy that allows users to control the traffic to and from their assigned Namespace. NetworkPolicies are designed to give project owners or tenants the ability to protect their own namespace. Sometimes, however, I worked with customers where the cluster administrators or a dedicated (network) team need to enforce these policies. Since the NetworkPolicy API is namespace-scoped, it is not possible to enforce policies across namespaces. The only solution was to create custom (project) admin and edit roles, and remove the ability of creating, modifying or deleting NetworkPolicy objects. Technically, this is possible and easily done. But shifts the whole network security to cluster administrators. Luckily, this is where AdminNetworkPolicy (ANP) and BaselineAdminNetworkPolicy (BANP) comes into play.

  • Using Kustomize to post render a Helm Chart –

    Lately I came across several issues where a given Helm Chart must be modified after it has been rendered by Argo CD. Argo CD does a helm template to render a Chart. Sometimes, especially when you work with Subcharts or when a specific setting is not yet supported by the Chart, you need to modify it later …​ you need to post-render the Chart. In this very short article, I would like to demonstrate this on a real-live example I had to do. I would like to inject annotations to a Route objects, so that the certificate can be injected. This is done by the cert-utils operator. For the post-rendering the Argo CD repo pod will be extended with a sidecar container, that is watching for the repos and patches them if required.

  • Managing Certificates using GitOps approach –

    The article SSL Certificate Management for OpenShift on AWS explains how to use the Cert-Manager Operator to request and install a new SSL Certificate. This time, I would like to leverage the GitOps approach using the Helm Chart cert-manager I have prepared to deploy the Operator and order new Certificates. I will use an ACME Letsencrypt issuer with a DNS challenge. My domain is hosted at AWS Route 53. However, any other integration can be easily used.

  • Update Cluster Version using GitOps approach –

    During a GitOps journey at one point, the question arises, how to update a cluster? Nowadays it is very easy to update a cluster using CLI or WebUI, so why bother with GitOps in that case? The reason is simple: Using GitOps you can be sure that all clusters are updated to the correct, required version and the version of each cluster is also managed in Git. All you need is the channel you want to use and the desired cluster version. Optionally, you can define the exact image SHA. This might be required when you are operating in a restricted environment.

  • Multiple Sources for Applications in Argo CD –

    Argo CD or OpenShift GitOps uses Applications or ApplicationSets to define the relationship between a source (Git) and a cluster. Typically, this is a 1:1 link, which means one Application is using one source to compare the cluster status. This can be a limitation. For example, if you are working with Helm Charts and a Helm repository, you do not want to re-build (or re-release) the whole chart just because you made a small change in the values file that is packaged into the repository. You want to separate the configuration of the chart with the Helm package. The most common scenarios for multiple sources are (see: Argo CD documentation): Your organization wants to use an external/public Helm chart You want to override the Helm values with your own local values You don’t want to clone the Helm chart locally as well because that would lead to duplication and you would need to monitor it manually for upstream changes. This small article describes three different ways with a working example and tries to cover the advantages and disadvantages of each of them. They might be opinionated but some of them proved to be easier to use and manage.

Copyright © 2020 - 2024 Toni Schmidbauer & Thomas Jungbauer