When managing one or more clusters, the question arises as to how cluster configurations and applications can be installed securely, regularly, and in the same way.
This is where the so-called GitOps approach helps, according to the mantra: "If it is not in Git, it does not exist".
The idea is to have Git as the only source of truth on what happens inside the environment. While there are many articles about how to get GitOps into the deployment process of applications, this series of articles tries to set the focus on the cluster configuration and tasks system administrators usually have to do, for example: Setup an Operator.
In this series, I will focus on cluster configuration.
The GitOps approach is a very common practice and the-facto "standard" as of today.
When I write standard, then be assured, that the approach itself should be followed, but HOW this is done can be a topic of many tough discussions.
But what is it and why should a company invest time to follow this approach?
_GitOps is a declarative way to implement continuous deployment for cloud-native applications. It should be a repeatable process to manage multiple clusters.
GitOps adds the following features to company processes:
Everything as code: The entire state of the application, infrastructure and configuration is declaratively defined as code.
Git and the single source of truth: Every setting and every manifest is stored and versioned in Git. Any change must first be saved to Git.
Operations via Git workflows: Standard Git procedures, such as pull or merge requests, should be used to track any changes to the applications or cluster configurations.
It is important that not only the manifests of the applications but also the cluster configuration is stored in Git. The goal should be to ensure that no manual changes are made directly to the cluster.
Benefits/Challenges
Deploying new versions of applications or cluster configurations with a high degree of confidence is a desirable goal as getting features reliably to production is one of the most important characteristics of fast-moving organisations.
GitOps is a set of common practices where the entire code delivery process is controlled via Git, including infrastructure and application definition as code and automation to complete updates and rollbacks.
GitOps constantly watches for changes in Git repositories and compares them with the current state of the cluster. If there is a drift it will either automatically synchronise to the wanted state or warn accordingly (manual sync must then be performed).
The key GitOps advantages are:
Cluster and application configuration versioned in Git
Visualisation of desired system state
Automatically syncs configuration from Git to clusters (if enabled)
Drift detection, visualisation, and correction
Rollback and roll-forward to any Git commit.
Manifest templating support (Helm, Kustomize, etc.)
Visual insight into sync status and history.
Role-Based access support
Pipeline integration
Adopting GitOps has enormous benefits but does pose some challenges. Many teams will have to adjust their culture and way of working to support using Git as the single source of truth. Strictly adhering to GitOps processes will mean all changes will be committed. This may present a challenge when it comes to debugging a live environment. There may be times when that is necessary and will require suspending GitOps in some way.
Some other prerequisites for adopting GitOps include
Good testing and CI processes are in place.
A strategy for dealing with promotions between environments.
Strategy for Secrets management.
Used Tools
The following list of tools (or specifications) are used for our GitOps Approach.
I was recently asked about how to use Keycloak as an authentication provider for OpenShift. How to install Keycloak using the Operator and how to configure Keycloak and OpenShift so that users can log in to OpenShift using OpenID. I have to admit that the exact steps are not easy to find, so I decided to write a blog post about it, describing each step in detail. This time I will not use GitOps, but the OpenShift and Keycloak Web Console to show the steps, because before we put it into GitOps, we need to understand what is actually happening.
This article tries to explain every step required so that a user can authenticate to OpenShift using Keycloak as an Identity Provider (IDP) and that Groups from Keycloak are imported into OpenShift. This article does not cover a production grade installation of Keycloak, but only a test installation, so you can see how it works. For production, you might want to consider a proper database (maybe external, but at least with a backup), high availability, etc.).
During my day-to-day business, I am discussing the following setup with many customers: Configure App-of-Apps. Here I try to explain how I use an ApplicationSet that watches over a folder in Git and automatically adds a new Argo CD Application whenever a new folder is found. This works great, but there is a catch: The ApplicationSet uses the same Namespace default for all Applications. This is not always desired, especially when you have different teams working on different Applications.
Recently I was asked by the customer if this can be fixed and if it is possible to define different Namespaces for each Application. The answer is yes, and I would like to show you how to do this.
Classic Kubernetes/OpenShift offer a feature called NetworkPolicy that allows users to control the traffic to and from their assigned Namespace. NetworkPolicies are designed to give project owners or tenants the ability to protect their own namespace. Sometimes, however, I worked with customers where the cluster administrators or a dedicated (network) team need to enforce these policies.
Since the NetworkPolicy API is namespace-scoped, it is not possible to enforce policies across namespaces. The only solution was to create custom (project) admin and edit roles, and remove the ability of creating, modifying or deleting NetworkPolicy objects. Technically, this is possible and easily done. But shifts the whole network security to cluster administrators.
Luckily, this is where AdminNetworkPolicy (ANP) and BaselineAdminNetworkPolicy (BANP) comes into play.
Lately I came across several issues where a given Helm Chart must be modified after it has been rendered by Argo CD. Argo CD does a helm template to render a Chart. Sometimes, especially when you work with Subcharts or when a specific setting is not yet supported by the Chart, you need to modify it later … you need to post-render the Chart.
In this very short article, I would like to demonstrate this on a real-live example I had to do. I would like to inject annotations to a Route objects, so that the certificate can be injected. This is done by the cert-utils operator. For the post-rendering the Argo CD repo pod will be extended with a sidecar container, that is watching for the repos and patches them if required.
The article SSL Certificate Management for OpenShift on AWS explains how to use the Cert-Manager Operator to request and install a new SSL Certificate. This time, I would like to leverage the GitOps approach using the Helm Chart cert-manager I have prepared to deploy the Operator and order new Certificates.
I will use an ACME Letsencrypt issuer with a DNS challenge. My domain is hosted at AWS Route 53.
However, any other integration can be easily used.
Thomas Jungbauer
( Lastmod: 2024-11-08 )
-
3 min read