Articles by Toni Schmidbauer on TechBlog about OpenShift/Ansible/Satellite and much more

Onboarding to Ansible Automation Platform with Configuration as Code

Wed, 04 Mar 2026 00:00:00 +0000

We had the honor of presenting at the 2nd Ansible Anwendertreffen in Austria. The topic was the onboarding of application teams to the Ansible Automation Platform.

We created an extensive demo that:

Onboards a new tenant into an AAP organization.
Each tenant gets a configuration as code repository, cloned from a template.
A push event to the config-as-code repository triggers an update of AAP objects.
Provides an example repository for each tenant to get started.
The example repository provides a webhook that creates an OpenShift Virtualization VM for testing code changes when a feature branch is created.
Contains a playbook to display objects that are not currently managed by the tenant’s configuration-as-code repository.

The source code for the demo is available on GitHub.

The README contains more details about the implementation, and the slide deck is also available on GitHub.

OKD Single node installation in a disconnected environment

Mon, 09 Feb 2026 00:00:00 +0000

Because of reasons we had to set up a disconnected single node OKD "cluster". This is our brain dump as the OKD documentation sometimes refers to OpenShift image locations, we had to read multiple sections to get it working and we also consulted the OpenShift documentation at https://docs.redhat.com.

Updated on 2026-03-03: Fixed install-config.yaml and create manifest command

First we need to download the oc command in the version we would like to install. OKD provides it on its release page:

https://github.com/okd-project/okd/releases/download/4.21.0-okd-scos.3/openshift-client-linux-amd64-rhel9-4.21.0-okd-scos.3.tar.gz

To get an overview of releases available for OKD see https://amd64.origin.releases.ci.openshift.org/.

Another option is to extract the oc command from the release image, if you have a version of oc already installed:

oc adm release extract --tools quay.io/okd/scos-release:4.21.0-okd-scos.3

This will also extract the openshift-install tar.gz which we need later to generate the installation manifests and ignition configs.

Next we need the oc-mirror plugin, which is used to mirror the release content to our local registry. We were only able to find the plugin on console.redhat.com. Another option might be to compile the plugin from source code (https://github.com/openshift/oc-mirror/).

We copied the oc command and the oc-mirror plugin to /usr/local/bin/ and made them executable. After that we ran oc mirror --v2 --help to test the installation.

The upstream oc-mirror plugin documentation was also helpful.

As our private registry required authentication, we created a .dockerconfigjson file with the registry credentials:

{
"auths": {
"internal.registry": {
"auth": "<credentials base64 encoded>",
"email": "you@example.com"
}
}
}

You can encode the credentials with:

echo -n "username:password" | base64 -w0

For mirroring all required OKD images to our private registry we created an ImageSetConfiguration:

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
platform:
channels:
- name: 4-scos-stable
minVersion: 4.21.0-okd-scos.3
maxVersion: 4.21.0-okd-scos.3
graph: false
operators:
- registry: quay.io/okderators/catalog-index:testing-4.20 (1)
packages:
- name: aws-load-balancer-operator
- name: 3scale-operator
- name: node-observability-operator
additionalImages:
- name: registry.redhat.io/ubi8/ubi:latest
- name: registry.redhat.io/ubi9/ubi@sha256:20f695d2a91352d4eaa25107535126727b5945bff38ed36a3e59590f495046f0

We used a minimal configuration without operators and additional images, because downloading additional images did not work for us:

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
platform:
channels:
- name: 4-scos-stable (1)
type: okd
minVersion: 4.21.0-okd-scos.3 (2)
maxVersion: 4.21.0-okd-scos.3
graph: false
operators: []
additionalImages: []

The hard part for us was finding the right values for channels and the min/max versions. Once again the OKD release status page was helpful.

Our private registry is based on Harbor for our experiments, but any Docker v2 compatible registry should work. The only thing required is a project within Harbor called openshift and a user/password for pulling/pushing images to this project. The project name might be configurable with oc mirror, but this requires further investigation.

oc mirror wants to verify signatures so we had to download a public key and provide the OCP_SIGNATURE_URL:

curl -LO https://raw.githubusercontent.com/openshift/cluster-update-keys/master/keys/verifier-public-key-openshift-ci-4
export OCP_SIGNATURE_URL="https://storage.googleapis.com/openshift-ci-release/releases/signatures/openshift/release/"
export OCP_SIGNATURE_VERIFICATION_PK="verifier-public-key-openshift-ci-4"
# for debugging add --log-level debug, but we had intermittent errors with this option.
# after removing --log-level debug oc mirror ran without problems
oc mirror -c ImageSetConfiguration.yaml --authfile docker-auth.json --workspace file:///workspace docker://internal.registry --v2

This will create the ImageDigestMirrorSource in /workspace/working-dir/cluster-resources/idms-oc-mirror.yaml and ImageTagMirrorSource custom resources in /workspace/working-dir/cluster-resources/itms-oc-mirror.yaml.

The content of ImageDigestMirrorSource will be reused in the install-config.yaml. This is required to redirect image pulls done by the node from quay.io to our private registry.

Next we need DNS records for our cluster:

api.sno.internal
api-int.sno.internal
*.apps.sno.internal (wildcard DNS entry for applications deployed on the cluster)

Now we prepared the required install-config.yaml file to trigger the openshift-install command. We basically followed Installing a Single Node OpenShift Cluster.

apiVersion: v1
baseDomain: internal
compute:
- hyperthreading: Enabled
name: worker
replicas: 0 (1)
controlPlane:
hyperthreading: Enabled
name: master
replicas: 1 (2)
metadata:
name: okd-sno
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
networkType: OVNKubernetes
serviceNetwork:
- 172.30.0.0/16
platform:
none: {}
fips: false
sshKey: 'the key'
bootstrapInPlace: (3)
installationDisk: /dev/vda
pullSecret: |
{
"auths": {
"registry.internal": {
"auth": "<username:password base64 encoded>",
"email": "some@email"
}
}
}
additionalTrustBundle: | (4)
-----BEGIN CERTIFICATE-----
ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ
-----END CERTIFICATE-----
ImageDigestSources: (5)
- mirrors:
- registry.internal/openshift/release
source: quay.io/okd/scos-content
- mirrors:
- registry.internal/openshift/release-images
source: quay.io/okd/scos-release

1	We set the number of workers to zero (0). Because this is a SNO installation. Worker nodes could be added later, even for a SNO cluster
2	We set the number of master nodes to one (1)
3	This tells the installer that we do not have a bootstrap node and it should create one large ignition config file
4	If required the CA certificate of our registry. This is required if the registry uses an internal certificate
5	ImageDigestSources configures CRIO to redirect requests to quay.io to our private registry.

It’s time to create the installation manifests and ignition configs. We created a directory install, copied the install-config.yaml into this directory and triggered the openshift-install command:

$ mkdir install
$ cp install-config.yaml install/
$ openshift-install --dir=install create single-node-ignition-config

We are now ready to download the installation ISO and to embed our ignition config into it. We also want to set kernel boot arguments to configure the network interface with a static IP address.

curl -L $( ./openshift-install coreos print-stream-json |jq -r ".architectures.x86_64.artifacts.metal.formats.iso.disk.location" ) -o fhcos-live.iso

The next step is to modify the ISO. We basically followed the instructions in the OKD documentation: https://docs.okd.io/4.14/installing/installing_sno/install-sno-installing-sno.html#generating-the-install-iso-manually_install-sno-installing-sno-with-the-assisted-installer

alias coreos-installer='podman run --privileged --pull always --rm -v /dev:/dev -v /run/udev:/run/udev -v $PWD:/data -w /data quay.io/coreos/coreos-installer:release'
coreos-installer iso ignition embed -fi install/bootstrap-in-place-for-live-iso.ign fcos-live.iso
coreos-installer iso kargs modify --append 'ip=10.0.0.99::10.0.0.1:255.255.255.0:sno.internal:ens33:none:10.0.0.255' fcos-live.iso

We can take a look at the ISO bootstrap config and the kernel arguments with:

coreos-installer iso ignition show fcos-live.iso
coreos-installer iso kargs show fcos-live.iso

For understanding the kernel boot arguments see https://access.redhat.com/solutions/5499911.

The final step is to boot the modified ISO on the target machine and wait for the installation to complete.

What's new in OpenShift, 4.20 Edition

Mon, 10 Nov 2025 00:00:00 +0000

This article covers news and updates in the OpenShift 4.20 release. We focus on points that got our attention, but this is not a complete summary of the release notes.

Configuring a local arbiter node

Configuring a local arbiter node describes how to configure an OCP cluster with only two control plane (ETCD) nodes. Might be useful in a pure bare metal environment where three bare metal control plane nodes might be overkill.

Two node clusters with fencing

Two-node with Fencing still in tech preview. Useful for environments with only two active datacenters. Especially if you have bare metal control plane nodes. We got quite a few customers with only to data centers available.

About the HostFirmwareComponents resource

About the HostFirmwareComponents. When Metal3 is used it’s now possible to update the NIC (Network interface card) firmware.

Boot image updates on Vmware

Boot image management can be used to update the boot image for new nodes. Up until now when you installed you cluster on VMware with a certain boot image (let’s say 4.15) it never got updates. New nodes were always booted from this old image and later updated to the current cluster release.

About on-cluster image mode

About on-cluster image mode. Image mode allows you to customize the node operating system image. You basically create a Containerfile, install custom RPM’s or deploy custom configuration files in the Containerfile and OCP creates a layered image for you cluster. The image is automatically rolled out to the cluster with the Machine Config Operator. Fancy stuff….

Pinning images

Pinning images. If your internet connection is flaky, or you have reasons to not trust the availablity of Red Hat registries like quay.io or registry.redhat.io you can now pin images those images. They are pulled down immediately and will not be garbage collected (hopefully…).

BGP routing

About BGP routing. MetalLB had support for announcing IP’s via BGP for a long time, but now it’s also possible to announce UDN’s or EgressIP. IMHO this is especially interesting for EgressIP because there were some nasty bugs around using GRAP (Gratuitous ARP) for announcing IP’s to switches.

Migrating a configured br-ex bridge to NMState

Migrating a configured br-ex bridge to NMState. There’s this ominous configure-ovs.sh that reconfigures the public interface of a node an brings up br-ex. There was support for deploying an NMState-based configuration during cluster installation and not using configure-ovs.sh. Now it’s also possible to get rid of the shell script after installation.

Configuration for a Bond CNI secondary network

Configuration for a Bond CNI secondary network. Bond interface can now be created for interface in containers. This currently only supports SR-IOV virtual functions.

Manage secure signatures with sigstore

Manage secure signatures with sigstore. Sigstore support can now be enabled on a cluster level or on an individual namespace level.

Running pods in Linux user namespaces

Running pods in Linux user namespaces. This is IMHO a big one. Linux user namespaces are finally supported in OpenShift! So it’s possible to grant root inside a container and map the root UID to an unprivileged ID outside the container.

Adjust pod resource levels without pod disruption

Adjust pod resource levels without pod disruption. Resize CPU and memory resource without restarting a Pod!

Introducing the oc adm upgrade recommend command (General Availability)

Understanding OpenShift upgrades. oc know officially supports cluster upgrade from the command line. oc adm upgrade recommended performs some pre-flight checks before upgrading a cluster.

Additional cluster latency requirements for etcd

Cluster latency requirements for etcd. ETCD latency requirements were updated in the documentation.

Sunset of the Red Hat Marketplace, operated by IBM

Sunset of the Red Hat Marketplace, operated by IBM. It seems Red Hat Marktplace is no more…

How to use Changed Block Tracking (Dev Preview) in OpenShift 4.20

How to use Changed Block Tracking (Dev Preview) in OpenShift 4.20. Change block tracking for PV’s is now in dev preview. Whatever this means exactly, needs more investigation.

A second look into the Kubernetes Gateway API on OpenShift

Sun, 31 Aug 2025 00:00:00 +0000

This is our second look into the Kubernetes Gateway API an it’s integration into OpenShift. This post covers TLS configuration.

The Kubernetes Gateway API is new implementation of the ingress, load balancing and service mesh API’s. See upstream for more information.

Also the OpenShift documentation provides an overview of the Gateway API and it’s integration.

We demonstrate how to add TLS to our Nginx deployment, how to implement a shared Gateway and finally how to implement HTTP to HTTPS redirection with the Gateway API. Furthermore we cover how HTTPRoute objects attach to Gateways and dive into ordering of HTTPRoute objects.

b2cc commented that deploying the gateway in the openshift-ingress namespace, as we did below, is not support by Red Hat. We are not aware this is documented somewhere (please let use now if it is). Only after opening a support ticket he learned that this not supported. We will keep the examples below as they are, so be careful and this may eat your cat.

References

A first look into the Kubernetes Gateway API on OpenShift

Adding TLS to our Nginx deployment

In our fist post we simply exposed a Nginx web server via the Gateway API. We only enabled HTTP, so let’s try to do the same with HTTPS now.

Remember we use a DNS wildcard domain *.gtw.ocp.lan.stderr.at which points to our Gateway. The gateway is exposed via a Service of type LoadBalancer. We use MetalLB for this.

The first step is setting up a wildcard TLS certificate for our custom domain *.gtw.ocp.lan.stderr.at. We are using EasyRSA here, but use whatever tool you like.

Just for reference this is how we created a wildcard cert with EasyRSA:

$ EASYRSA_CERT_EXPIRE=3650 EASYRSA_EXTRA_EXTS="subjectAltName=DNS:*.gtw.ocp.lan.stderr.at" ./easyrsa gen-req gtw.ocp.lan.stderr.at
$ EASYRSA_CERT_EXPIRE=3650 ./easyrsa sign-req serverClient gtw.ocp.lan.stderr.at

EasyRSA stores the public key under pki/issued and the private key under pki/private. We copied the certificate and the private key to a temporary directory.

Next we need to remove the private key passphrase and create a Kubernetes secret from the private and pubic key:

$ openssl rsa -in gtw.ocp.lan.stderr.at.key -out gtw.ocp.lan.stderr.at-insecure.key
$ oc create secret -n openshift-ingress tls gateway-api --cert=gtw.ocp.lan.stderr.at.crt --key=gtw.ocp.lan.stderr.at-insecure.key

Now it’s time to add a TLS listener to our Gateway resource in the openshift-ingress namespace. Remember for the OpenShift Gateway API implementation, Gateways have to be deployed in the openshift-ingress namespace.

---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: http-gateway
namespace: openshift-ingress
spec:
gatewayClassName: openshift-default
listeners:
- name: http
protocol: HTTP
port: 80
hostname: "*.gtw.ocp.lan.stderr.at"
- name: https
protocol: HTTPS (1)
port: 443 (2)
hostname: "*.gtw.ocp.lan.stderr.at" (3)
tls:
mode: Terminate (4)
certificateRefs:
- name: gateway-api (5)
allowedRoutes: (6)
namespaces:
from: All

1	We want to support HTTPS
2	We use the default HTTPS port 443
3	The URLs we support with this listener are the same as for HTTP
4	We use edge termination for now, this means HTTP traffic will only be encrypted up to the gateway. From the gateway to our pod we speak plain HTTP.
5	This is the name of the TLS secret we created above
6	We accept routes from all namespaces

Also remember from our first post that we created a ReferenceGrant in the namespace where Nginx is running. Otherwise HTTP routes will not be accepted.

Finally lets try to access our Nginx pod via HTTPS:

$ curl -v https://nginx.gtw.ocp.lan.stderr.at
* Host nginx.gtw.ocp.lan.stderr.at:443 was resolved.
* IPv6: (none)
* IPv4: 10.0.0.150
* Trying 10.0.0.150:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
* CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
* subject: CN=gtw.ocp.lan.stderr.at
* start date: Aug 30 10:01:33 2025 GMT
* expire date: Aug 28 10:01:33 2035 GMT
* subjectAltName: host "nginx.gtw.ocp.lan.stderr.at" matched cert's "*.gtw.ocp.lan.stderr.at"
* issuer: CN=tntinfra CA
* SSL certificate verify ok.
* Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* Connected to nginx.gtw.ocp.lan.stderr.at (10.0.0.150) port 443
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://nginx.gtw.ocp.lan.stderr.at/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: nginx.gtw.ocp.lan.stderr.at]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.11.1]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: nginx.gtw.ocp.lan.stderr.at
> User-Agent: curl/8.11.1
> Accept: */*
>
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/2 200
< server: nginx/1.29.1
< date: Sat, 30 Aug 2025 14:30:20 GMT
< content-type: text/html
< content-length: 615
< last-modified: Wed, 13 Aug 2025 14:33:41 GMT
< etag: "689ca245-267"
< accept-ranges: bytes
(output omitted)

Yes, we can reach our Nginx via HTTPS, and the gateway presents the TLS certificate we created.

Be aware that we are still using the same HTTPRoute for Nginx from our previous blog post.

Just for completeness here is the HTTPRoute:

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: nginx-route
spec:
parentRefs:
- name: http-gateway
namespace: openshift-ingress
hostnames: ["nginx.gtw.ocp.lan.stderr.at"]
rules:
- backendRefs:
- name: nginx
namespace: gateway-api-test
port: 8080

Also Remember that we are using a dedicated Gateway and all HTTPRoutes must be in the namespace openshift-ingress

Moving to a shared gateway

Up until now we had to create all HTTPRoute objects in the openshift-ingress namespace. The Gateway API support two modes of operations:

Dedicated gateway: all HTTPRoute object need to be in the same namespace as the gateway
Shared gateway: The gateway runs in the openshift-ingress namespace and we allow HTTPRoute objects from all or specific namespaces.

The first step in creating a shared gateway is to modify the gateway resource:

---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: http-gateway
namespace: openshift-ingress
spec:
gatewayClassName: openshift-default
listeners:
- name: http
protocol: HTTP
port: 80
hostname: "*.gtw.ocp.lan.stderr.at"
allowedRoutes: (1)
namespaces:
from: All

1	We now allow HTTPRoute objects from all namespaces in the cluster

Next we delete the existing HTTPRoute for Nginx in the openshift-ingress namespaces, and verify that we can’t reach Nginx:

$ oc delete httproutes.gateway.networking.k8s.io -n openshift-ingress nginx-route
httproute.gateway.networking.k8s.io "nginx-route" deleted
$ curl -I http://nginx.gtw.ocp.lan.stderr.at
HTTP/1.1 404 Not Found (1)
date: Sat, 30 Aug 2025 15:02:23 GMT
transfer-encoding: chunked

1	Our Nginx route stopped working

Next we apply our modified Gateway resource in the openshift-ingress namespace and the HTTPRoute object in the gateway-api-test namespace.

$ oc apply -n openshift-ingress -f gateway--selector.yaml
gateway.gateway.networking.k8s.io/http-gateway configured
$ oc apply -n gateway-api-test -f httproute.yaml (1)
httproute.gateway.networking.k8s.io/nginx-route created
$ curl -I http://nginx.gtw.ocp.lan.stderr.at
HTTP/1.1 200 OK (2)
server: nginx/1.29.1
date: Sat, 30 Aug 2025 15:04:34 GMT
content-type: text/html
content-length: 615
last-modified: Wed, 13 Aug 2025 14:33:41 GMT
etag: "689ca245-267"
accept-ranges: bytes

1	We create the HTTPRoute in the gateway-api-test namespace
2	We can reach our Nginx pod again

So our shared gateway seems to be working. But what if we want to restrict which namespaces are allowed to create route objects?

The Gateway API allows the following settings under spec.listeners[].allowedRoutes.namespaces.from field

All: Allow from all namespaces
Selector: Specify a selector
Same: Only allow HTTPRoutes in the same namespaces
None: Do not allow any routes to attach

See the API specification FromNamespaces for details.

Let’s try to use a more specific selector for our gateway:

---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: http-gateway
namespace: openshift-ingress
spec:
gatewayClassName: openshift-default
listeners:
- name: http
protocol: HTTP
port: 80
hostname: "*.gtw.ocp.lan.stderr.at"
allowedRoutes:
namespaces:
from: Selector (1)
selector:
matchLabels:
kubernetes.io/metadata.name: gateway-api-test (2)

1	Now we are using the Selector option
2	Because we do not have a specific label on the namespace we would like to use, let’s use the metadata.name label Kubernetes created for us

We create a new yaml file gateway-selector.yaml and appy the new configuration:

$ oc apply -n openshift-ingress -f gateway-selector.yaml
gateway.gateway.networking.k8s.io/http-gateway configured
$ curl -I http://nginx.gtw.ocp.lan.stderr.at
HTTP/1.1 200 OK
server: nginx/1.29.1
date: Sat, 30 Aug 2025 15:17:17 GMT
content-type: text/html
content-length: 615
last-modified: Wed, 13 Aug 2025 14:33:41 GMT
etag: "689ca245-267"
accept-ranges: bytes

All good, still working.

Just for testing we modified the namespace name in the Gateway definition to NOT match the namespace of our Nginx deployment and confirmed that we receive a 404 not found response.

Implementing HTTP to HTTPS redirect

As a last test for this post let’s try to implement HTTP to HTTPS redirects.

We deployed the following Gateway configuration:

---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: http-gateway
namespace: openshift-ingress (1)
spec:
gatewayClassName: openshift-default
listeners:
- name: http
protocol: HTTP
port: 80
hostname: "*.gtw.ocp.lan.stderr.at"
allowedRoutes:
namespaces:
from: Selector
selector:
matchLabels:
kubernetes.io/metadata.name: gateway-api-test2
- name: https (2)
protocol: HTTPS
port: 443
hostname: "*.gtw.ocp.lan.stderr.at"
tls:
mode: Terminate
certificateRefs:
- name: gateway-api
allowedRoutes:
namespaces:
from: All

1	Always deploy the gateway to the openshift-ingress namespace for the OpenShift Gateway API implementation
2	We added the HTTPS configuration back

The upstream documentation contains an example on how to implements HTTP to HTTPS redirects. We created the following additional HTTPRoute object in the gateway-api-test namespace:

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-https-redirect
spec:
parentRefs:
- name: http-gateway (1)
namespace: openshift-ingress
sectionName: http (2)
hostnames:
- nginx.gtw.ocp.lan.stderr.at
rules:
- filters:
- type: RequestRedirect
requestRedirect:
scheme: https
statusCode: 301

1	Match our Gateway http-gateway
2	Match the http section in our gateway

Just for reference this is the HTTPRoute object to expose Nginx:

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: nginx-route
spec:
parentRefs:
- name: http-gateway
namespace: openshift-ingress
hostnames: ["nginx.gtw.ocp.lan.stderr.at"]
rules:
- backendRefs:
- name: nginx
namespace: gateway-api-test
port: 8080

First we re-applied our Gateway configuration

$ oc apply -f gateway-https-selector.yaml
gateway.gateway.networking.k8s.io/http-gateway configured

Let’s try and verify if our redirect is working, we need to apply both routes:

$ oc apply -f httproute.yaml
httproute.gateway.networking.k8s.io/nginx-route created
$ oc apply -f http-https-redirect-route.yaml
httproute.gateway.networking.k8s.io/http-https-redirect created

And test with curl:

$ curl -I http://nginx.gtw.ocp.lan.stderr.at
HTTP/1.1 200 OK (1)
server: nginx/1.29.1
date: Sat, 30 Aug 2025 15:37:20 GMT
content-type: text/html
content-length: 615
last-modified: Wed, 13 Aug 2025 14:33:41 GMT
etag: "689ca245-267"
accept-ranges: bytes

1	Hm, strange we still get 200 OK and NOT a redirect to HTTPS

Understanding HTTPRoute ordering

After a longer search through the documentation we found some hints on why this is happening.

Let’s take a more detailed look at our http-to-https route again, as a HTTPRoute attaches to a Gateway, we focus on the parentRefs in the HTTPRoute object. In our current understanding parentRefs select a Gateway:

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-https-redirect
spec:
parentRefs: (1)
- name: http-gateway (2)
namespace: openshift-ingress (3)
sectionName: http (4)
hostnames:
- nginx.gtw.ocp.lan.stderr.at
rules:
- filters:
- type: RequestRedirect
requestRedirect:
scheme: https
statusCode: 301

1	Ok, this is the parentRefs section we are looking for
2	name selects the name of the Gateway we want to attach to
3	namespace specifies the namespace where we can find the Gateway
4	sectionName selects the section in the Gateway where we want to attach to.

So this HTTPRoute explicitly attaches to a Gateway in a Namespace that has a Section http defined.

If you look at the Gateway configuration above you will see that we have a section for HTTP traffic and one for HTTPS traffic.

Let’s compare this with our Nginx HTTPRoute definition:

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: nginx-route
spec:
parentRefs: (1)
- name: http-gateway (2)
namespace: openshift-ingress (3)
hostnames: ["nginx.gtw.ocp.lan.stderr.at"]
rules:
- backendRefs:
- name: nginx
namespace: gateway-api-test
port: 8080

1	The parentRefs section
2	The Gateway we would like to attach to
3	The namespace where the Gateway is deploy

Note that Section is missing in this configuration.

So this HTTPRoute actually attaches to both sections in our Gateway definition, HTTP and HTTPS. Which is not what we want.

When a client hits the HTTP endpoint we want to redirect the traffic to HTTPS
When a client hits the HTTPS endpoint we want the traffic to be forward to our Nginx deployment

We found the following statement statement how ordering works in the Gateway API:

If ties still exist across multiple Routes, matching precedence MUST be
determined in order of the following criteria, continuing on ties:
The oldest Route based on creation timestamp.

When we look at the timestamps of our HTTPRoutes:

oc get httproute -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.creationTimestamp}{"\n"}{end}'
http-https-redirect 2025-08-31T09:17:46Z (1)
nginx-route 2025-08-31T09:17:40Z (2)

1	Creation timestamp of the redirect route
2	Creation timestamp of the nginx route

The Nginx HTTPRoute is older than the HTTP-to-HTTP HTTPRoute. So this matches first and a 200 OK is returned.

So let’s try to revers how we applied our HTTPRoutes:

$ oc delete httproutes.gateway.networking.k8s.io --all
httproute.gateway.networking.k8s.io "http-https-redirect" deleted
httproute.gateway.networking.k8s.io "nginx-route" deleted
$ oc apply -f http-to-https-httproute.yaml
httproute.gateway.networking.k8s.io/http-https-redirect created
$ oc apply -f nginx-httproute.yaml
httproute.gateway.networking.k8s.io/nginx-route created
$ oc get httproute -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.creationTimestamp}{"\n"}{end}'
http-https-redirect 2025-08-31T10:34:55Z (1)
nginx-route 2025-08-31T10:35:11Z (2)

1	Creation timestamp of the HTTP-to-HTTPS route
2	Creation timestamp of the nginx route

Now the HTTP-to-HTTPS route is the oldest route. Let’s try again calling Nginx with curl:

$ curl -I http://nginx.gtw.ocp.lan.stderr.at
HTTP/1.1 301 Moved Permanently (1)
location: https://nginx.gtw.ocp.lan.stderr.at/
date: Sun, 31 Aug 2025 10:37:13 GMT
transfer-encoding: chunked
$ curl -I https://nginx.gtw.ocp.lan.stderr.at
HTTP/2 200 (2)
server: nginx/1.29.1
date: Sun, 31 Aug 2025 10:37:17 GMT
content-type: text/html
content-length: 615
last-modified: Wed, 13 Aug 2025 14:33:41 GMT
etag: "689ca245-267"
accept-ranges: bytes

1	The HTTP endpoint returns a redirect
2	the HTTPS endpoint returns 200 OK from Nginx

So now we have the expected behavior: HTTP is redirect to HTTPS!

As depending on the time when an object is created is definitely NOT a good idea, let’s be more specific in our Nginx HTTPRoute:

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: nginx-route
spec:
parentRefs:
- name: http-gateway
namespace: openshift-ingress
sectionName: https (1)
hostnames: ["nginx.gtw.ocp.lan.stderr.at"]
rules:
- backendRefs:
- name: nginx
namespace: gateway-api-test
port: 8080

1	We explicitly select the HTTPS section in our Gateway configuration

Next we delete our HTTPRoutes again, and re-apply them in the order that didn’t work the first time (Nginx is the oldest route):

$ oc delete httproutes.gateway.networking.k8s.io --all
httproute.gateway.networking.k8s.io "http-https-redirect" deleted
httproute.gateway.networking.k8s.io "nginx-route" deleted
$ oc apply -f http-to-https-httproute.yaml
httproute.gateway.networking.k8s.io/http-https-redirect created
$ oc get httproute -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.creationTimestamp}{"\n"}{end}'
http-https-redirect 2025-08-31T10:45:01Z
nginx-route 2025-08-31T10:44:57Z (1)
$ curl -I http://nginx.gtw.ocp.lan.stderr.at
HTTP/1.1 301 Moved Permanently (2)
location: https://nginx.gtw.ocp.lan.stderr.at/
date: Sun, 31 Aug 2025 10:46:22 GMT
transfer-encoding: chunked
$ curl -I https://nginx.gtw.ocp.lan.stderr.at
HTTP/2 200 (3)
server: nginx/1.29.1
date: Sun, 31 Aug 2025 10:46:30 GMT
content-type: text/html
content-length: 615
last-modified: Wed, 13 Aug 2025 14:33:41 GMT
etag: "689ca245-267"
accept-ranges: bytes

1	The Nginx route is the oldest route
2	The HTTP endpoint returns a redirect to HTTPS
3	The response from our Nginx deployment

Finally everything works as expected!

A HTTPRoute attaches to a Gateway. Always be as specific as possible which Gateway to match and which section in the Gateway.

Conclusion

In this blog post we demonstrated to implement TLS with the Gateway API. We also implemented a shared Gateway with HTTPRoute objects in different namespaces.

Furthermore we configured HTTP to HTTPS redirects and dove into HTTPRoute ordering if a route matches multiple listeners in a Gateway definition.

A first look into the Kubernetes Gateway API on OpenShift

Fri, 29 Aug 2025 00:00:00 +0000

This blog post summarizes our first look into the Kubernetes Gateway API and how it is integrated in OpenShift.

The Kubernetes Gateway API is new implementation of the ingress, load balancing and service mesh API’s. See upstream for more information.

Also the OpenShift documentation provides an overview of the Gateway API and it’s integration.

Things to consider when using Gateway API with OpenShift

Currently UDN (User Defined Networks) with Gateway API are not supported.
Only TLS termination on the edge is supported (no pass-through or re-encrypt), this needs to be confirmed. We can’t find the original source of this statement
The standard OpenShift ingress controller manages Gateway API Resources
Gateway API provides a standard on how to get client traffic into a Kubernetes cluster. Vendors provide an implementation of the API. So OpenShift provides ONE possible implementation, but there could be more than one in a cluster.
- We found the following sentence in the OpenShift documentation interesting:
  Because OpenShift Container Platform uses a specific version of Gateway API CRDs, any use of third-party implementations of Gateway API must conform to the OpenShift Container Platform implementation to ensure that all fields work as expected

Setting up Gateway API on OpenShift

Before you begin, ensure you have the following:

OpenShift 4.19 or higher with cluster-admin access

First you need to create a GatewayClass object.

Be aware that the GatewayClass object is NOT namespaced.

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: openshift-default
spec:
controllerName: openshift.io/gateway-controller/v1 (1)

1	The controller name needs to be exactly as shown. Otherwise the ingress controller will NOT manage the gateway and associated resources.

This creates a new pod in the openshift-ingress namespace:

$ oc get po -n openshift-ingress
NAME READY STATUS RESTARTS AGE
istiod-openshift-gateway-7b567bc8b4-4lrt2 1/1 Running 0 12m (1)
router-default-6db958cbd-dlbwz 1/1 Running 12 14d

1	this pod got create after applying the gateway class resource

router-default is the default openshift ingress pod. The first difference seems to be the SCC (security context constraint) the pods are using.

$ oc get po -n openshift-ingress -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.openshift\.io/scc}{"\n"}{end}'
istiod-openshift-gateway-7b567bc8b4-4lrt2 restricted-v2
router-default-6db958cbd-dlbwz hostnetwork

The standard router used host networking for listing on port 80 and 443 on the node where it is running. Our GatewayClass currently only provides a pod running Istiod awaiting further configuration. To actually listen for client request additional configuration is required.

A Gateway is required to listen for client requests. We create the following gateway:

---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: http-gateway
namespace: openshift-ingress (1)
spec:
gatewayClassName: openshift-default
listeners:
- name: http
protocol: HTTP
port: 80
hostname: "*.apps.ocp.lan.stderr.at"

1	We create this gateway in the same namespace as the istio deployment. This is required for OpenShift.

This creates an additional pod in the openshift-ingress namespace:

$ oc get po
NAME READY STATUS RESTARTS AGE
http-gateway-openshift-default-d476664f5-h87mp 1/1 Running 0 36s

We also got a new service for the our http-gateway

➜ oc get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
http-gateway-openshift-default LoadBalancer 172.30.183.48 10.0.0.150 15021:30251/TCP,80:30437/TCP 4m52s

The interesting thing is the TYPE of the service. It’s of type LoadBalancer. We have MetalLB deployed in our cluster, this might be the reason for this. We will try to configure a gateway without MetalLB in in upcoming post.

Lets take a look at the gateway resource

$ oc get gtw
NAME CLASS ADDRESS PROGRAMMED AGE
http-gateway openshift-default 10.0.0.150 True 3m

So this seems to be working. But know we have a problem: the *.apps domain that we used for our gateway points already to the default OpenShift Ingress. We could either

redeploy the gateway with a different wildcard domain (e.g. *.gtw…)
create a more specific DNS record that points to our new load balancer

Let’s try to confirm this with curl:

$ curl -I http://bla.apps.ocp.lan.stderr.at
HTTP/1.0 503 Service Unavailable
pragma: no-cache
cache-control: private, max-age=0, no-cache, no-store
content-type: text/html

503 is the response of default OpenShift Ingress.

$ curl -I http://10.0.0.150
HTTP/1.1 404 Not Found
date: Fri, 29 Aug 2025 14:31:31 GMT
transfer-encoding: chunked

Our new gateway returns a 404 not found response. We choose the first option and create another wildcard DNS entry for *.gtw.ocp.lan.stderr.at. We re-deployed our gateway with the new hostname:

---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: http-gateway
namespace: openshift-ingress
spec:
gatewayClassName: openshift-default
listeners:
- name: http
protocol: HTTP
port: 80
hostname: "*.gtw.ocp.lan.stderr.at" (1)

1	New hostname for resources exposed via our gateway

$ oc apply -f gateway.yaml
gateway.gateway.networking.k8s.io/http-gateway created

This also creates a DNSRecord resource:

$ oc describe dnsrecords.ingress.operator.openshift.io -n openshift-ingress http-gateway-c8d7bfc67-wildcard
Name: http-gateway-c8d7bfc67-wildcard
Namespace: openshift-ingress
Labels: gateway.istio.io/managed=openshift.io-gateway-controller-v1
gateway.networking.k8s.io/gateway-name=http-gateway
istio.io/rev=openshift-gateway
Annotations: <none>
API Version: ingress.operator.openshift.io/v1
Kind: DNSRecord
Metadata:
Creation Timestamp: 2025-08-29T14:49:45Z
Finalizers:
operator.openshift.io/ingress-dns
Generation: 1
Owner References:
API Version: v1
Kind: Service
Name: http-gateway-openshift-default
UID: a023de5d-c428-4249-a190-de3cbfeb6964
Resource Version: 141150968
UID: 7a61a867-216e-40b7-88f3-e3934493c477
Spec:
Dns Management Policy: Managed
Dns Name: *.gtw.ocp.lan.stderr.at.
Record TTL: 30
Record Type: A
Targets:
10.0.0.150
Events: <none>

This resource is only internally used by the OpenShift ingress operator (see oc explain dnsrecord for details).

Creating HTTPRoutes for exposing our service.

To actually expose a HTTP pod via our new gateway we need:

A Namespace to deploy an example pod. We will use a Nginx for this
A Service that exposes our Nginx pod
and finally a HTTPRoute resource

For the nginx deployment we used the following manifest:

---
apiVersion: v1
kind: Namespace
metadata:
name: gateway-api-test
spec:
finalizers:
- kubernetes
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
namespace: gateway-api-test
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: quay.io/nginx/nginx-unprivileged:1.29.1
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: nginx
namespace: gateway-api-test
spec:
selector:
app: nginx
ports:
- protocol: TCP
port: 8080
targetPort: 8080

Let’s see if our nginx pod got deployed successfully:

$ oc get po,svc -n gateway-api-test
NAME READY STATUS RESTARTS AGE
pod/nginx-deployment-796cdf7474-b7bqz 1/1 Running 0 20s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/nginx ClusterIP 172.30.42.36 <none> 8080/TCP 21s

And finally confirm our Service is working:

$ oc port-forward -n gateway-api-test svc/nginx 8080 &
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080
$ curl -I localhost:8080
Handling connection for 8080
HTTP/1.1 200 OK
Server: nginx/1.29.1
Date: Fri, 29 Aug 2025 15:45:12 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Wed, 13 Aug 2025 14:33:41 GMT
Connection: keep-alive
ETag: "689ca245-267"
Accept-Ranges: bytes

We received a response from our nginx pod, hurray!

So next let’s try to create a HTTPRoute to expose our nginx service to external clients:

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: nginx-route
spec:
parentRefs:
- name: http-gateway
namespace: openshift-ingress
hostnames: ["nginx.gtw.ocp.lan.stderr.at"]
rules:
- backendRefs:
- name: nginx
namespace: gateway-api-test
port: 8080

One important point here, the Gateway actually come in two flavors

dedicated gateways, only accepting HTTP routes in the same namespace (openshift-ingress) in our case.
shared gateways, which also accept HTTP route objects from other namespaces

see Gateway API deployment topologies in the OpenShift documentation for more information.

As this post is already rather long, we focus on the dedicated gateway topology for now.

The HTTP route must be deployed in the same namespace as the gateway if the dedicated topology is used.

So let’s deploy our HTTPRoute:

$ oc apply -f httproute.yaml

Verify we can reach our nginx pod:

curl -I http://nginx.gtw.ocp.lan.stderr.at
HTTP/1.1 500 Internal Server Error
date: Fri, 29 Aug 2025 15:57:34 GMT
transfer-encoding: chunked

This return a 500 error, something seems to be wrong with our route, let’s take a look at the status of the HTTPRoute:

$ oc describe gtw http-gateway
.
. (output omitted)
.
Status:
Parents:
Conditions:
Last Transition Time: 2025-08-29T15:54:43Z
Message: Route was valid
Observed Generation: 1
Reason: Accepted
Status: True
Type: Accepted
Last Transition Time: 2025-08-29T15:54:43Z
Message: backendRef nginx/gateway-api-test not accessible to a HTTPRoute in namespace "openshift-ingress" (missing a ReferenceGrant?) (2)
Observed Generation: 1
Reason: RefNotPermitted
Status: False (1)
Type: ResolvedRefs
Controller Name: openshift.io/gateway-controller/v1
Parent Ref:
Group: gateway.networking.k8s.io
Kind: Gateway
Name: http-gateway
Namespace: openshift-ingress

1	Something seems to be wrong as the status is False
2	Seems we are missing a ReferenceGrant

Looking at the upstream documentation reveals a security feature of the Gateway API. Before a HTTPRoute can reach a service in a different namespace we must create a ReferenceGrant in the namespace providing the service.

So let’s try to deploy following ReferenceGrant in the gateway-api-test namespace:

---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
name: nginx
namespace: gateway-api-test
spec:
from:
- group: gateway.networking.k8s.io
kind: HTTPRoute
namespace: openshift-ingress
to:
- group: ""
kind: Service

Checking the status field of our HTTPRoute again:

(output omitted)
Status:
Addresses:
Type: IPAddress
Value: 10.0.0.150
Conditions:
Last Transition Time: 2025-08-29T15:47:07Z
Message: Resource accepted
Observed Generation: 1
Reason: Accepted
Status: True
Type: Accepted
Last Transition Time: 2025-08-29T15:47:08Z
Message: Resource programmed, assigned to service(s) http-gateway-openshift-default.openshift-ingress.svc.cluster.local:80
Observed Generation: 1
Reason: Programmed
Status: True (1)
Type: Programmed

1	Status is now True

and finally calling the nginx pod again via our gateway:

$ curl -I http://nginx.gtw.ocp.lan.stderr.at
HTTP/1.1 200 OK
server: nginx/1.29.1
date: Fri, 29 Aug 2025 16:01:33 GMT
content-type: text/html
content-length: 615
last-modified: Wed, 13 Aug 2025 14:33:41 GMT
etag: "689ca245-267"
accept-ranges: bytes

Finally everything seems to be in place and working.

Conclusion

In this blog post we took a first look at the Kubernetes Gateway API and it’s integration into OpenShift. We enabled the Gateway API via a GatewayClass resource, created a simple HTTP Gateway via a Gateway, deploy a Nginx pod and a Service and exposed the service via a HTTPRoute and a ReferenceGrant.

Hopefully an upcoming blog post will cover how to

How to deploy a Gateway without MetalLB
Deploy a TLS secured service
implement HTTP redirects
rewriting URL’s (if possible)
and other possibilities of the Gateway API

Running Falco on OpenShift 4.12

Sun, 26 Nov 2023 00:00:00 +0000

As mentioned in our previous post about Falco, Falco is a security tool to monitor kernel events like system calls or Kubernetes audit logs to provide real-time alerts.

In this post I'll show to customize Falco for a specific use case. We would like to monitor the following events:

An interactive shell is opened in a container
Log all commands executed in an interactive shell in a container
Log read and writes to files within an interactive shell inside a container
Log commands execute via `kubectl/oc exec` which leverage the pod/exec K8s endpoint

The rules we created for those kind of events are available here.

Deploying custom rules and disabling the default ruleset

Falco comes with a quite elaborate ruleset for creating security relevant events. But for our use case we just want to deploy a specific set of rules (see the list above).

As we are deploying Falco via Helm, we use the following values for `rules_files`:

falco:
 rules_file:
 - /etc/falco/extra-rules.d
 - /etc/falco/rules.d

This instructs Falco to only load rules from the directories mentioned above.

We use a Kustomize configMapGenerator to create a Kubernetes ConfigMap from our custom rules file:

configMapGenerator:
 - name: falco-extra-rules
 options:
 disableNameSuffixHash: true
 files:
 - falco-extra-rules.yaml

The complete Kustomize configuration is here.

Furthermore we instruct Falco to mount our custom rule ConfigMap created above in the Helm values file:

mounts:
 volumes:
 - name: falco-extra-rules-volume
 optional: true
 configMap:
 name: falco-extra-rules
 volumeMounts:
 - mountPath: /etc/falco/extra-rules.d
 name: falco-extra-rules-volume

The complete Helm values files is available here.

Disable automatic rule updates

Falco updates all rules when it starts (via an initContainer) and also updates those rules on a regular basis. We would also like to disable this behavior:

falcoctl:
 artifact:
 install:
 enabled: false
 follow:
 enabled: false

Create events for kubectl/oc exec

One problem problem is monitoring pod exec events. Using Falcos eBPF monitoring capabilities we found no way to limit those events to pod exec's. This might be because the Falco rule language is new to us and maybe there is a way to use eBPF filtering. Just let us know if you find a solution!

But we came up with a different way of capturing pod/exec events:

Falco also allows monitoring Kubernetes audit events, logged by the kube-apiserver. Every time you hit the pod/exec endpoint, K8s logs the following event in the audit log:

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"5c19c1d0-00a7-4af5-a236-5345b5963581","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/falco/pods/falco-8mqj7/exec?command=cat\u0026command=%2Fetc%2Ffalco%2Fextra-rules.d%2Ffalco-extra-rules.yaml\u0026container=falco\u0026stderr=true\u0026stdout=true","verb":"create","user":{"username":"root","uid":"d82ec74a-75e3-4798-a084-4b766dcea5ef","groups":["cluster-admins","system:authenticated:oauth","system:authenticated"],"extra":{"scopes.authorization.openshift.io":["user:full"]}},"sourceIPs":["10.0.32.220"],"userAgent":"oc/4.13.0 (linux/amd64) kubernetes/92b1a3d","objectRef":{"resource":"pods","namespace":"falco","name":"falco-8mqj7","apiVersion":"v1","subresource":"exec"},"responseStatus":{"metadata":{},"code":101},"requestReceivedTimestamp":"2023-11-13T17:23:16.999602Z","stageTimestamp":"2023-11-13T17:23:17.231121Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"root-cluster-admin\" of ClusterRole \"cluster-admin\" to User \"root\""}}

As you can hopefully see, the command executed is available in the requestURI field.

So we enabled the k8saudit Falco plugin and created an additional rule for those kind of events.

falco:
 plugins:
 - name: k8saudit
 library_path: libk8saudit.so
 init_config:
 # maxEventSize: 262144
 # webhookMaxBatchSize: 12582912
 # sslCertificate: /etc/falco/falco.pem
 # open_params: "http://:9765/k8s-audit"
 open_params: "/host/var/log/kube-apiserver/audit.log"
 - name: json
 library_path: libjson.so
 init_config: ""

Implementing event routing

We had an additional requirement to route events based on the following rules:

Events that do not contain sensitive data (like usernames) should go to a specific Kafka topic
Events that do contain sensitive data (like usernames) should be routed to another Kafka topic

Our first thought was to leverage Falcosidekick's minimumpriority field for routing. Events with sensitive data would get a higher priority. But the sink with a lower minimumpriority would get events with higher priority as well, which means events with sensitive data.

Furthermore as far as we know Falco currently only supports one Kafka configuration (we need two for two topics).

At this point in time we are not aware of a possibility to implement this with Falco or Falcosidekick directly.

There are some discussions upstream on implementing such a feature:

Our current idea is to use Vector for event routing. We will try to implement the following pipeline:

Tips and Tricks

Monitor Redis disk usage

One small hint when using falcosidekick-ui to debug/monitor events. It happened to us that the Redis volume was full and suddenly we couldn't see new events in the UI.

We stopped the UI and Redis pods, removed the PVC and just ran our kustomization again, to recreate the PVC and the pods.

Monitor falco pod logs when changing rules

It's always wise to monitor one Falco pod for errors when deploying new rules, for example at one point we hit the following error:

{"hostname":"falco-2hlkm","output":"Falco internal: hot restart failure: /etc/falco/extra-rules.d/falco-extra-rules.yaml: Invalid\n1 Errors:\nIn rules content: (/etc/falco/extra-rules.d/falco-extra-rules.yaml:0:0)\n rule 'Terminal shell in container': (/etc/falco/extra-rules.d/falco-extra-rules.yaml:25:2)\n condition expression: (\"spawned_process a...\":26:71)\n------\n...ocess and container and shell_procs and proc.tty != 0 and container_entrypoint\n ^\n------\nLOAD_ERR_VALIDATE (Error validating rule/macro/list/exception objects): Undefined macro 'container_entrypoint' used in filter.\n","output_fields":{},"priority":"Critical","rule":"Falco internal: hot restart failure","source":"internal","time":"2023-11-13T11:47:14.639547735Z"}

Falco is quite resilient when it comes to errors in rules files and provides useful hints on what might be wrong:

Undefined macro 'container_entrypoint' used in filter

So we just added the missing macro and all was swell again.

Setting up Falco on OpenShift 4.12

Mon, 23 Oct 2023 00:00:00 +0000

Falco is a security tool to monitor kernel events like system calls to provide real-time alerts. In this post I'll document the steps taken to get Open Source Falco running on an OpenShift 4.12 cluster.

UPDATE: Use the falco-driver-loader-legacy image for OpenShift 4.12 deployments.

First Try

We will use the Falco Helm chart version 3.8.0 for our first try of setting up Falco on our OpenShift cluster.

This is our values file:

driver:
 kind: ebpf

falco:
 json_output: true
 json_include_output_property: true
 log_syslog: false
 log_level: info

falcosidekick:
 enabled: true
 webui:
 enabled: true

We would like to use the eBPF driver to monitor kernel events, enable falco sidekick, which is used to route events and the falco sidekick UI for easier testing.

Because of reasons we leverage kustomize to render the helm chart. The final kustomize config is here (after fixing all problems mentioned below).

So after deploying the chart via ArgoCD (another story), we have the following pods:

$ oc get pods -n falco
NAME READY STATUS RESTARTS AGE
falco-4cc8j 0/2 Init:CrashLoopBackOff 5 (95s ago) 4m31s
falco-bx87j 0/2 Init:CrashLoopBackOff 5 (75s ago) 4m29s
falco-ds9w6 0/2 Init:CrashLoopBackOff 5 (99s ago) 4m30s
falco-falcosidekick-ui-redis-0 1/1 Running 0 4m28s
falco-gxznz 0/2 Init:CrashLoopBackOff 3 (14s ago) 4m30s
falco-vtnk5 0/2 Init:CrashLoopBackOff 5 (98s ago) 4m29s
falco-wbn2k 0/2 Init:CrashLoopBackOff 5 (80s ago) 4m29s

hm, not so good. Seems like some initContainers are failing.

Let's check the falco-driver-loader initContainer:

* Setting up /usr/src links from host
* Running falco-driver-loader for: falco version=0.36.1, driver version=6.0.1+driver, arch=x86_64, kernel release=4.18.0-372.73.1.el8_6.x86_64, kernel version=1
* Running falco-driver-loader with: driver=bpf, compile=yes, download=yes
* Mounting debugfs
mount: /sys/kernel/debug: permission denied.
 dmesg(1) may have more information after failed mount system call.
* Filename 'falco_rhcos_4.18.0-372.73.1.el8_6.x86_64_1.o' is composed of:
 - driver name: falco
 - target identifier: rhcos
 - kernel release: 4.18.0-372.73.1.el8_6.x86_64
 - kernel version: 1
* Trying to download a prebuilt eBPF probe from https://download.falco.org/driver/6.0.1%2Bdriver/x86_64/falco_rhcos_4.18.0-372.73.1.el8_6.x86_64_1.o
curl: (22) The requested URL returned error: 404
Unable to find a prebuilt falco eBPF probe
* Trying to compile the eBPF probe (falco_rhcos_4.18.0-372.73.1.el8_6.x86_64_1.o)
expr: syntax error: unexpected argument '1'
make[1]: *** /lib/modules/4.18.0-372.73.1.el8_6.x86_64/build: No such file or directory. Stop.
make: *** [Makefile:38: all] Error 2
mv: cannot stat '/usr/src/falco-6.0.1+driver/bpf/probe.o': No such file or directory
Unable to load the falco eBPF probe

Seems to be an issue with a missing directory /lib/modules/4.18.0-372.73.1.el8_6.x86_64/build.

And we told the helm chart to enable falco-sidekick and falco-sidekick-ui, but where are they?

Let's check the events with oc get events as well, and what do we see?

8s Warning FailedCreate replicaset/falco-falcosidekick-7cfbbbf89f Error creating: pods "falco-falcosidekick-7cfbbbf89f-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: []int64{1234}: 1234 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1234: must be in the ranges: [1000730000, 1000739999], provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "falco": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount]
6s Warning FailedCreate replicaset/falco-falcosidekick-ui-76885bd484 Error creating: pods "falco-falcosidekick-ui-76885bd484-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: []int64{1234}: 1234 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1234: must be in the ranges: [1000730000, 1000739999], provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "falco": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount]

Looks like a problem with OpenShifts Security Context constraints (SCC's).

Summary of problems

The falco DaemonSet fails to start pods because there is an issue with a missing directory
Falco Sidekick and Falco Sidekick UI fails to start because of Security Context Constraint (SCC) issues

Fixing the Falco daemonset

Falco tries to download a pre-compiled eBPF probe, fails and then tries to compile that probe for our host OS kernel. This fails with the message:

make[1]: *** /lib/modules/4.18.0-372.73.1.el8_6.x86_64/build: No such file or directory. Stop.

As far as we know there are no kernel sources installed on RHCOS nodes in OpenShift. After a little bit of searching the interweb we found the following issue comment on Github:

OpenShift under vsphere: Download failed, consider compiling your own falco module and loading it or getting in touch with the Falco community

So we need to enable the kernel-devel extension, the official docs are here. It does not mention kernel-devel, but there's a knowledge base article mentioning kernel-devel, so let's give it a try.

We deploy two MachineConfigs, one for worker and one for master nodes to rollout the extension, the worker configuration looks like this:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
 labels:
 machineconfiguration.openshift.io/role: worker
 name: 99-worker-kernel-devel-extensions
spec:
 extensions:
 - kernel-devel

See also our Kustomize configuration here.

As soon as we apply our MachineConfigs, OpenShift starts the rollout via MaschineConfigPool's:

$ oc get mcp
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
master rendered-master-ce464ff45cc049fce3e8a63e36a4ee9e False True False 3 0 0 0 13d
worker rendered-worker-a0f8f0d915ef01ba4a1ab3047b6c863d False True False 3 0 0 0 13d

When the rollout is done, let's restart all Falco DaemonSet pods:

$ oc delete pods -l app.kubernetes.io/name=falco

And check the status:

$ oc get pods
NAME READY STATUS RESTARTS AGE
falco-5wfnk 0/2 Init:Error 1 (3s ago) 7s
falco-66fxw 0/2 Init:0/2 1 (2s ago) 6s
falco-6fbc7 0/2 Init:CrashLoopBackOff 1 (2s ago) 8s
falco-8h8n4 0/2 Init:0/2 1 (2s ago) 6s
falco-falcosidekick-ui-redis-0 1/1 Running 0 18m
falco-nhld2 0/2 Init:CrashLoopBackOff 1 (2s ago) 6s
falco-xqv4b 0/2 Init:CrashLoopBackOff 1 (3s ago) 8s

still, the initContainers fail. Lets check the log again

$ oc logs -c falco-driver-loader falco-5wfnk
* Setting up /usr/src links from host
* Running falco-driver-loader for: falco version=0.36.1, driver version=6.0.1+driver, arch=x86_64, kernel release=4.18.0-372.73.1.el8_6.x86_64, kernel version=1
* Running falco-driver-loader with: driver=bpf, compile=yes, download=yes
* Mounting debugfs
mount: /sys/kernel/debug: permission denied.
 dmesg(1) may have more information after failed mount system call.
* Filename 'falco_rhcos_4.18.0-372.73.1.el8_6.x86_64_1.o' is composed of:
 - driver name: falco
 - target identifier: rhcos
 - kernel release: 4.18.0-372.73.1.el8_6.x86_64
 - kernel version: 1
* Trying to download a prebuilt eBPF probe from https://download.falco.org/driver/6.0.1%2Bdriver/x86_64/falco_rhcos_4.18.0-372.73.1.el8_6.x86_64_1.o
curl: (22) The requested URL returned error: 404
Unable to find a prebuilt falco eBPF probe
* Trying to compile the eBPF probe (falco_rhcos_4.18.0-372.73.1.el8_6.x86_64_1.o)
Makefile:1005: *** "Cannot generate ORC metadata for CONFIG_UNWINDER_ORC=y, please install libelf-dev, libelf-devel or elfutils-libelf-devel". Stop.
make: *** [Makefile:38: all] Error 2
mv: cannot stat '/usr/src/falco-6.0.1+driver/bpf/probe.o': No such file or directory
Unable to load the falco eBPF probe

So this time we get another error, the culprit is the following line

Makefile:1005: *** "Cannot generate ORC metadata for CONFIG_UNWINDER_ORC=y, please install libelf-dev, libelf-devel or elfutils-libelf-devel". Stop.

Back to searching the interweb only reveals an old issue, that should be fixed already.

So as a quick hack we modified the falco-driver-loader image to contain libelf-dev and pushed to image to quay.

We then modified our falco helm configuration to use the updated image:

driver:
 kind: ebpf
 loader:
 initContainer:
 image:
 registry: quay.io
 repository: tosmi/falco-driver-loader
 tag: 0.36.1-libelf-dev

falco:
 json_output: true
 json_include_output_property: true
 log_syslog: false
 log_level: info

falcosidekick:
 enabled: true
 webui:
 enabled: true

Note the updated diver.loader.initContainer section.

Let's check the our pods again:

$ oc get pods
NAME READY STATUS RESTARTS AGE
falco-2ssgx 2/2 Running 0 66s
falco-5hqgg 1/2 Running 0 66s
falco-82kq9 2/2 Running 0 65s
falco-99zxw 2/2 Running 0 65s
falco-falcosidekick-test-connection 0/1 Error 0 67s
falco-falcosidekick-ui-redis-0 1/1 Running 0 31m
falco-slx5k 2/2 Running 0 65s
falco-tzm8d 2/2 Running 0 65s

Success! This time the DaemonSet pods started successfully. Just note that you have to be patient. The first start took about 1-2 minutes to complete.

Let's check the logs of one DaemonSet pod just to sure:

oc logs -c falco-driver-loader falco-2ssgx
* Setting up /usr/src links from host
* Running falco-driver-loader for: falco version=0.36.1, driver version=6.0.1+driver, arch=x86_64, kernel release=4.18.0-372.73.1.el8_6.x86_64, kernel version=1
* Running falco-driver-loader with: driver=bpf, compile=yes, download=yes
* Mounting debugfs
mount: /sys/kernel/debug: permission denied.
 dmesg(1) may have more information after failed mount system call.
* Filename 'falco_rhcos_4.18.0-372.73.1.el8_6.x86_64_1.o' is composed of:
 - driver name: falco
 - target identifier: rhcos
 - kernel release: 4.18.0-372.73.1.el8_6.x86_64
 - kernel version: 1
* Trying to download a prebuilt eBPF probe from https://download.falco.org/driver/6.0.1%2Bdriver/x86_64/falco_rhcos_4.18.0-372.73.1.el8_6.x86_64_1.o
curl: (22) The requested URL returned error: 404
Unable to find a prebuilt falco eBPF probe
* Trying to compile the eBPF probe (falco_rhcos_4.18.0-372.73.1.el8_6.x86_64_1.o)
* eBPF probe located in /root/.falco/6.0.1+driver/x86_64/falco_rhcos_4.18.0-372.73.1.el8_6.x86_64_1.o
* Success: eBPF probe symlinked to /root/.falco/falco-bpf.o

Especially the line

* Success: eBPF probe symlinked to /root/.falco/falco-bpf.o

looks promising. So up to the next problem, getting falco-sidekick and falco-sidekick-ui running.

We also opened a bug report upstream to get feedback from the developers on this issue.

UPDATE

Andreagit97 was so nice mentioning in the issue above that actually there is an image with libelf-dev available, falco-driver-loader-legacy. We can confirm that this image fixes the problem mentioned above.

So this is our final falco helm chart values.yaml:

driver:
 kind: ebpf
 loader:
 initContainer:
 image:
 repository: falcosecurity/falco-driver-loader-legacy

falco:
 json_output: true
 json_include_output_property: true
 log_syslog: false
 log_level: info

falcosidekick:
 enabled: true
 webui:
 enabled: true

Fixing falco-sidekick and falco-sidekick-ui

Remember pod startup actually failed because of the following event (check with oc get events):

.spec.securityContext.fsGroup: Invalid value: []int64{1234}: 1234 is not an allowed group

It seems the sidekick pods want to run with a specific UID. The default OpenShift Security Context Constraint (SCC) restricted prohibits this.

Lets confirm our suspicion:

$ oc get deploy -o jsonpath='{.spec.template.spec.securityContext}{"\n"}' falco-falcosidekick
{"fsGroup":1234,"runAsUser":1234}
$ oc get deploy -o jsonpath='{.spec.template.spec.securityContext}{"\n"}' falco-falcosidekick-ui
{"fsGroup":1234,"runAsUser":1234}

Bingo! securityContext is set to 1234 for both deployments. There is another SCC that we could leverage, nonroot, which basically allows any UID expect 0. We just need to get the ServiceAccount that falco-sidekick and falco-sidekick-ui are actually using:

$ oc get deploy -o jsonpath='{.spec.template.spec.serviceAccount}{"\n"}' falco-falcosidekick
falco-falcosidekick
$ oc get deploy -o jsonpath='{.spec.template.spec.serviceAccount}{"\n"}' falco-falcosidekick-ui
falco-falcosidekick-ui

So falco-sidekick uses falco-sidekick as ServiceAccount and falco-sidekick-ui falco-sidekick-ui. Lets grant both ServiceAccounts access to the nonroot SCC.

kind: ClusterRoleBinding
metadata:
 name: falco-falcosidekick-scc:nonroot
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: system:openshift:scc:nonroot
subjects:
- kind: ServiceAccount
 name: falco-falcosidekick
 namespace: falco
- kind: ServiceAccount
 name: falco-falcosidekick-ui
 namespace: falco

We've already added this file to our Kustomize configuration.

Let's trigger a redeployment by deleting the ReplicaSets of both deployments, they will be re-created automatically:

$ oc delete rs -l app.kubernetes.io/name=falcosidekick
$ oc delete rs -l app.kubernetes.io/name=falcosidekick-ui

Finally let's confirm everything is up and running:

$ oc get deploy,ds,pods
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/falco-falcosidekick 2/2 2 2 5d
deployment.apps/falco-falcosidekick-ui 2/2 2 2 5d

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/falco 6 6 6 6 6 <none> 6d2h

NAME READY STATUS RESTARTS AGE
pod/falco-2ssgx 2/2 Running 0 21m
pod/falco-5hqgg 2/2 Running 0 21m
pod/falco-82kq9 2/2 Running 0 21m
pod/falco-99zxw 2/2 Running 0 21m
pod/falco-falcosidekick-7cfbbbf89f-qxwxs 1/1 Running 0 118s
pod/falco-falcosidekick-7cfbbbf89f-rz5lj 1/1 Running 0 118s
pod/falco-falcosidekick-ui-76885bd484-p7lqm 1/1 Running 0 2m18s
pod/falco-falcosidekick-ui-76885bd484-sfgh4 1/1 Running 0 2m18s
pod/falco-falcosidekick-ui-redis-0 1/1 Running 0 51m
pod/falco-slx5k 2/2 Running 0 21m
pod/falco-tzm8d 2/2 Running 0 21m

Testing Falco

Now that everything seems to be running, lets do a quick test. First we will try to access the Falco Sidekick user interface.

Falco will not deploy a route for the UI automatically, instead we've created a Kustomize overlay with a custom route:

apiVersion: route.openshift.io/v1
kind: Route
metadata:
 name: falco-falcosidekick-ui
 namespace: falco
spec:
 host: falcosidekick-ui.apps.hub.aws.tntinfra.net
 port:
 targetPort: http
 tls:
 termination: edge
 to:
 kind: Service
 name: falco-falcosidekick-ui
 wildcardPolicy: None

After deploying the Route we can access the Falco UI with the hostname specified in the route object. The default username seems to be admin/admin which is kind of strange for a security tool, maybe that's the reason Falco does not expose the UI per default.

Creating an event

As a last test let's try to trigger an event. We open a shell to one of the falco DaemonSet pods and execute a suspicious command:

$ oc rsh falco-2ssgx
Defaulted container "falco" out of: falco, falcoctl-artifact-follow, falco-driver-loader (init), falcoctl-artif# cat /etc/shadow
root:*:19639:0:99999:7:::
daemon:*:19639:0:99999:7:::
bin:*:19639:0:99999:7:::
sys:*:19639:0:99999:7:::
sync:*:19639:0:99999:7:::
games:*:19639:0:99999:7:::
man:*:19639:0:99999:7:::
lp:*:19639:0:99999:7:::
mail:*:19639:0:99999:7:::
news:*:19639:0:99999:7:::
uucp:*:19639:0:99999:7:::
proxy:*:19639:0:99999:7:::
www-data:*:19639:0:99999:7:::
backup:*:19639:0:99999:7:::
list:*:19639:0:99999:7:::
irc:*:19639:0:99999:7:::
_apt:*:19639:0:99999:7:::
nobody:*:19639:0:99999:7:::
#

and we can see an event with priority Warning in the Falco ui.

That's it, seems like Falco is successfully running on OpenShift 4.12.

How to force a MachineConfig rollout

Wed, 18 Oct 2023 00:00:00 +0000

While playing around with Falco (worth another post) I had to force a MachineConfig update even so the actual configuration of the machine did not change.

This posts documents the steps taken.

As this seems to be not clearly documented here it comes

Get the list of current MachineConfigs

$ oc get mc
NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE
00-master 7101fb0720d05771bdc174af918b64deb4efa604 3.2.0 8d
00-worker 7101fb0720d05771bdc174af918b64deb4efa604 3.2.0 8d
01-master-container-runtime 7101fb0720d05771bdc174af918b64deb4efa604 3.2.0 8d
01-master-kubelet 7101fb0720d05771bdc174af918b64deb4efa604 3.2.0 8d
01-worker-container-runtime 7101fb0720d05771bdc174af918b64deb4efa604 3.2.0 8d
01-worker-kubelet 7101fb0720d05771bdc174af918b64deb4efa604 3.2.0 8d
99-kernel-devel-extensions 25h
99-master-generated-registries 7101fb0720d05771bdc174af918b64deb4efa604 3.2.0 8d
99-master-ssh 3.2.0 8d
99-worker-generated-registries 7101fb0720d05771bdc174af918b64deb4efa604 3.2.0 8d
99-worker-ssh 3.2.0 8d
rendered-master-b8a2011b0b09e36088acf47e225b0ed2 7101fb0720d05771bdc174af918b64deb4efa604 3.2.0 5h49m
rendered-master-ce464ff45cc049fce3e8a63e36a4ee9e 7101fb0720d05771bdc174af918b64deb4efa604 3.2.0 8d
rendered-worker-5baefb5bb7ad1d69cd7a0c3dc52ef2f3 7101fb0720d05771bdc174af918b64deb4efa604 3.2.0 25h
rendered-worker-a0f8f0d915ef01ba4a1ab3047b6c863d 7101fb0720d05771bdc174af918b64deb4efa604 3.2.0 8d

We want to force the rollout of a worker node, so remember the name of an old worker config, in our case rendered-worker-5baefb5bb7ad1d69cd7a0c3dc52ef2f3

Currently the desiredConfig and the currentConfig should have the same value

$oc get node node1 -o jsonpath='{.metadata.annotations.machineconfiguration\.openshift\.io/desiredConfig}{"\n"}'
rendered-worker-a0f8f0d915ef01ba4a1ab3047b6c863d
$ oc get node node1 -o jsonpath='{.metadata.annotations.machineconfiguration\.openshift\.io/currentConfig}{"\n"}'
rendered-worker-a0f8f0d915ef01ba4a1ab3047b6c863d

Touch a file called 4 touch /run/machine-config-daemon-force

oc debug node/node1 -- touch /host/run/machine-config-daemon-force

patch the node and set the annotation machineconfiguration.openshift.io/currentConfig to the old rendered config rendered-worker-5baefb5bb7ad1d69cd7a0c3dc52ef2f3

oc patch node ip-10-0-182-18.eu-central-1.compute.internal --patch '{ "metadata": { "annotations": { "machineconfiguration.openshift.io/currentConfig": "rendered-worker-5baefb5bb7ad1d69cd7a0c3dc52ef2f3" } } }'

Watch the MachineConfigPool
```
$ oc get mcp
```

Wait for the config rollout to complete.

Overview of Red Hat's Multi Cloud Gateway (Noobaa)

Fri, 22 Apr 2022 00:00:00 +0000

This is my personal summary of experimenting with Red Hat's Multi Cloud Gateway (MCG) based on the upstream Noobaa project. MCG is part of Red Hat's OpenShift Data Foundation (ODF). ODF bundles the upstream projects Ceph and Noobaa.

Overview

Noobaa, or the Multicloud Gateway (MCG), is a S3 based data federation tool. It allows you to use S3 backends from various sources and

sync
replicate
or simply use existing

S3 buckets. Currently the following sources, or backing stores are supported:

AWS S3
Azure Blob
Google Cloud Storage
Any other S3 compatible storage, for example
- Ceph
- Minio

Noobaa also supports using a local file system as a backing store for S3.

The main purpose is to provide a single API endpoint for applications using various S3 backends.

One of the main features of Noobaa is the storage pipeline. With a standard Noobaa S3 bucket, when storing a new Object Noobaa executes the following steps:

Chunking of the Object
De-duplication
Compression
and Encryption

This means that data stored in public cloud S3 offerings is automatically encrypted. Noobaa also supports using Hashicorp Vault for storing and retrieving encryption keys.

If you need to skip the storage pipeline, Noobaa also supports namespace buckets. For example these type of buckets allow you to write directly to AWS S3 and retrieve Objects via Noobaa. Or it could be used to migrate buckets from one cloud provider to another.

Noobaa also has support for triggering JavaScript based function when

creating new objects
reading existing objects
deleting objects

Setup

With OpenShift Plus or an OpenShift Data Foundation subscription you can use the OpenShift Data Foundation Operator.

For testing Noobaa we used the standalone installation method without setting up Ceph storage (see here). OpenShift was running in AWS for testing.

If you would like to use the upstream version you can use the Noobaa operator (https://github.com/noobaa/noobaa-operator). This is what the OpenShift Data Foundation (ODF) is using as well.

Command line interface

Noobaa also comes with a command line interface noobaa. It's available via an ODF subscription or can be installed separately. See the noobaa-operator readme for more information.

Resources

Before using an S3 object store with Noobaa we need to create so called Resources. This can be done via the Noobaa user interface or via the command line. For example the following commands create a new Resource using an AWS S3 bucket as a backing store

# create an S3 bucket in eu-north-1
aws s3api create-bucket \
 --region eu-north-1 \
 --bucket tosmi-eu-north-1 \
 --create-bucket-configuration LocationConstraint=eu-north-1

# create an S3 bucket in eu-north-1
aws s3api create-bucket \
 --region eu-west-1 \
 --bucket tosmi-eu-west-1 \
 --create-bucket-configuration LocationConstraint=eu-west-1

# create Noobaa backing store using the tosmi-eu-north-1 bucket above
noobaa backingstore create aws-s3 \
 --region eu-north-1 \
 --target-bucket tosmi-eu-north-1 aws-eu-north

# create Noobaa backing store using the tosmi-eu-west-1 bucket above
noobaa backingstore create aws-s3 \
 --region eu-west-1 \
 --target-bucket tosmi-eu-west-1 aws-eu-west

Or if we would like to use Azure blob

# create two resource groups for storage
az group create --location northeurope -g mcg-northeurope

# create two storage accounts
az storage account create --name mcgnortheurope -g mcg-northeurope --location northeurope --sku Standard_LRS --kind StorageV2

# create containers for storing blobs
az storage container create --account-name mcgnortheurope -n mcg-northeurope

# list storage account keys for noobaa
az storage account list
az storage account show -g mcg-northeurope -n mcgnortheurope
az storage account keys list -g mcg-westeurope -n mcgwesteurope
az storage account keys list -g mcg-northeurope -n mcgnortheurope

noobaa backingstore create \
 azure-blob azure-northeurope \
 --account-key="<the key>" \
 --account-name=mcgnortheurope \
 --target-blob-container=mcg-northeurope

Using

noobaa backingstore list

we are able to confirm that our stores were created successfully.

Buckets

After creating the backend stores we are able to create Buckets and define the layout of backends.

There are two ways how to create buckets, either directly via the Noobaa UI, or using Kubernetes (K8s) objects.

We will focus on using K8s objects in this post.

Required K8s objects

The Noobaa operator provides the following Custom Resource Definitions:

BackingStore: we already created BackingStores in the Resources section
BucketClass: a bucket class defines the layout of our bucket (single, mirrored or tiered)
StorageClass: a standard K8s StorageClass referencing the BucketClass
ObjectBucketClaim: A OBC or ObjectBucketClaim creates the bucket for us in Noobaa. Additionally the Noobaa operator creates a ConfigMap and a Secret with the same name as the Bucket, storing access details (ConfigMap) and credentials (Secret) for accessing the bucket.

BucketClass

Let's create a example BucketClass which mirrors objects between the AWS S3 buckets eu-west-1 and eu-north-1.

apiVersion: noobaa.io/v1alpha1
kind: BucketClass
metadata:
 labels:
 app: noobaa
 name: aws-mirrored-bucket-class
 namespace: openshift-storage
spec:
 placementPolicy:
 tiers:
 - backingStores:
 - aws-eu-north
 - aws-eu-west
 placement: Mirror

So we are defining a BucketClass aws-mirrored-bucket-class that has the following placement policy:

A single tier with one backing store
The backing store uses two AWS buckets
- aws-eu-north
- aws-eu-west
The placement policy is mirror, so all objects uploaded to buckets using this BucketClass will be mirrored between aws-eu-north and aws-eu-west.

A BucketClass could have multiple tiers, moving cold data transparently to a lower tier, but let's keep this simple.

StorageClass

After creating our BucketClass we are now able to define a standard K8s StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 annotations:
 description: Provides Mirrored Object Bucket Claims (OBCs) in AWS
 name: aws-mirrored-openshift-storage.noobaa.io
parameters:
 bucketclass: aws-mirrored-bucket-class
provisioner: openshift-storage.noobaa.io/obc
reclaimPolicy: Delete
volumeBindingMode: Immediate

This StorageClass uses our BucketClass aws-mirrored-bucket-class as a backend. All buckets created leveraging this StorageClass will mirror data between aws-eu-north and aws-eu-west (see the previous chapter).

ObjectBucketClaim

Finally we are able to create ObjectBucketClaims for projects requiring object storage. An ObjectBucketClaim is similar to an PersistentVolumeClaim. Every time a claim is created the Noobaa operator will create a corresponding S3 bucket for us.

Let's start testing this out by creating a new OpenShift project

oc new-project obc-test

Now we define a ObjectBucketClaim to create a new bucket for our application:

apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
 labels:
 app: noobaa
 name: aws-mirrored-claim
spec:
 generateBucketName: aws-mirrored
 storageClassName: aws-mirrored-openshift-storage.noobaa.io

We use the StorageClass created in the previous step. This will create

a S3 Bucket in the requested StorageClass
a ConfigMap storing access information
a Secret storing credentials for accessing the S3 Bucket

For testing we will upload some data via s3cmd and use a pod to monitor data within the bucket.

Let's do the upload with s3cmd, we need the following config file:

[default]
check_ssl_certificate = False
check_ssl_hostname = False
access_key = <access key>
secret_key = <secret key>
host_base = s3-openshift-storage.apps.ocp.aws.tntinfra.net
host_bucket = %(bucket).s3-openshift-storage.apps.ocp.aws.tntinfra.net

Of course you must change host-base according to your cluster name. It's a route in the openshift-storage namespace:

oc get route -n openshift-storage s3 -o jsonpath='{.spec.host}'

You can extract the access and secret key from the K8s secret via:

oc extract secret/aws-mirrored-claim --to=-

Copy the access key and the secret key to the s3 command config file (we've called our config noobaa-s3.cfg). Now we can list all available buckets via:

$ s3cmd ls -c noobaa-s3.cfg
2022-04-22 13:56 s3://aws-mirrored-c1087a17-5c84-4c62-9f36-29081a6cf5a4

Now we are going to upload a sample file:

$ s3cmd -c noobaa-s3.cfg put simple-aws-mirrored-obc.yaml s3://aws-mirrored-c1087a17-5c84-4c62-9f36-29081a6cf5a4
upload: 'simple-aws-mirrored-obc.yaml' -> 's3://aws-mirrored-c1087a17-5c84-4c62-9f36-29081a6cf5a4/simple-aws-mirrored-obc.yaml' [1 of 1]
 226 of 226 100% in 0s 638.18 B/s done

We can also list available files via

s3cmd -c noobaa-s3.cfg ls s3://aws-mirrored-c1087a17-5c84-4c62-9f36-29081a6cf5a4
2022-04-22 13:57 226 s3://aws-mirrored-c1087a17-5c84-4c62-9f36-29081a6cf5a4/simple-aws-mirrored-obc.yaml

Our we could use a Pod to list available files from within OpenShift. Note how we use the ConfigMap and the Secret the Noobaa operater created for us, when we created the ObjectBucketClaim:

apiVersion: batch/v1
kind: Job
metadata:
 name: s3-test-job
spec:
 template:
 metadata:
 name: s3-pod
 spec:
 containers:
 - image: d3fk/s3cmd:latest
 name: s3-pod
 env:
 - name: BUCKET_NAME
 valueFrom:
 configMapKeyRef:
 name: aws-mirrored-claim
 key: BUCKET_NAME
 - name: BUCKET_HOST
 valueFrom:
 configMapKeyRef:
 name: aws-mirrored-claim
 key: BUCKET_HOST
 - name: BUCKET_PORT
 valueFrom:
 configMapKeyRef:
 name: aws-mirrored-claim
 key: BUCKET_PORT
 - name: AWS_ACCESS_KEY_ID
 valueFrom:
 secretKeyRef:
 name: aws-mirrored-claim
 key: AWS_ACCESS_KEY_ID
 - name: AWS_SECRET_ACCESS_KEY
 valueFrom:
 secretKeyRef:
 name: aws-mirrored-claim
 key: AWS_SECRET_ACCESS_KEY
 command:
 - /bin/sh
 - -c
 - 's3cmd --host $BUCKET_HOST --host-bucket "%(bucket).$BUCKET_HOST" --no-check-certificate ls s3://$BUCKET_NAME'
 restartPolicy: Never

That's all for now. If time allows we are going to write a follow up blog post on

Replicating Buckets and
Functions

Adventures in Java Land: JPA disconnected entities

Fri, 25 Feb 2022 00:00:00 +0000

An old man tries to refresh his Java skills and does DO378. He fails spectacularly at the first real example but learns a lot on the way.

The exception

There is this basic example where you build a minimal REST API for storing speaker data in a database. Quarkus makes this quite easy. You just have to define your database connection properties in resources/application.properties and off you go developing your Java Quarkus REST service:

quarkus.datasource.db-kind=h2
quarkus.datasource.jdbc.url=jdbc:h2:mem:default
quarkus.datasource.username=admin
quarkus.hibernate-orm.database.generation=drop-and-create
quarkus.hibernate-orm.log.sql=true

So next we define our entity class to be stored in the database. I will skip the import statements and any other code not relevant for this post.

// import statements skipped
@Entity
public class Speaker extends PanacheEntity {
 public UUID uuid;

 public String nameFirst;
 public String nameLast;
 public String organization;

 @JsonbTransient
 public String biography;
 public String picture;
 public String twitterHandle;

 // Constructors, getters and setters, toString and other methods skipped
 ....

}

We define an entity Speaker which extends the PanacheEntity class. Panache is a thin wrapper around Hibernate providing convince features. For example the base class PanacheEntity defines a autoincrement Id column for us. This inherited Id column is of importance for understanding the problem ahead of us.

So next you define your SpeakerService class which uses the entity. Once again I will skip the imports and any code not relevant for understanding the problem:

// imports omitted

@ApplicationScoped
public class SpeakerService {

 // other code omitted

 public Speaker create(Speaker speaker) {
 speaker.persist();
 return speaker;
 }

We focus on the create method here because the call to speaker.persist() was the reason for all the headache.

But we are still in coding mode and last but not least we define our SpeakerResource class, again everything not relevant for understanding the problem was removed:

// import statements omitted

@Path("/speaker")
@Produces(MediaType.APPLICATION_JSON)
@Consumes(MediaType.APPLICATION_JSON)
public class SpeakerResource {

 @Inject
 SpeakerService service;

 // other code omitted

 @POST
 @Transactional
 public Speaker create(Speaker newSpeaker) {
 service.create(newSpeaker);
 return newSpeaker;
 }
}

The root path for our SpeakerResource is /speaker. We inject the SpeakerService and define a method create() for creating a Speaker. We would like to be able to send @Post requests to this endpoint and Jsonb or Jackson, whichever we currently prefer, will deserialize the JSON body in a Speaker object for us.

Splendid, time to switch from coding mode to testing.

We launch that Quarkus application in developer mode

mvn quarkus:dev

Quarkus is so friendly and provides a swagger-ui in dev mode for testing our endpoint. Super duper lets call the create() endpoint via Swagger:

Because we are lazy we accept the default Swagger provides for us and just click Execute.

BOOM, 500 internal server error. And a beautiful Java exception:

org.jboss.resteasy.spi.UnhandledException: javax.persistence.PersistenceException: org.hibernate.PersistentObjectException: detached entity passed to persist: org.acme.conference.speaker.Speaker

What? Detached entity what does this mean and why?

Enlightenment

Behind the scenes Hibernate uses a so called EntityManager for managing entities. An Entity can be in the following states when managed by Hibernate:

NEW: The entity object was just created and is not persisted to the database
MANAGED: The entity is managed by a running Session and all changes to the entity will be propagated to the database. After call to entitymanager.persist() or in our case newSpeaker.persist() the entity is stored in the database and in the managed state.
REMOVED: The entity is removed from the database. And finally
DETACHED: The Entity was detached from the EntityManager, e.g. by calling entitymanager.detach() or entitymanager.close().

See this blog for a way better explanation what is going on with entity states.

Ok, cool but why the hell is our Speaker entity in the DETACHED state? It was just created and never saved to the database before!

After checking the database (was empty), I started my Java debugger of choice (IntellJ, but use whatever fit's your needs. I'm to old for IDE vs Editor and Editor vs Editor wars).

So looking at the Speaker entity before calling persist() revealed the following:

The Speaker object passed into create() has an Id of 0 and all the internal Hibernate fields are set to null. So this seems to indicate that this Speaker object is currently not attached to an EntityManager session. This might explain the DETACHED state.

I started playing around with EntityManager and calling merge() on the speaker object. The code looked like this:

@ApplicationScoped
public class SpeakerService {

 @Inject
 EntityManager em;

 // lots of code skipped

 public Speaker create(Speaker speaker) {
 var newSpeaker = em.merge(speaker);
 newSpeaker.persist();
 return speaker;
 }

Looking at the newSpeaker object returned by calling entitymanager.merge() in the debugger revealed the following:

newSpeaker has an Id of 1 (hm, why no 0?) and some those special Hibernate fields starting with $$ have a value assigned. So for me this indicates that the object is now managed by an EntityManager session and in the MANAGED state.

And the Id, already assigned to the original Speaker object, de-serialized form JSON is actually the reason for the beautiful exception above.

Explanation

So after a little bit of internet search magic I found an explanation for the exception:

If an Id is already assigned to an entity object, Hibernate assumes that this is an entity in the DETACHED state (if the Id is auto-generated). For an entity to be persisted to the database it has to be transferred in the MANAGED state by calling entitymanager.merge()

For more information see the Hibernate documentation.

We can only call persist() if the object is in the transient state, to quote the Hibernate documentation:

transient: the entity has just been instantiated and is not associated with a persistence context. It has no persistent representation in the database and typically no identifier value has been assigned (unless the assigned generator was used).

And reading on we also get explanation for the detached state:

detached: the entity has an associated identifier but is no longer associated with a persistence context (usually because the persistence context was closed or the instance was evicted from the context)

Just removing the Id from the POST request will solve the issue and the example started to work.

This is also why the Id column is different in the Speaker object (deserialized from JSON) and newSpeaker object (create by calling entitymanager.merge()). The Speaker Id got passed in from JSON, and has nothing to do with the auto generated primary key Id within our database. After calling entitymanager.merge() the entity is actually associated with a database session and the Id is auto generated.

So maybe this is basic stuff, but it took me quite a few hours to understand what was going on.

Maybe this is also a bad example. Should one expose the Id if it is auto generated and only used internally? Or the code just needs to handle that case… But this needs me more learning about API design.

Stumbling into Azure Part II: Setting up a private ARO cluster

Fri, 29 Oct 2021 00:00:00 +0000

In Part I of our blog post we covered setting up required resources in Azure. Now we are finally going to set up a private cluster. Private

As review from Part I here is our planned setup, this time including the ARO cluster.

Azure Setup

The diagram below depicts our planned setup:

On the right hand side can see the resources required for our lab:

a virtual network (vnet 192.168.128.0/19). This vnet will be split into 3 separate subnets
a master subnet (192.168.129.0/24) holding the ARO control plane nodes
a node subnet (192.168.130.0/24) holding ARO worker nodes
and finally a subnet call GatewaySubnet where we are going to deploy our Azure VPN gateway (called a vnet-gateway)
The subnet where the Azure VPN gateway is located needs to have the name GatewaySubnet. Otherwise creating the Azure VPN gateway will fail.
we also need a publicIP resource that we are going to connect to our vnet-gateway (the VPN gateway)
and finally a local-gateway resource that tells the vnet-gateway which networks are reachable on the left, in our case the Hetzner server.

Creating the private Azure Red Hat OpenShift cluster

az provider register -n Microsoft.RedHatOpenShift --wait
az provider register -n Microsoft.Compute --wait
az provider register -n Microsoft.Storage --wait
az provider register -n Microsoft.Authorization --wait

First we are going to set some environment variable. Those variables are used in the upcoming commands:

export RESOURCEGROUP=aro-rg
export CLUSTER="aro1"
export GATWAY_SUBNET="192.168.128.0/24"
export MASTER_SUBNET="192.168.129.0/24"
export WORKER_SUBNET="192.168.130.0/24"
export HETZNER_VM_NETWORKS="10.0.0.0/24 192.168.122.0/24 172.16.100.0/24"

Disable subnet private endpoint policies

az network vnet subnet update \
--name master-subnet \
--resource-group $RESOURCEGROUP \
--vnet-name aro-vnet \
--disable-private-link-service-network-policies true

Create a private DNS zone for our cluster

az network private-dns zone create -n private2.tntinfra.net -g aro-rg

Create the cluster

az aro create \
--resource-group $RESOURCEGROUP \
--name $CLUSTER \
--vnet aro-vnet \
--master-subnet master-subnet \
--worker-subnet worker-subnet \
--apiserver-visibility Private \
--ingress-visibility Private \
--domain private.tntinfra.net
# --pull-secret @pull-secret.txt # [OPTIONAL]

After successful cluster creating add DNS entry for the API and Ingress

Query the Azure API for the API server IP and the ingress IP addresses:

az aro show -n aro1 -g aro-rg --query '{api:apiserverProfile.ip, ingress:ingressProfiles[0].ip}'

Example output

Api Ingress
------------- ---------------
192.168.129.4 192.168.130.254

Add entries to Azure private DNS

az network private-dns record-set a add-record -g aro-rg -z private.tntinfra.net -a "192.168.129.4" -n api
az network private-dns record-set a add-record -g aro-rg -z private.tntinfra.net -a "192.168.130.254" -n "*.apps"

List entries to verify configuration

az network private-dns record-set a list -g aro-rg -z private.tntinfra.net

Output:

Name ResourceGroup Ttl Type AutoRegistered Metadata
------ --------------- ----- ------ ---------------- ----------
api aro-rg 3600 A False
*.apps aro-rg 3600 A False

List cluster credentials after successful setup

az aro list-credentials \
--name $CLUSTER \
--resource-group $RESOURCEGROUP

Get the console URL

az aro show \
--name $CLUSTER \
--resource-group $RESOURCEGROUP \
--query "consoleProfile.url" -o tsv

DNS, curl

this works, dunno why?

dig @192.168.129.7 console-openshift-console.apps.xm7rdz4r.westeurope.aroapp.io

use curl to access the internal API and see if it works:

curl -kv https://192.168.129.4:6443

Additional Resources

Stumbling into Azure Part I: Building a site-to-site VPN tunnel for testing

Sat, 16 Oct 2021 00:00:00 +0000

So we want to play with ARO (Azure Red Hat OpenShift) private clusters. A private cluster is not reachable from the internet (surprise) and is only reachable via a VPN tunnel from other networks.

This blog post describes how we created a site-to-site VPN between a Hetzner dedicated server running multiple VM's via libvirt and Azure.

An upcoming blog post is going to cover the setup of the private ARO cluster.

Azure Setup

The diagram below depicts our planned setup:

On the right hand side can see the resources required for our lab:

a virtual network (vnet 192.168.128.0/19). This vnet will be split into 3 separate subnets
a master subnet (192.168.129.0/24) holding the ARO control plane nodes
a node subnet (192.168.130.0/24) holding ARO worker nodes
and finally a subnet call GatewaySubnet where we are going to deploy our Azure VPN gateway (called a vnet-gateway)
The subnet where the Azure VPN gateway is located needs to have the name GatewaySubnet. Otherwise creating the Azure VPN gateway will fail.
we also need a publicIP resource that we are going to connect to our vnet-gateway (the VPN gateway)
and finally a local-gateway resource that tells the vnet-gateway which networks are reachable on the left, in our case the Hetzner server.

Creating the required Azure resources

First we are going to set some environment variable. Those variables are used in the upcoming commands:

export RESOURCEGROUP=aro-rg
export GATWAY_SUBNET="192.168.128.0/24"
export MASTER_SUBNET="192.168.129.0/24"
export WORKER_SUBNET="192.168.130.0/24"
export HETZNER_VM_NETWORKS="10.0.0.0/24 192.168.122.0/24 172.16.100.0/24"

Next create a VNET resource holding our sub networks:

az network vnet create \
--resource-group $RESOURCEGROUP \
--name aro-vnet \
--address-prefixes 192.168.128.0/18

Create the GatewaySubnet subnet

az network vnet subnet create \
--resource-group $RESOURCEGROUP \
--vnet-name aro-vnet \
--name GatewaySubnet \
--address-prefixes $GATEWAY_SUBNET

Create the master subnet

az network vnet subnet create \
--resource-group $RESOURCEGROUP \
--vnet-name aro-vnet \
--name master-subnet \
--address-prefixes $MASTER_SUBNET \
--service-endpoints Microsoft.ContainerRegistry

Create the worker subnet

az network vnet subnet create \
--resource-group $RESOURCEGROUP \
--vnet-name aro-vnet \
--name worker-subnet \
--address-prefixes $WORKER_SUBNET \
--service-endpoints Microsoft.ContainerRegistry

Create a public IP resource

az network public-ip create \
--name GatewayIP \
--resource-group $RESOURCEGROUP \
--allocation-method Dynamic

Create a local-gateway resource

az network local-gateway create \
--name playground \
--resource-group $RESOURCEGROUP \
--local-address-prefixes $HETZNER_VM_NETWORKS \
--gateway-ip-address 95.217.42.98

Create a vnet-gateway resource (takes around 30 minutes)

az network vnet-gateway create \
--name vpn-gateway \
--public-ip-address GatewayIP \
--resource-group $RESOURCEGROUP \
--vnet aro-vnet \
--gateway-type Vpn \
--vpn-type RouteBased \
--sku Basic \
--no-wait

Define a vpn-connection

az network vpn-connection create \
--name VNet1toSite2 \
--resource-group $RESOURCEGROUP \
--vnet-gateway1 vpn-gateway \
--local-gateway2 playground \
--location westeurope \
--shared-key thepassword

Required iptables (nf tables) hacks for libvirt

Skip NAT rules if the destination network is in Azure and the client network deploy via libvirt

iptables -I LIBVIRT_PRT 2 -t nat -d 192.168.129.0/24 -j RETURN
iptables -I LIBVIRT_PRT 2 -t nat -d 192.168.130.0/24 -j RETURN

Skip NAT rules if the destination network is in Azure and the client is connected via tailscale

iptables -I ts-postrouting 1 -t nat -d 192.168.129.0/24 -j RETURN
iptables -I ts-postrouting 1 -t nat -d 192.168.130.0/24 -j RETURN

Libreswan setup on CentOS Stream

Install the Libreswan packages
```
dnf install libreswan
```

Create a Azure configuration for Libreswan in ~/etc/ipsec.d/azure.conf

conn masterSubnet
also=azureTunnel
leftsubnet=192.168.129.0/24
rightsubnet=172.16.100.0/24
auto=start

conn workerSubnet
also=azureTunnel
leftsubnet=192.168.130.0/24
rightsubnet=172.16.100.0/24
auto=start

conn azureTunnel
authby=secret
auto=start
dpdaction=restart
dpddelay=30
dpdtimeout=120
ike=aes256-sha1;modp1024
ikelifetime=3600s
ikev2=insist
keyingtries=3
pfs=yes
phase2alg=aes128-sha1
left=51.137.113.44
leftsubnets=192.168.128.0/24
right=%defaultroute
rightsubnets=172.16.100.0/24
salifetime=3600s
type=tunnel
ipsec-interface=yes

Create a Libreswan secrets file for Azure in /etc/ipsec.d/azure.secrets:
```
%any %any : PSK "abc123"
```
Enable and start the IPsec service
```
systemctl enable --now ipsec
```

We had to explicitly load the IPsec configuration via

ipsec addconn --config /etc/ipsec.d/azure.conf azureTunnel

Libreswan IPSEC debugging tips

Check the state of the IPsec systemd service
```
systemctl status ipsec
```
Check the full log of the IPsec systemd service
```
journalctl -e -u ipsec
```

Check the state of the tunnels with the ipsec command line tool

ipsec status

Check for the following lines

000 Total IPsec connections: loaded 5, active 2
000
000 State Information: DDoS cookies not required, Accepting new IKE connections
000 IKE SAs: total(1), half-open(0), open(0), authenticated(1), anonymous(0)
000 IPsec SAs: total(2), authenticated(2), anonymous(0)
000
000 #130: "azureTunnel/1x1":500 STATE_V2_ESTABLISHED_CHILD_SA (IPsec SA established); EVENT_SA_REKEY in 2003s; newest IPSEC; eroute owner; isakmp#131; idle;
000 #130: "azureTunnel/1x1" esp.56cf4304@51.137.113.44 esp.6f49e8d3@95.217.42.98 tun.0@51.137.113.44 tun.0@95.217.42.98 Traffic: ESPin=0B ESPout=0B! ESPmax=0B
000 #129: "masterSubnet/0x0":500 STATE_V2_ESTABLISHED_CHILD_SA (IPsec SA established); EVENT_SA_REKEY in 1544s; newest IPSEC; eroute owner; isakmp#131; idle;
000 #129: "masterSubnet/0x0" esp.6e81e8da@51.137.113.44 esp.6f72bbc8@95.217.42.98 tun.0@51.137.113.44 tun.0@95.217.42.98 Traffic: ESPin=0B ESPout=0B! ESPmax=0B
000 #131: "masterSubnet/0x0":500 STATE_V2_ESTABLISHED_IKE_SA (established IKE SA); EVENT_SA_REKEY in 2121s; newest ISAKMP; idle;

IPsec specifies properties of connections via security associations (SA). The parent SA is describes the IKEv2 connections, the child SA is the ESP (encapsulated security payload) connection.

Check IPsec transformation policies

ip xfrm policy

Check the state of IPsec transformation policies

ip xfrm state

Check for dropped packages on the IPsec interface (ipsec1 in our case)

ip -s link show dev ipsec1

Additonal Resources

Stumbling into Quay: Upgrading from 3.3 to 3.4 with the quay-operator

Sat, 16 Oct 2021 00:00:00 +0000

We had the task of answering various questions related to upgrading Red Hat Quay 3.3 to 3.4 and to 3.5 with the help of the quay-operator.

Thankfully (sic!) everything changed in regards to the Quay operator between Quay 3.3 and Quay 3.4.

So this is a brain dump of the things to consider.

Operator changes

With Quay 3.4 the operator was completely reworked and it basically changed from opinionated to very opinionated. The upgrade works quite well but you have to be aware about the following points:

The name of the custom resource changed from QuayEcosystem to QuayRegistry
The configHostname, used for providing the quay configuration UI, is no longer configurable
The password for the configuration UI is always regenerated after a configuration re-deployment
The volume size of the PostgreSQL PVC will change to 50G

In my test cluster I was using a 10G Ceph block device and the StorageClass did not support volume expansion. So my upgrade stopped at this point and I had to allow volume expansion in the storage class.

A horizontal pod autoscaler is also deploy during the upgrade. The default is to scale automatically to 20 pods, you might reconsider this…
With the Quay operator version 3.5 the operator is monitoring all namespaces for custom resources and needs to be installed in the openshift-operators namespace, this is how we upgraded Quay to 3.5 including the operator:
- change the quay operator channel to 3.5
- trigger an upgrade
- now Quay gets upgraded to version 3.5
- after the Quay upgrade you need to reinstall the operator:
  - deinstall the quay operator, Quay is not affected by this
  - reinstall the Quay operator (3.5) in all-namespaces
  - the re-installation of the operator triggers a quay deployment, all Quay pods are restarted!
You have to manually cleanup old
- postgres-config-secrets and
- quay-config-bundle's

Backup and restore considerations

Red Hat is working on providing documentation on how to backup and restore Quay in various scenarios. There's an open task for this that provides more information https://issues.redhat.com/browse/PROJQUAY-2242.

Ansible Tower and downloading collections

Sat, 31 Jul 2021 00:00:00 +0000

Every wondered why Ansible Tower does not start downloading required collections when you synchronize a project? Here are the stumbling blocks we discovered so far:

Wrong name for requirements.yml

When downloading collections Ansible Tower searches for a file requirements.yml in the collections directory.

Be careful with the file extension: requirements.yml has to end with the extension .yml and not .yaml.

Collections download is disabled in Ansible Tower

Within Ansible Tower there is a setting called ENABLE COLLECTION(S) DOWNLOAD under Settings/Jobs. This has to be set to true, which is also the default.

No Ansible Galaxy credential defined for the organization

Last but not least an Ansible Galaxy credential needs to be defined for the organization where the project is defined. With the default installation of Ansible Tower, when the sample playbooks are installed there is a credential called Ansible Galaxy defined. You need to assign this credential to the organization.

If you skip installing the sample playbooks, no Ansible Galaxy credential will be defined for you and you have to create it manually.

How does this actually work?

Ansible Tower uses a Python virtual environment for running Ansible. The default environment is installed in /var/lib/awx/venv/awx. You can also create custom environments, see Using virtualenv with Ansible Tower.

In the default setup the following files define how collections are downloaded:

lib/python3.6/site-packages/awx/main/tasks.py
lib/python3.6/site-packages/awx/playbooks/project_update.yml

task.py

task.py defines various internal tasks Tower has to run on various occasions. For example in line number 1930 (Ansible Tower 3.8.3) the task RunProjectUpdate gets defined. This is the task Tower has to run whenever a project update is required.

In our case the function build_extra_vars_file (line 2083 with Ansible Tower 3.8.3) defines the variable galaxy_creds_are_defined only if the organization has a galaxy credential defined (line 2099 Ansible Tower 3.8.3).

Line 2120 (Ansible Tower 3.8.3) finally defines the Ansible extra variable collections_enabled depending on galaxy_creds_are_defined.

project_update.yml

So task.py defines the extra variable collections_enabled (see above). Finally the playbook project_update.yml consumes this extra variable and only downloads collections if collections_enabled is set to true, see the block string at line 192 (Ansible Tower 3.8.3) in `project_update.yml.

So long and thanks for all the fish!

Understanding RWO block device handling in OpenShift

Sat, 27 Feb 2021 00:00:00 +0000

In this blog post we would like to explore OpenShift / Kubernetes block device handling. We try to answer the following questions:

What happens if multiple pods try to access the same block device?
What happens if we scale a deployment using block devices to more than one replica?

And finally we want to give a short, high level overview about how the container storage interface (CSI) actually works.

A block device provides Read-Write-Once (RWO) storage. This basically means a local file system mounted by a single node. Do not confuse this with a cluster (CephFS, GlusterFS) or network file system (NFS). These file systems provide Read-Write-Many (RWX) storage mountable on more than one node.

Test setup

For running our tests we need the following resources

A new namespace/project for running our tests
A persistent volume claim (PVC) to be mounted in our test pods
Two pods definitions for mounting the PVC

Step 1: Creating a new namespace/project

To run our test cases we created a new project with OpenShift

oc new-project blockdevices

Step 2: Defining a block PVC

Our cluster is running the rook operator (https://rook.io) and provides a ceph-block storage class for creating block devices:

$ oc get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
ceph-block rook-ceph.rbd.csi.ceph.com Delete Immediate false 4d14h

Let’s take a look a the details of the storage class:

$ oc get sc -o yaml ceph-block
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ceph-block
parameters:
clusterID: rook-ceph
csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
csi.storage.k8s.io/fstype: ext4 (1)
csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
imageFeatures: layering
imageFormat: "2"
pool: blockpool
provisioner: rook-ceph.rbd.csi.ceph.com
reclaimPolicy: Delete
volumeBindingMode: Immediate

1	So whenever we create a PVC using this storage class the Ceph provisioner will also create an EXT4 file system on the block device.

To test block device handling we create the following persistent volume claim (PVC):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: block-claim
spec:
accessModes:
- ReadWriteOnce (1)
resources:
requests:
storage: 1Gi
storageClassName: ceph-block

1	The access mode is set to ReadWriteOnce (RWO), as block devices

oc create -f pvc.yaml

$ oc get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
block-claim Bound pvc-bd68be5d-c312-4c31-86a8-63a0c22de844 1Gi RWO ceph-block 91s

To test our shiny new block device we are going to use the following three pod definitions:

block-pod-a

apiVersion: v1
kind: Pod
metadata:
labels:
run: block-pod-a
name: block-pod-a
spec:
containers:
- image: registry.redhat.io/ubi8/ubi:8.3
name: block-pod-a
command:
- sh
- -c
- 'df -h /block && findmnt /block && sleep infinity'
volumeMounts:
- name: blockdevice
mountPath: /block
volumes:
- name: blockdevice
persistentVolumeClaim:
claimName: block-claim

block-pod-b

apiVersion: v1
kind: Pod
metadata:
labels:
run: block-pod-b
name: block-pod-b
spec:
affinity:
podAntiAffinity: (1)
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: run
operator: In
values:
- block-pod-a
topologyKey: kubernetes.io/hostname
containers:
- image: registry.redhat.io/ubi8/ubi:8.3
name: block-pod-b
command:
- sh
- -c
- 'df -h /block && findmnt /block && sleep infinity'
volumeMounts:
- name: blockdevice
mountPath: /block
volumes:
- name: blockdevice
persistentVolumeClaim:
claimName: block-claim

1	We use an AntiAffinity rule for making sure that block-pod-b runs on a different node than block-pod-a.

block-pod-c

apiVersion: v1
kind: Pod
metadata:
labels:
run: block-pod-c
name: block-pod-c
spec:
affinity:
podAffinity: (1)
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: run
operator: In
values:
- block-pod-a
topologyKey: kubernetes.io/hostname
containers:
- image: registry.redhat.io/ubi8/ubi:8.3
name: block-pod-c
command:
- sh
- -c
- 'df -h /block && findmnt /block && sleep infinity'
volumeMounts:
- name: blockdevice
mountPath: /block
volumes:
- name: blockdevice
persistentVolumeClaim:
claimName: block-claim

1	We use an Affinity rule for making sure that block-pod-c runs on the same node as block-pod-a.

In our first test we want to make sure that both pods are running on separate cluster nodes. So we create block-pod-a and block-pod-b:

$ oc create -f block-pod-a.yml
$ oc create -f block-pod-b.yml

After a few seconds we can check the state of our pods:

$ oc get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
block-pod-a 1/1 Running 0 46s 10.130.6.4 infra02.lan.stderr.at <none> <none>
block-pod-b 0/1 ContainerCreating 0 16s <none> infra01 <none> <none>

Hm, block-pod-b is in the state ContainerCreating, let’s check the events. Also note that it is running on another node (infra01) then block-pod-a (infra02).

10s Warning FailedAttachVolume pod/block-pod-b Multi-Attach error for volume "pvc-bd68be5d-c312-4c31-86a8-63a0c22de844" Volume is already used by pod(s) block-pod-a

Ah, so because of our block device with RWO access mode and block-pod-b running on separate cluster node, OpenShift or K8s can’t attach the volume to our block-pod-b.

But let’s try another test and let’s create a third pod block-pod-c that should run on the same node as block-pod-a:

$ oc create -f block-pod-c.yml

Now let’s check the status of block-pod-c:

$ oc get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
block-pod-a 1/1 Running 0 6m49s 10.130.6.4 infra02.lan.stderr.at <none> <none>
block-pod-b 0/1 ContainerCreating 0 6m19s <none> infra01 <none> <none>
block-pod-c 1/1 Running 0 14s 10.130.6.5 infra02.lan.stderr.at <none> <none>

Oh, block-pod-c is running on node infra02 and mounted the RWO volume. Let’s check the events for block-pod-c:

3m6s Normal Scheduled pod/block-pod-c Successfully assigned blockdevices/block-pod-c to infra02.lan.stderr.at
2m54s Normal AddedInterface pod/block-pod-c Add eth0 [10.130.6.5/23]
2m54s Normal Pulled pod/block-pod-c Container image "registry.redhat.io/ubi8/ubi:8.3" already present on machine
2m54s Normal Created pod/block-pod-c Created container block-pod-c
2m54s Normal Started pod/block-pod-c Started container block-pod-c

When we compare this with the events for block-pod-a:

9m41s Normal Scheduled pod/block-pod-a Successfully assigned blockdevices/block-pod-a to infra02.lan.stderr.at
9m41s Normal SuccessfulAttachVolume pod/block-pod-a AttachVolume.Attach succeeded for volume "pvc-bd68be5d-c312-4c31-86a8-63a0c22de844"
9m34s Normal AddedInterface pod/block-pod-a Add eth0 [10.130.6.4/23]
9m34s Normal Pulled pod/block-pod-a Container image "registry.access.redhat.com/ubi8/ubi:8.3" already present on machine
9m34s Normal Created pod/block-pod-a Created container block-pod-a
9m34s Normal Started pod/block-pod-a Started container block-pod-a

So the AttachVolume.Attach message is missing in the events for block-pod-c. Because the volume is already attached to the node, interesting.

Even with RWO block device volumes it is possible to use the same volume in multiple pods if the pods a running on the same node.

I was not aware of this possibility and always had the believe with an RWO block device only one pod can access the volume. That’s the problem with believing :-)

Thanks or reading this far.

Basic usage of git

Fri, 08 May 2020 00:00:00 +0000

This is a very short and hopefully simple introduction on how to use Git when you would like to contribute to projects hosted on github.com. The same workflow should also work for projects on gitlab.com.

Introduction

There is this fancy mega application hosted on github called megaapp that you would like to contribute to. It’s perfect but there’s just this little feature missing to make it even more perfect.

This is how we would tackle this.

rocket science ahead

Glossary

Term	Definition
fork	A (personal) copy of a repository you created on github or gitlab.
upstream	When creating forks of repositories on github or gitlab, the original repository hosting the project
index	The staging area git uses before you can commit to a repository
remote repository	A repository hosted on a server shared by developers
local repository	A local copy of a repository stored on you machine.

Term

Definition

fork

A (personal) copy of a repository you created on github or gitlab.

upstream

When creating forks of repositories on github or gitlab, the original repository hosting the project

index

The staging area git uses before you can commit to a repository

remote repository

A repository hosted on a server shared by developers

local repository

A local copy of a repository stored on you machine.

Step 1: Fork the repository on github.com

Click on the the fork button, as depicted in the image below:

If you are a member of several projects on github.com, github is going to ask you into which project you would like to clone this repository.

After selecting the project or your personal account, github is going to clone the repository into the project you selected. For this example I’m going to use my personal github account "tosmi".

Step 2: Clone the repository to you workstation

Next we are going to clone our fork from Step 1: Fork the repository on github.com to our workstation and start working on the new feature.

After forking the upstream project you are redirect to your personal copy of the project. Click on the "Clone or download" button and select the link. You can choose between SSH and HTTPS protocols for downloading the project. We are going to use SSH.

Copy the link into a terminal and execute the git clone command:

$ git clone git@github.com:tosmi/megaapp.git

Step 3: Create a feature branch for your new fancy feature

Change into the directory of the project you downloaded in Step 2: Clone the repository to you workstation

cd megaapp

Now we create a feature branch with a short name that describes our new feature:

git checkout -b tosmi/addoption

Because we would like to add a new option to megaapp we call this feature branch addoption.

We are also prefixing the feature branch with our github username so that it is clear for the upstream project maintainer(s) who is contributing this.

How you name you branches is opinionated, so we would search for upstream project guidelines and if there are none maybe look at some existing pull request how other people are naming there branches. If we find no clue upstream we sticking with <github username>/<branch name>.

We can now start adding our mega feature to the project.

Step 4: Add you changes to the Git index

Before we can commit our changes, we have to place the changes made in the so called index or staging area:

$ git add <path to file you have changed>

If we would like to place all of our changes onto the index we could execute

$ git add -A

Step 5: Commit your changes

After adding our changes to the Git index we can commit with

$ git commit

This will open our favorite editor and we can type a commit message. The first line should be a short description of our change, probably not longer than 70 to 80 characters. After two newlines we can enter a detailed explanation of your changes.

This is an example commit message

Added a `version` option to output the current version of megaapp
This change introduces a `version` option to megaapp. The purpose is
to output the current version of megaapp for users. This might be
helpful when users open a bug report so we can see what version is
affected.

After saving the message and we have successfully created a commit.

Remember this is now only stored in the local copy of the repository! We still have to push our changes to github.

There is also the option to add the commit comment directly on the command line

$ git commit -m 'Added a `version` option to output the current version of megaapp
This change introduces a `version` option to megaapp. The purpose is
to output the current version of megaapp for users. This might be
helpful when users open a bug report so we can see what version is
affected.'

Step 6: Pushing our local changes to our forked repo on github.com

We execute

$ git push

to push our local changes to the forked repository hosted on github.com.

Step 7: Creating a pull request on github.com

We navigate to our personal project page of the forked repository on github. For the fork we are using in this example this is http://github.com/tosmi/megaapp.

Github is going to show us a button "Compare & pull request":

After clicking on that button we are able to review the changes we would like to include in this pull request.

If we are happy with our changes we click on "Create pull request". The upstream owner of the repository will get notified and we can see our open pull request on the upstream project page under "Pull requests".

If there are CI test configured for that project they will start to run and we can see if our pull request is going to pass all test configured.

Rebasing to current upstream if required

Sometimes a upstream project maintainer asks you to rebase your work on the current upstream master branch. The following steps explain the basic workflow.

First we are going to create a new remote location of our repository called upstream. Upstream points to the upstream project repository. We will not push to this location, in most cases this is not possible because you do not have write access to a remote upstream repository. It is just used for pulling upstream changes in our forked repository.

Execute the following commands to add the upstream repository as a new remote location and display all remote locations currently defined.

$ git remote add upstream https://github.com/rhatservices/megaapp.git
$ git remote -v origin
git@github.com:tosmi/megaapp.git (fetch) origin
git@github.com:tosmi/megaapp.git (push) upstream
https://github.com/rhatservices/megaapp.git (fetch) upstream
https://github.com/rhatservices/megaapp.git (push)

As we hopefully implemented our new feature in feature branch, we can pull changes from the upstream master branch into our local copy of the master branch. Remember we are using a feature branch and master should be kept clean from local changes.

$ git checkout master
Switched to branch 'master'
Your branch is up to date with 'origin/master'.

So now we have this older copy of the upstream master branch checked out and we would like to update it to the latest and greatest from the upstream master branch.

$ git pull upstream master
remote: Enumerating objects: 10, done.
remote: Counting objects: 100% (10/10), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 6 (delta 2), reused 6 (delta 2), pack-reused 0
Unpacking objects: 100% (6/6), 630 bytes | 157.00 KiB/s, done.
From https://github.com/rhatservices/megaapp
* branch master -> FETCH_HEAD
* [new branch] master -> upstream/master
Updating 4d8584e..ddfd077
Fast-forward
cmd/megaapp/main.go | 2 ++
cmd/megaapp/rule.go | 20 ++++++++++++++++++++
2 files changed, 22 insertions(+)
create mode 100644 cmd/megaapp/rule.go

With the pull command above you pulled all changes from the upstream master branch into you local copy of master. Just to be sure let’s display all available branches, local and remote ones.

Branches with a name remote/<remote name>/<branch name> are remote branches that git knows about. Origin points to our forked repository and is also the default location for push operations.

$ git branch -a
master
* tosmi/megafeature
remotes/origin/HEAD -> origin/master
remotes/origin/master
remotes/origin/tosmi/megafeature
remotes/upstream/master

So finally to rebase our feature branch to the upstream master branch we first need to checkout our feature branch via

$ git checkout tosmi/megafeature

Now we are able to rebase our changes to upstream master. Git basically pulls in all changes from the master branch and re-applies the changes we did in our feature branch.

git rebase upstream/master
Successfully rebased and updated refs/heads/tosmi/megafeature.

There might be merge conflicts when git tries to apply you changes from your feature branch. You have to fix those changes, git add the fixed files and execute git rebase continue. Luckily this is not the case for your megafeature.

As we have successfully rebased our feature branch to upstream master we can now try to push changes made to our forked github repository.

$ git push
To github.com:tosmi/megaapp.git
! [rejected] tosmi/megafeature -> tosmi/megafeature (non-fast-forward)
error: failed to push some refs to 'git@github.com:tosmi/megaapp.git'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

Oh, this fails of course! The reason is that our local feature branch and the remote feature branch have a different commit history. The remote feature branch is missing the commits from master that we applied when rebasing on the current master branch.

So let’s try again, this time using the --force-with-lease option. You could also use -f or --force but --force-with-lease will stop you if someone else (our you) has modified the remote feature branch meanwhile. If you push with -f or --force anyways you might loose changes.

$ git push --force-with-lease
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 8 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 295 bytes | 295.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To github.com:tosmi/megaapp.git
+ acf66a3...39357b2 tosmi/megafeature -> tosmi/megafeature (forced update)

But as no one modified the remote feature branch while we did our rebase the force push goes through.

Our merge request (if we opened one already) is now updated to the latest upstream master branch and merging our feature should be a breeze. You might notify the upstream project maintainer that you feature branch is up to date and ready for merging

Using git’s interactive rebase to change you commit history

When working with upstream projects it might be that a project maintainer requests that you rework your git history before he is willing to merge your changes. For example this could be that case if you have plenty of commits with very small changes (e.g. fixed typos).

The general rule is that one commit should implement one change. This is not a hard rule, but usually works.

Let’s look at an example. For the implementation of our new feature that we would like to bring upstream we have the following commit history

$ git log --oneline
0a5221d (HEAD -> tosmi/megafeature) fixed typo
0e60d12 update README
bf2ef3c update

We have updated README.md in the repository but there a three commits for this little change. Before bringing this upstream in our pull request, we would like to convert those three commits into a single one and also make the commit message a little more meaningful.

We execute the following command to start reworking our commit history

$ git rebase -i

Git will drop us into our beloved editor (vi in this case), under Linux you could change the editor git uses by modifying the $EDITOR environment variable. We are going to see the following output:

pick bf2ef3c update
pick 0e60d12 update README
pick 0a5221d fixed typo
# Rebase 39357b2..0a5221d onto 39357b2 (3 commands)
#
# Commands:
# p, pick <commit> = use commit
# r, reword <commit> = use commit, but edit the commit message
# e, edit <commit> = use commit, but stop for amending
# s, squash <commit> = use commit, but meld into previous commit
# f, fixup <commit> = like "squash", but discard this commit's log message
# x, exec <command> = run command (the rest of the line) using shell
# b, break = stop here (continue rebase later with 'git rebase --continue')
# d, drop <commit> = remove commit
# l, label <label> = label current HEAD with a name
# t, reset <label> = reset HEAD to a label
# m, merge [-C <commit> | -c <commit>] <label> [# <oneline>]
# . create a merge commit using the original merge commit's
# . message (or the oneline, if no original merge commit was
# . specified). Use -c <commit> to reword the commit message.
#
# These lines can be re-ordered; they are executed from top to bottom.
#
# If you remove a line here THAT COMMIT WILL BE LOST.
#
# However, if you remove everything, the rebase will be aborted.
#

Git automatically selected commit id bf2ef3c as the basis for our rebase. We could also have specified the commit id where we would like to start our rebase operation e.g.

git rebase -i bf2ef3c

In our editor of choice we can now tell git what it should do with the selected commits. Please go ahead and read the helpfull explanation text in comments (prefixed with '#') to get a better understanding of the operations supported.

In our case we would like to squash the last commits. So we change the lines with pick to squash until it looks like the following:

pick bf2ef3c update
squash 0e60d12 update README
squash 0a5221d fixed typo

We would like to squash commits 0a5221d and 0e60d12 onto commit bf2ef3c. Keep in mind that git actually reverses the order of commits. So 0a5221d is the last commit we added.

If we save the file and quit our editor (I’m using vi here), git drops us into another buffer where we can finally modify the commits

 This is a combination of 3 commits.
# This is the 1st commit message:
update
# This is the commit message #2:
update README
# This is the commit message #3:
fixed typo
# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
#
# Date: Mon May 18 15:46:37 2020 +0200
#
# interactive rebase in progress; onto 39357b2
# Last commands done (3 commands done):
# squash 0e60d12 update README
# squash 0a5221d fixed typo
# No commands remaining.
# You are currently rebasing branch 'tosmi/megafeature' on '39357b2'.
#
# Changes to be committed:
# modified: README.md
#

We can see all three commit message and we are going to modify those messages until we are happy

# This is a combination of 3 commits.
# This is the 1st commit message:
updated README.md to megafeature
as we added megafeature, it makes sense to include a short note about it also in README.md
# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
#
# Date: Mon May 18 15:46:37 2020 +0200
#
# interactive rebase in progress; onto 39357b2
# Last commands done (3 commands done):
# squash 0e60d12 update README
# squash 0a5221d fixed typo
# No commands remaining.
# You are currently rebasing branch 'tosmi/megafeature' on '39357b2'.
#
# Changes to be committed:
# modified: README.md
#

When we are happy with new commit message we just save and quit our editor. Git will now rewirte the history and when we take look at the commit history again we will see our changes:

$ git log --oneline
91d1ae2 (HEAD -> tosmi/megafeature) updated README.md to megafeature
39357b2 (origin/tosmi/megafeature) added a mega feature
ddfd077 (upstream/master, master) added rule command
4d8584e (origin/master, origin/HEAD) Update README.md
eb6ccbc Create README.md
60fcabc start using cobra for argument parsing
5140ed0 import .gitignore
d2b55d1 import a simple Makefile
2ecb412 initial import

We only have commit 91d1ae2 now , which includes all three changes from the commits before.

Rewriting the history of a repository is a dangerous operation. Especially when you are working in a team. It is not advised to change the history of commits that got already pushed to a remote location. Otherwise your teammates will get confused next time they try to push or pull from the shared repository.

So it’s OK to change the commit history of a feature branch that only you are using, but be careful when working on branches more than one developer is using.

Red Hat Satellite Cheat Sheet

Wed, 15 Apr 2020 00:00:00 +0000

Cheat sheet for various Red Hat Satellite tasks from a newbie to a newbie.

Requirements

up to Satellite 6.7 RHEL 7.X
4 CPU Cores
20 GB of RAM
300 GB disk space

for more info see the prerequistes guide

Installation

Satellite up to version 6.7 uses puppet for installation. You can use

puppet filebucket

to restore files modified by puppet.

Satellite requires the Red Hat Satellite Infrastructure Subscription, check if it’s available with

subscription-manager list --all --available --matches 'Red Hat Satellite Infrastructure Subscription'

If not attach it with

subscription-manager attach --pool=pool_id

Next disable all repos and enable only supported repostories via

subscription-manager repos --disable "*"

and enable required repositories

subscription-manager repos --enable=rhel-7-server-rpms \
--enable=rhel-7-server-satellite-6.6-rpms \
--enable=rhel-7-server-satellite-maintenance-6-rpms \
--enable=rhel-server-rhscl-7-rpms \
--enable=rhel-7-server-ansible-2.8-rpms

then clean cached all repo data via

 yum clean all

and install satellite packages via

yum install satellite

Install satellite with

satellite-installer --scenario satellite \
--foreman-initial-organization "initial_organization_name" \
--foreman-initial-location "initial_location_name" \
--foreman-initial-admin-username admin_user_name \
--foreman-initial-admin-password admin_password

Backup / Restore / Cloning

Use satellite-maintain for doing offline and online backups

satellite-maintain backup offline /backup/

when using the online option make sure that no new content view or content view versions should be created while the backup is running. basically satellite should be idle.

Cloning Satellite

The online/offline options also backup /var/lib/pulp, which contains all downloaded packages. This could be huge. There’s an option to skip this so

satellite-maintain backup offline --skip-pulp-tar /backup/

For a restore you always need the content of /var/lib/pulp.

This is mainly usefull for cloning satellite. You backup everything except /var/lib/pulp, copy the backup to a second system and rsync /var/lib/pulp to the new system. Then restore the backup and satellite should work as normal on the clone.

Snaphot backups

Satellite also supports backups via LVM snapshots. For more information see Snapshot backup

Upgrades

Read the Satellite release notes
Do a offline backup see [Backup / Restore]
You could clone satellite to a other system
If there are local changes to dhcpd or dns configurations use
```
satellite-installer --foreman-proxy-dns-managed=false --foreman-proxy-dhcp-managed=false
```
to stop satellite-install from overwriting those files.
install the latest version of satellite-maintain via
```
yum install rubygem-foreman_maintain
```
check for available satellite versions with
```
satellite-maintain upgrade list-versions
```

test the possible upgrade with

satellite-maintain upgrade check --target-version 6.7

and finally run the upgrade and PRAY!

satellite-maintain upgrade run --target-version 6.7

Various tips and tricks

Installing packages via yum

Satellite installs a yum plugin called foreman-protector. If you try to install a package via yum you get the following message

WARNING: Excluding 12190 packages due to foreman-protector.
Use foreman-maintain packages install/update <package>
to safely install packages without restrictions.
Use foreman-maintain upgrade run for full upgrade.

so use

satellite-maintain install <package name>

OS package upgrade

This should be done via satellite-maintain because all packages are locked by default (see Installing packages via yum).

This basically comes down to running

oreman-maintain upgrade run --target-version 6.6.z

for upgrading OS packages if you have satellite 6.6 installed.

Helpful oc / kubectl commands

Wed, 01 Apr 2020 00:00:00 +0000

This is a list of useful oc and/or kubectl commands so they won’t be forgotton. No this is not a joke…

List all pods in state Running

oc get pods --field-selector=status.phase=Running

List all pods in state Running and show there resource usage

oc get pods --field-selector=status.phase=Running -o json|jq ".items[] | {name: .metada
ta.name, res: .spec.containers[].resources}"

List events sort by time created

 oc get events --sort-by='.lastTimestamp'

Explain objects while specifing the api version

Sometimes when you run oc explain you get a message in DESCRIPTION that this particular version is deprecated, e.g. you are running oc explain deployment and get

DESCRIPTION:
DEPRECATED - This group version of Deployment is deprecated by
apps/v1/Deployment. See the release notes for more information. Deployment
enables declarative updates for Pods and ReplicaSets.

Note the DEPRECTATED message above

if you want to see the documentation for the object that has not been deprecated you can use

oc explain deployment --api-version=apps/v1

Magic with oc set

oc set is actually a very versatile command. Studying oc set -h is a good idea, here are some examples

Set route weights when alternateBackends in a route are defined

oc set route-backends bluegreen blue=1 green=9

Set resources on the command line

oc set resources dc cakephp-mysql-example --limits=memory=1Gi,cpu=200m