Stumbling into Azure Part II: Setting up a private ARO cluster

October 29, 2021 - By: Toni Schmidbauer ( Lastmod: 2021-11-16 )

In Part I of our blog post we covered setting up required resources in Azure. Now we are finally going to set up a private cluster. Private

As review from Part I here is our planned setup, this time including the ARO cluster.

Azure Setup

The diagram below depicts our planned setup:

/Azure/images/azure_network_setup_with_aro.png

On the right hand side can see the resources required for our lab:

a virtual network (vnet 192.168.128.0/19). This vnet will be split into 3 separate subnets
a master subnet (192.168.129.0/24) holding the ARO control plane nodes
a node subnet (192.168.130.0/24) holding ARO worker nodes
and finally a subnet call GatewaySubnet where we are going to deploy our Azure VPN gateway (called a vnet-gateway)
The subnet where the Azure VPN gateway is located needs to have the name GatewaySubnet. Otherwise creating the Azure VPN gateway will fail.
we also need a publicIP resource that we are going to connect to our vnet-gateway (the VPN gateway)
and finally a local-gateway resource that tells the vnet-gateway which networks are reachable on the left, in our case the Hetzner server.

Creating the private Azure Red Hat OpenShift cluster

az provider register -n Microsoft.RedHatOpenShift --wait
az provider register -n Microsoft.Compute --wait
az provider register -n Microsoft.Storage --wait
az provider register -n Microsoft.Authorization --wait

First we are going to set some environment variable. Those variables are used in the upcoming commands:

export RESOURCEGROUP=aro-rg
export CLUSTER="aro1"
export GATWAY_SUBNET="192.168.128.0/24"
export MASTER_SUBNET="192.168.129.0/24"
export WORKER_SUBNET="192.168.130.0/24"
export HETZNER_VM_NETWORKS="10.0.0.0/24 192.168.122.0/24 172.16.100.0/24"

Disable subnet private endpoint policies

az network vnet subnet update \
--name master-subnet \
--resource-group $RESOURCEGROUP \
--vnet-name aro-vnet \
--disable-private-link-service-network-policies true

Create a private DNS zone for our cluster

az network private-dns zone create -n private2.tntinfra.net -g aro-rg

Create the cluster

az aro create \
--resource-group $RESOURCEGROUP \
--name $CLUSTER \
--vnet aro-vnet \
--master-subnet master-subnet \
--worker-subnet worker-subnet \
--apiserver-visibility Private \
--ingress-visibility Private \
--domain private.tntinfra.net
# --pull-secret @pull-secret.txt # [OPTIONAL]

After successful cluster creating add DNS entry for the API and Ingress

Query the Azure API for the API server IP and the ingress IP addresses:

az aro show -n aro1 -g aro-rg --query '{api:apiserverProfile.ip, ingress:ingressProfiles[0].ip}'

Example output

Api            Ingress
-------------  ---------------
192.168.129.4  192.168.130.254

Add entries to Azure private DNS

az network private-dns record-set a add-record -g aro-rg -z private.tntinfra.net -a "192.168.129.4" -n api
az network private-dns record-set a add-record -g aro-rg -z private.tntinfra.net -a "192.168.130.254" -n "*.apps"

List entries to verify configuration

az network private-dns record-set a list -g aro-rg -z private.tntinfra.net

Output:

Name    ResourceGroup    Ttl    Type    AutoRegistered    Metadata
------  ---------------  -----  ------  ----------------  ----------
api     aro-rg           3600   A       False
*.apps  aro-rg           3600   A       False

List cluster credentials after successful setup

az aro list-credentials \
--name $CLUSTER \
--resource-group $RESOURCEGROUP

Get the console URL

az aro show \
--name $CLUSTER \
--resource-group $RESOURCEGROUP \
--query "consoleProfile.url" -o tsv

DNS, curl

this works, dunno why?

dig @192.168.129.7 console-openshift-console.apps.xm7rdz4r.westeurope.aroapp.io

use curl to access the internal API and see if it works:

curl -kv https://192.168.129.4:6443

Stumbling into Azure Part II: Setting up a private ARO cluster

Azure Setup

Creating the private Azure Red Hat OpenShift cluster

DNS, curl

Additional Resources