Stumbling into Azure Part II: Setting up a private ARO cluster

- By: Toni Schmidbauer ( Lastmod: 2021-11-16 )

In Part I of our blog post we covered setting up required resources in Azure. Now we are finally going to set up a private cluster. Private

As review from Part I here is our planned setup, this time including the ARO cluster.

Azure Setup

The diagram below depicts our planned setup:

/Azure/images/azure_network_setup_with_aro.png

On the right hand side can see the resources required for our lab:

  • a virtual network (vnet 192.168.128.0/19). This vnet will be split into 3 separate subnets

  • a master subnet (192.168.129.0/24) holding the ARO control plane nodes

  • a node subnet (192.168.130.0/24) holding ARO worker nodes

  • and finally a subnet call GatewaySubnet where we are going to deploy our Azure VPN gateway (called a vnet-gateway)

    The subnet where the Azure VPN gateway is located needs to have the name GatewaySubnet. Otherwise creating the Azure VPN gateway will fail.

  • we also need a publicIP resource that we are going to connect to our vnet-gateway (the VPN gateway)

  • and finally a local-gateway resource that tells the vnet-gateway which networks are reachable on the left, in our case the Hetzner server.

Creating the private Azure Red Hat OpenShift cluster

  1. Register required resource providers

    az provider register -n Microsoft.RedHatOpenShift --wait
    az provider register -n Microsoft.Compute --wait
    az provider register -n Microsoft.Storage --wait
    az provider register -n Microsoft.Authorization --wait
  1. First we are going to set some environment variable. Those variables are used in the upcoming commands:

    export RESOURCEGROUP=aro-rg
    export CLUSTER="aro1"
    export GATWAY_SUBNET="192.168.128.0/24"
    export MASTER_SUBNET="192.168.129.0/24"
    export WORKER_SUBNET="192.168.130.0/24"
    export HETZNER_VM_NETWORKS="10.0.0.0/24 192.168.122.0/24 172.16.100.0/24"
  2. Disable subnet private endpoint policies

    az network vnet subnet update \
    --name master-subnet \
    --resource-group $RESOURCEGROUP \
    --vnet-name aro-vnet \
    --disable-private-link-service-network-policies true
  3. Create a private DNS zone for our cluster

    az network private-dns zone create -n private2.tntinfra.net -g aro-rg
  4. Create the cluster

    az aro create \
    --resource-group $RESOURCEGROUP \
    --name $CLUSTER \
    --vnet aro-vnet \
    --master-subnet master-subnet \
    --worker-subnet worker-subnet \
    --apiserver-visibility Private \
    --ingress-visibility Private \
    --domain private.tntinfra.net
    # --pull-secret @pull-secret.txt # [OPTIONAL]
  5. After successful cluster creating add DNS entry for the API and Ingress

    Query the Azure API for the API server IP and the ingress IP addresses:

    az aro show -n aro1 -g aro-rg --query '{api:apiserverProfile.ip, ingress:ingressProfiles[0].ip}'

    Example output

    Api            Ingress
    -------------  ---------------
    192.168.129.4  192.168.130.254

    Add entries to Azure private DNS

    az network private-dns record-set a add-record -g aro-rg -z private.tntinfra.net -a "192.168.129.4" -n api
    az network private-dns record-set a add-record -g aro-rg -z private.tntinfra.net -a "192.168.130.254" -n "*.apps"

    List entries to verify configuration

    az network private-dns record-set a list -g aro-rg -z private.tntinfra.net

    Output:

    Name    ResourceGroup    Ttl    Type    AutoRegistered    Metadata
    ------  ---------------  -----  ------  ----------------  ----------
    api     aro-rg           3600   A       False
    *.apps  aro-rg           3600   A       False
  6. List cluster credentials after successful setup

    az aro list-credentials \
    --name $CLUSTER \
    --resource-group $RESOURCEGROUP
  7. Get the console URL

    az aro show \
    --name $CLUSTER \
    --resource-group $RESOURCEGROUP \
    --query "consoleProfile.url" -o tsv

DNS, curl

this works, dunno why?

dig @192.168.129.7 console-openshift-console.apps.xm7rdz4r.westeurope.aroapp.io

use curl to access the internal API and see if it works:

curl -kv https://192.168.129.4:6443