Stumbling into Azure Part I: Building a site-to-site VPN tunnel for testing

OpenShift

- Toni Schmidbauer Toni Schmidbauer ( Lastmod: 2024-05-08 ) - 3 min read

So we want to play with ARO (Azure Red Hat OpenShift) private clusters. A private cluster is not reachable from the internet (surprise) and is only reachable via a VPN tunnel from other networks.

This blog post describes how we created a site-to-site VPN between a Hetzner dedicated server running multiple VM's via libvirt and Azure.

An upcoming blog post is going to cover the setup of the private ARO cluster.

Azure Setup

The diagram below depicts our planned setup:

/azure/images/azure_network_setup.png

On the right hand side can see the resources required for our lab:

  • a virtual network (vnet 192.168.128.0/19). This vnet will be split into 3 separate subnets
  • a master subnet (192.168.129.0/24) holding the ARO control plane nodes
  • a node subnet (192.168.130.0/24) holding ARO worker nodes
  • and finally a subnet call GatewaySubnet where we are going to deploy our Azure VPN gateway (called a vnet-gateway)

    The subnet where the Azure VPN gateway is located needs to have the name GatewaySubnet. Otherwise creating the Azure VPN gateway will fail.

  • we also need a publicIP resource that we are going to connect to our vnet-gateway (the VPN gateway)
  • and finally a local-gateway resource that tells the vnet-gateway which networks are reachable on the left, in our case the Hetzner server.

Creating the required Azure resources

  1. First we are going to set some environment variable. Those variables are used in the upcoming commands:

    export RESOURCEGROUP=aro-rg
export GATWAY_SUBNET="192.168.128.0/24"
export MASTER_SUBNET="192.168.129.0/24"
export WORKER_SUBNET="192.168.130.0/24"
export HETZNER_VM_NETWORKS="10.0.0.0/24 192.168.122.0/24 172.16.100.0/24"

  2. Next create a VNET resource holding our sub networks:

    az network vnet create \
--resource-group $RESOURCEGROUP \
--name aro-vnet \
--address-prefixes 192.168.128.0/18

  3. Create the GatewaySubnet subnet

    az network vnet subnet create \
--resource-group $RESOURCEGROUP \
--vnet-name aro-vnet \
--name GatewaySubnet \
--address-prefixes $GATEWAY_SUBNET

  4. Create the master subnet

    az network vnet subnet create \
--resource-group $RESOURCEGROUP \
--vnet-name aro-vnet \
--name master-subnet \
--address-prefixes $MASTER_SUBNET \
--service-endpoints Microsoft.ContainerRegistry

  5. Create the worker subnet

    az network vnet subnet create \
--resource-group $RESOURCEGROUP \
--vnet-name aro-vnet \
--name worker-subnet \
--address-prefixes $WORKER_SUBNET \
--service-endpoints Microsoft.ContainerRegistry

  6. Create a public IP resource

    az network public-ip create \
--name GatewayIP \
--resource-group $RESOURCEGROUP \
--allocation-method Dynamic

  7. Create a local-gateway resource

    az network local-gateway create \
--name playground \
--resource-group $RESOURCEGROUP \
--local-address-prefixes $HETZNER_VM_NETWORKS \
--gateway-ip-address 95.217.42.98

  8. Create a vnet-gateway resource (takes around 30 minutes)

    az network vnet-gateway create \
--name vpn-gateway \
--public-ip-address GatewayIP \
--resource-group $RESOURCEGROUP \
--vnet aro-vnet \
--gateway-type Vpn \
--vpn-type RouteBased \
--sku Basic \
--no-wait

  9. Define a vpn-connection

    az network vpn-connection create \
--name VNet1toSite2 \
--resource-group $RESOURCEGROUP \
--vnet-gateway1 vpn-gateway \
--local-gateway2 playground \
--location westeurope \
--shared-key thepassword

Required iptables (nf tables) hacks for libvirt

Skip NAT rules if the destination network is in Azure and the client network deploy via libvirt

iptables -I LIBVIRT_PRT 2 -t nat -d 192.168.129.0/24 -j RETURN
iptables -I LIBVIRT_PRT 2 -t nat -d 192.168.130.0/24 -j RETURN

Skip NAT rules if the destination network is in Azure and the client is connected via tailscale

iptables -I ts-postrouting 1 -t nat -d 192.168.129.0/24 -j RETURN
iptables -I ts-postrouting 1 -t nat -d 192.168.130.0/24 -j RETURN

Libreswan setup on CentOS Stream

  1. Install the Libreswan packages

    dnf install libreswan

  2. Create a Azure configuration for Libreswan in ~/etc/ipsec.d/azure.conf

    conn masterSubnet
also=azureTunnel
leftsubnet=192.168.129.0/24
rightsubnet=172.16.100.0/24
auto=start

conn workerSubnet
also=azureTunnel
leftsubnet=192.168.130.0/24
rightsubnet=172.16.100.0/24
auto=start

conn azureTunnel
authby=secret
auto=start
dpdaction=restart
dpddelay=30
dpdtimeout=120
ike=aes256-sha1;modp1024
ikelifetime=3600s
ikev2=insist
keyingtries=3
pfs=yes
phase2alg=aes128-sha1
left=51.137.113.44
leftsubnets=192.168.128.0/24
right=%defaultroute
rightsubnets=172.16.100.0/24
salifetime=3600s
type=tunnel
ipsec-interface=yes

  3. Create a Libreswan secrets file for Azure in /etc/ipsec.d/azure.secrets:

    %any %any : PSK "abc123"

  4. Enable and start the IPsec service

    systemctl enable --now ipsec

  5. We had to explicitly load the IPsec configuration via

    ipsec addconn --config /etc/ipsec.d/azure.conf azureTunnel

Libreswan IPSEC debugging tips

  • Check the state of the IPsec systemd service

    systemctl status ipsec

  • Check the full log of the IPsec systemd service

    journalctl -e -u ipsec

  • Check the state of the tunnels with the ipsec command line tool

    ipsec status

    Check for the following lines

    000 Total IPsec connections: loaded 5, active 2
000
000 State Information: DDoS cookies not required, Accepting new IKE connections
000 IKE SAs: total(1), half-open(0), open(0), authenticated(1), anonymous(0)
000 IPsec SAs: total(2), authenticated(2), anonymous(0)
000
000 #130: "azureTunnel/1x1":500 STATE_V2_ESTABLISHED_CHILD_SA (IPsec SA established); EVENT_SA_REKEY in 2003s; newest IPSEC; eroute owner; isakmp#131; idle;
000 #130: "azureTunnel/1x1" esp.56cf4304@51.137.113.44 esp.6f49e8d3@95.217.42.98 tun.0@51.137.113.44 tun.0@95.217.42.98 Traffic: ESPin=0B ESPout=0B! ESPmax=0B
000 #129: "masterSubnet/0x0":500 STATE_V2_ESTABLISHED_CHILD_SA (IPsec SA established); EVENT_SA_REKEY in 1544s; newest IPSEC; eroute owner; isakmp#131; idle;
000 #129: "masterSubnet/0x0" esp.6e81e8da@51.137.113.44 esp.6f72bbc8@95.217.42.98 tun.0@51.137.113.44 tun.0@95.217.42.98 Traffic: ESPin=0B ESPout=0B! ESPmax=0B
000 #131: "masterSubnet/0x0":500 STATE_V2_ESTABLISHED_IKE_SA (established IKE SA); EVENT_SA_REKEY in 2121s; newest ISAKMP; idle;

    IPsec specifies properties of connections via security associations (SA). The parent SA is describes the IKEv2 connections, the child SA is the ESP (encapsulated security payload) connection.

    Check IPsec transformation policies

    ip xfrm policy

    Check the state of IPsec transformation policies

    ip xfrm state

    Check for dropped packages on the IPsec interface (ipsec1 in our case)

    ip -s link show dev ipsec1

See Also:

  • Introducing AdminNetworkPolicies –

    Classic Kubernetes/OpenShift offer a feature called NetworkPolicy that allows users to control the traffic to and from their assigned Namespace. NetworkPolicies are designed to give project owners or tenants the ability to protect their own namespace. Sometimes, however, I worked with customers where the cluster administrators or a dedicated (network) team need to enforce these policies. Since the NetworkPolicy API is namespace-scoped, it is not possible to enforce policies across namespaces. The only solution was to create custom (project) admin and edit roles, and remove the ability of creating, modifying or deleting NetworkPolicy objects. Technically, this is possible and easily done. But shifts the whole network security to cluster administrators. Luckily, this is where AdminNetworkPolicy (ANP) and BaselineAdminNetworkPolicy (BANP) comes into play.

  • Using Kustomize to post render a Helm Chart –

    Lately I came across several issues where a given Helm Chart must be modified after it has been rendered by Argo CD. Argo CD does a helm template to render a Chart. Sometimes, especially when you work with Subcharts or when a specific setting is not yet supported by the Chart, you need to modify it later …​ you need to post-render the Chart. In this very short article, I would like to demonstrate this on a real-live example I had to do. I would like to inject annotations to a Route objects, so that the certificate can be injected. This is done by the cert-utils operator. For the post-rendering the Argo CD repo pod will be extended with a sidecar container, that is watching for the repos and patches them if required.

  • Managing Certificates using GitOps approach –

    The article SSL Certificate Management for OpenShift on AWS explains how to use the Cert-Manager Operator to request and install a new SSL Certificate. This time, I would like to leverage the GitOps approach using the Helm Chart cert-manager I have prepared to deploy the Operator and order new Certificates. I will use an ACME Letsencrypt issuer with a DNS challenge. My domain is hosted at AWS Route 53. However, any other integration can be easily used.

  • Update Cluster Version using GitOps approach –

    During a GitOps journey at one point, the question arises, how to update a cluster? Nowadays it is very easy to update a cluster using CLI or WebUI, so why bother with GitOps in that case? The reason is simple: Using GitOps you can be sure that all clusters are updated to the correct, required version and the version of each cluster is also managed in Git. All you need is the channel you want to use and the desired cluster version. Optionally, you can define the exact image SHA. This might be required when you are operating in a restricted environment.

  • Multiple Sources for Applications in Argo CD –

    Argo CD or OpenShift GitOps uses Applications or ApplicationSets to define the relationship between a source (Git) and a cluster. Typically, this is a 1:1 link, which means one Application is using one source to compare the cluster status. This can be a limitation. For example, if you are working with Helm Charts and a Helm repository, you do not want to re-build (or re-release) the whole chart just because you made a small change in the values file that is packaged into the repository. You want to separate the configuration of the chart with the Helm package. The most common scenarios for multiple sources are (see: Argo CD documentation): Your organization wants to use an external/public Helm chart You want to override the Helm values with your own local values You don’t want to clone the Helm chart locally as well because that would lead to duplication and you would need to monitor it manually for upstream changes. This small article describes three different ways with a working example and tries to cover the advantages and disadvantages of each of them. They might be opinionated but some of them proved to be easier to use and manage.

Copyright © 2020 - 2024 Toni Schmidbauer & Thomas Jungbauer