Networking on TechBlog about OpenShift/Ansible/Satellite and much more

OpenShift Virtualization Networking - The Overview

Mon, 29 Dec 2025 00:00:00 +0000

It’s time to dig into OpenShift Virtualization. You read that right, OpenShift Virtualization, based on kubevirt allows you to run Virtual Machines on top of OpenShift, next to Pods. If you come from a pure Kubernetes background, OpenShift Virtualization can feel like stumbling into a different dimension. In the world of Pods, we rarely care about Layer 2, MAC addresses, or VLANs. The SDN (Software Defined Network) handles the magic and we are happy.

But Virtual Machines are different….

They are needy creatures. They often refuse to live in the bubble of the default Pod network. They want to talk to the external Oracle database on the bare metal server next door, they need a static IP from the corporate range, or they need to be reachable via a specific VLAN. Networking is a key part of the Virtualization story and we need to master it.

To achieve this, we need to understand two tools: NMState Operator and Multus.

In this guide, I will try to discuss some of the basics for "Multihomed Nodes" and look at the three main ways to connect your VMs to the outside world.

Who does what: NMState vs. Multus

Before we look at YAML, we need to clear up a common confusion. Who does what?

Think of your OpenShift Node as a house.

NMState (The Electrician): This operator works on the Node (Host) level. It drills holes in the walls, lays the physical cables, and installs the wall sockets. In technical terms: It configures Bridges, Bonds, and VLANs on the Linux OS of the worker nodes.
Multus (The Extension Cord): This CNI plugin works on the VM/Pod level. It plugs the VM into the socket that NMState created. It allows a VM to have more than one network interface (eth0, eth1, etc.).

You usually need both. First, NMState builds the bridge. Then, Multus connects the VM to it.

NMState Operator Installation

While Multus is installed by default, NMState is not. You need to install the Operator first.

To install the NMState Operator, you can use the Web UI or the CLI. For the Web UI, you can use the following steps:

In the OpenShift Web Console, navigate to Operators → OperatorHub (or Ecosystem → Software Catalog in version 4.20+).
In the OperatorHub, search for NMState and select the Operator from the list.
Click on the NMState Operator and click on the Install button.
Select the Install button to install the Operator.
After the Operator is installed, you can check the status of the Operator in the Installed Operators view.

Once done you will need to create an instance of the NMState Operator. Simply create the following resource:

apiVersion: nmstate.io/v1
kind: NMState
metadata:
name: nmstate
spec:
probeConfiguration:
dns:
host: root-servers.net

NMState Operator Resource Types

The NMState Operator provides the three custom resource types to manage the node networking:

NodeNetworkState (NNS) → The "As-Is" State:
- This is an automatically generated report of the current status. It effectively tells you: "On Server 1, I currently see Network Card A and B, and IP address X is configured."
- Purpose: To verify the current reality on the ground before you make changes.
NodeNetworkConfigurationPolicy (NNCP) → The "To-Be" State (The Blueprint):
- This is the most critical resource. This resource is used to configure the network on the node. Here you tell the cluster that "I want a bridge named br1 to exist on all servers, and it must be connected to port ens4."
- The operator will try to configure this configuration across your nodes.
NodeNetworkConfigurationEnactment (NNCE) → The Result report:
- This resource is created autoamtically after the NNCP (the blueprint) has been created.
- It resports back: This resource is used to enact the network configuration on the node.
- It reports back on the execution: "Success! The bridge has been built," or "Failure! Cable not found."

With NMState in place, we can now start to configure the network on the node.

Part 1: Building the Bridge (NMState)

By default, Virtual Machines (VMs) are connected to the default pod network of the cluster as any other Pod. Using this network, the VM can communicate with other resources inside the cluster, or any resources that are reachable through the node network. Sometimes the VM needs a connection to a different network. In such a case you must connect the VM to an additional network. The VM will be configured with an additional network interface and is considered as multihomed VM.

To connect a VM to the physical network, we first need a bridge on the worker nodes. A bridge forwards packets between connected interfaces, similar to a network switch. We have two choices: Linux Bridge or OVS (Open vSwitch) Bridge.

Linux Bridge: The sturdy, wooden bridge. Simple, robust, standard Linux kernel tech. Use this for 90% of your use cases (Static IPs, simple VLAN access).
OVS Bridge: The magical, floating bridge. Complex, programmable, supports SDN logic. Use this only if you need advanced tunneling or integration with OVN policies on the physical interface.

Let’s create a standard Linux Bridge called br1 on all our worker nodes, attached to physical interface ens4.

We do this by applying a NodeNetworkConfigurationPolicy (NNCP).

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: br1-policy
spec:
nodeSelector:
node-role.kubernetes.io/worker: "" (1)
desiredState:
interfaces:
- name: br1 (2)
description: Linux bridge linked to ens4 (3)
type: linux-bridge (4)
state: up (5)
ipv4:
dhcp: true (6)
enabled: true
bridge: (7)
options:
stp:
enabled: false
port: (8)
- name: ens4

1	Apply to all worker nodes. If omitted, the policy will be applied to all nodes.
2	Name of the bridge
3	Description of the bridge
4	Type of the bridge (linux-bridge)
5	State of the bridge (up)
6	DHCP is enabled for the bridge (true)
7	Options for the bridge
8	Port of the bridge (ens4)

Setting an interface to absent or deleting an NNCP resource does not restore the previous configuration. A cluster administrator must define a policy with the previous configuration to restore settings.

Part 2: Defining the NetworkAttachmentDefinition (NAD)

Now that the bridge br1 exists on the nodes, we need to tell OpenShift how to use it. We do this with a NetworkAttachmentDefinition (NAD).

There are three distinct "Types" of networks you can define in a NAD.

1. The Linux Bridge (CNI-Plugin)

Linux bridges and OVS bridges both connect VMs to additional networks. A Linux bridge offers a simple, stable and sturdy solution and is well established. The OVS bridge on the other hand provides an advanced feature set for software-defined networks and is more challenging to troubleshoot.

Use Case: Direct Layer 2 connection to the physical network.

Behavior: The VM is practically "on the wire" of your datacenter. It can do DHCP with your corporate router.

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: bridge-network
namespace: my-vm-namespace
spec:
config: '{
"cniVersion": "0.3.1",
"name": "bridge-network",
"type": "bridge", (1)
"bridge": "br1", (2)
"ipam": {
"type": "dhcp"
}
}'

1	Type of the network (bridge)
2	Name of the bridge (br1)

type: "bridge" refers to the CNI plugin that utilizes the Linux Bridge we created earlier.

A note on IPAM (IP Address Management)

You might have noticed the "ipam" section in the NAD examples. You have three main choices there:

dhcp: Passes DHCP requests to the physical network (requires an external DHCP server).
static: You manually define IPs in the VM config (hard work).
whereabouts: A "Cluster-wide DHCP" for private networks. Perfect for the OVN L2 Overlay scenario where no external DHCP exists.

2. OVN Kubernetes L2 Overlay

A switched (layer 2) topology network interconnects workloads through a cluster-wide logical switch. This configuration allows East-West traffic only (packets between Pods within a cluster).

Use Case: You need a private network between your VMs (East-West traffic only), but you don’t want them to talk to the physical network.

Behavior: It creates a Geneve tunnel (like VXLAN) over the network.

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: private-overlay
namespace: my-vm-namespace
spec:
config: '{
"cniVersion": "0.4.0",
"name": "private-overlay",
"type": "ovn-k8s-cni-overlay",
"topology": "layer2", (1)
"netAttachDefName": "my-vm-namespace/private-overlay" (2)
}'

1	Topology of the network (layer2)
2	NetworkAttachmentDefinition name (my-vm-namespace/private-overlay)

A pod with the annotation k8s.v1.cni.cncf.io/networks: private-overlay will be connected to the private-overlay network.

3. OVN Kubernetes Secondary Localnet

OVN-Kubernetes also supports a localnet topology for secondary networks. This type creates a connection between a secondary network and a physical network, allowing VMs to communicate with destinations that are outside of the cluster. You must map the secondary network to the OVS bridge on the node. This is done by using a NodeNetworkConfigurationPolicy resource.

Use Case: You want to connect to the physical network (like the Linux Bridge), but you want to leverage OVN features (like NetworkPolicies) on that traffic.

Behavior: Uses Open vSwitch mapping to connect the OVN logic to the physical port.

Let’s create the bridge mapping by applying the following NodeNetworkConfigurationPolicy resource:

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: br0-ovs
spec:
nodeSelector:
node-role.kubernetes.io/worker: "" (1)
desiredState:
interfaces:
- name: br0-ovs (2)
description: OVS bridge with ens4 as a port
type: ovs-bridge (3)
state: up
ipv4:
dhcp: true
enabled: true
bridge:
options:
stp: true
port:
- name: ens4 (4)
ovn:
bridge-mappings:
- localnet: br0-network (5)
bridge: br0-ovs (6)
state: present

1	Apply to all worker nodes. If omitted, the policy will be applied to all nodes.
2	Name of the bridge
3	Type of the bridge (ovs-bridge)
4	Port of the bridge (ens4)
5	Localnet network name (br0-network)
6	OVS bridge name (br0-ovs)

Now let’s create the NetworkAttachmentDefinition for the localnet network so that VMs can connect to it. This NAD is created in the namespace of the VM.

If the NAD is created in the default namespace, the configuration will be available to all namespaces.

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: ovn-localnet
namespace: my-vm-namespace
spec:
config: '{
"cniVersion": "0.4.0",
"name": "ovn-localnet", (1)
"type": "ovn-k8s-cni-overlay", (2)
"topology": "localnet", (3)
"netAttachDefName": "my-vm-namespace/br0-network" (4)
}'

1	Name of the network (ovn-localnet)
2	Type of the network (ovn-k8s-cni-overlay)
3	Topology of the network (localnet)
4	Reference to the localnet mapping name defined in the NNCP (my-vm-namespace/br0-network)

Part 3: Connecting the VM

Finally, we have our "socket" (the NAD). Let’s plug the VM in. In your VirtualMachine manifest, you add the network to the spec.

This can be done via the Web Console or via the CLI.

Via the Web Console:

In the OpenShift Web Console, navigate to Virtualization → VirtualMachines.
Click on the VM you want to connect to the network.
Select the "Configuration" tab
Select "Network"

Click "Add network interface"
In the Popup enter a name for the interface
Model keep as "virtio"
On the "Network" select the NAD you created in the previous step.

Click "Save"

The VM will now have a Pending state. This is because the VM is waiting for a migration to activate the new network interface.

In the action menu you can select "Migrate" > "Compute" to start the migration process. After a while the new NIC will be active.

Via the CLI:

Here is a VM connected to the Linux Bridge NAD we created in step 1:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: helloworld-vm
namespace: my-vm-namespace
spec:
template:
spec:
domain:
devices:
interfaces:
- name: default
masquerade: {} (1)
- name: br0-network
bridge: {} (2)
networks:
- name: default
pod: {}
- name: br0-network (3)
multus:
networkName: br0-network (4)

1	The standard Pod Network
2	Our new connection
3	The NAD name
4	The NAD name

Advanced br-ex Configuration with Bonding on OpenShift

Thu, 05 Feb 2026 00:00:00 +0000

In standard OpenShift Container Platform (OCP) installations, the cluster networking operator relies on default scripts to configure the external bridge, br-ex. While these defaults work for simple setups, production environments frequently demand robust and redundant setups, such as LACP bonding or active-backup configurations that the default scripts cannot guess.

In this article, we explore advanced configurations for the br-ex bridge on OpenShift. We will configure a worker node with a high-availability active-backup bond, attach it to an OVS bridge (br-ex), and ensure the node maintains connectivity.

Introduction

The br-ex (external bridge) is a critical component in OpenShift’s OVN-Kubernetes network architecture. It handles external traffic flowing in and out of the cluster. By default, br-ex is attached to a single network interface, but in production environments, you often need:

High Availability: If one network link fails, traffic continues through the remaining links
Increased Bandwidth: Aggregate bandwidth from multiple interfaces
Load Balancing: Distribute traffic across multiple physical connections

If you already have a cluster running, you can migrate your br-ex bridge to NMState. In this article, I have summarized the process step by step.

The official documentation can be found at Migrating a configured br-ex bridge to NMState.

Why Customization is Necessary

By default, OVN-Kubernetes expects a bridge named br-ex to handle traffic exiting the cluster. If you need this bridge to sit on top of a specific bond (like bond-public) rather than a single physical interface, or if you need to enforce specific MAC address policies, you must intervene before the node boots into the cluster.

This method allows you to:

Migrate br-ex bridge to NMState and make postinstallation changes.
Define complex topologies (Bonds, VLANs, Bridges) declaratively.
Persist configuration across reboots and upgrades via the Machine Config Operator.
Decouple networking from the OS installation process.

Prerequisites

Before configuring bonding with br-ex, ensure you have:

OpenShift 4.x cluster with OVN-Kubernetes CNI
OPTIONAL: NMState Operator installed (helps to verify and visualize the configuration on the node)
Multiple network interfaces (interface1 and interface2) available on your nodes
Understanding of your network topology and switch configuration

Understanding Bonding Modes

Linux supports several bonding modes. The most common ones for OpenShift deployments and supported by OpenShift Virtualization are:

Mode	Name	Description
1	active-backup	Only one interface active at a time. Failover when active fails. No switch configuration required.
2	balance-xor	XOR-based load balancing. Requires switch configuration.
4	802.3ad (LACP)	Dynamic link aggregation. Requires switch support for LACP.

Mode

Name

Description

active-backup

Only one interface active at a time. Failover when active fails. No switch configuration required.

balance-xor

XOR-based load balancing. Requires switch configuration.

802.3ad (LACP)

Dynamic link aggregation. Requires switch support for LACP.

Mode LACP is recommended for production environments when switch support is available, as it provides both redundancy and load balancing.

The Technical Setup

We will implement the following network topology:

Physical Interfaces: interface1 and interface2 (configured as secondary).
Bond Interface: bond-public in active-backup mode.
OVS Bridge: br-ex (the standard OVN external bridge) using bond-public as a port.
Host Networking: The node’s IP address will be assigned to the br-ex OVS interface (internal port), effectively moving the management IP from the physical NIC to the bridge.

Migration Process

The migration process is quite simple. We will use the MachineConfig Operator to apply the configuration to the node.

This means we will:

Create the NMState configuration file
Encode it as base64
Create a MachineConfig object for the required nodes
Apply the MachineConfig object to the required nodes aka reboot the node

Create the NMState configuration file

First, we need to create the NMState configuration file. This file will be used to configure the network on the node. We will define two interfaces interface1 and interface2 and a bond interface bond-public in active-backup mode. The br-ex bridge will be configured to use the bond-public interface. The MAC address from interface2 will be copied to the bond-public and br-ex interfaces. This ensures consistent MAC addressing for ARP/DHCP and prevents IP conflicts during failover.

This is the crucial step in the migration process.

interfaces:
########################################################
# Interface 1
########################################################
- name: interface1 (1)
type: ethernet
state: up
ipv4:
enabled: false
ipv6:
enabled: false
########################################################
# Interface 2
########################################################
- name: interface2 (2)
type: ethernet
state: up
ipv4:
enabled: false
ipv6:
enabled: false
########################################################
# Bond interface
########################################################
- description: Bond interface for public LAN interfaces
ipv4:
enabled: false
link-aggregation:
mode: active-backup (3)
port: (4)
- interface1
- interface2
name: bond-public (5)
state: up
type: bond
copy-mac-from: interface2 (6)
########################################################
# OVS Bridge
########################################################
- name: br-ex (7)
type: ovs-bridge (8)
state: up
ipv4:
enabled: false
dhcp: false
ipv6:
enabled: false
dhcp: false
bridge:
options:
mcast-snooping-enable: true
port: (9)
- name: bond-public
- name: br-ex
########################################################
# OVS Interface
########################################################
- name: br-ex
type: ovs-interface
state: up
copy-mac-from: interface2 (10)
ipv4:
enabled: true
dhcp: true
auto-route-metric: 48 (11)
ipv6:
enabled: false

1	Specification of interface1
2	Specification of interface2
3	Bonding mode: active-backup in this example
4	Port of the bond interface (interface1 and interface2)
5	Name of the bond interface
6	Copy the MAC address from the interface2 to the bond interface
7	Specification of the bridge (ovs-bridge)
8	Type of the bridge (ovs-bridge)
9	Port of the bridge (bond-public and br-ex)
10	Copy the MAC address from the interface2 to the interface
11	Auto-route metric - Set the parameter to 48 to ensure the br-ex default route always has the highest precedence (lowest metric). This configuration prevents routing conflicts with any other interfaces that are automatically configured by the NetworkManager service.

With this configuration we can now create a MachineConfig object and roll out the changes.

Encode the Configuration as base64

We need to encode the configuration as base64. This is required by the MachineConfig Operator.

Assuming you have the command line tool base64 installed and stored the configuration in a file called br-ex-public.yaml, you can encode the configuration as follows.

cat br-ex-public.yaml | base64 -w 0

The result will be a long string of characters. This is the base64 encoded configuration, that we will use in the next step.

Create the MachineConfig object

To apply this during installation (or via the Machine Config Operator after the deployment), we wrap the NMState YAML above into a MachineConfig object.

We need the base64 encoded string and will place it in a specific path: /etc/nmstate/openshift/<filename>.yml

From the official documentation: "For each node in your cluster, specify the hostname path to your node and the base-64 encoded Ignition configuration file data for the machine type. The worker role is the default role for nodes in your cluster. You must use the .yml (not yaml !) extension for configuration files, such as $(hostname -s).yml when specifying the short hostname path for each node or all nodes in the MachineConfig manifest file."

In short: For each node, create a file named <hostname>.yml containing the base64-encoded NMState configuration. Note that you must use the .yml extension (not .yaml).

If you have one single configuration for ALL nodes, you can create one configuration file called cluster.yml. However, I personally do not recommend this approach, as over time new nodes might have different hardware and therefore different network configurations.

In the configuration below, we are targeting a specific worker named wrk01.

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
labels:
machineconfiguration.openshift.io/role: worker (1)
name: 10-br-ex-wrk01 (2)
spec:
config:
ignition:
version: 3.2.0
storage:
files:
- contents:
# This source is the Base64 encoded version of the NMState YAML above
source: data:text/plain;charset=utf-8;base64,<BASE64_ENCODED_STRING> (3)
mode: 0644
overwrite: true
# The path determines which node picks up the config.
# Use 'cluster.yml' to apply to all nodes in this MachineConfig pool.
path: /etc/nmstate/openshift/wrk01.yml (4)

1	Valid for node with the role worker
2	Name of the MachineConfig object
3	Base64 encoded configuration
4	Path to the configuration file. This configuration is valid for the node with the hostname wrk01.

Apply the MachineConfig Object

The above MachineConfig object must now be applied to the cluster. There are two scenarios to do this:

Scenario A: Post-Installation / Migration of existing nodes

Save the YAML to 10-br-ex-wrk01.yaml.
Run oc apply -f 10-br-ex-wrk01.yaml
Reboot the applicable nodes to apply the configuration. You can either do this manually or use the MachineConfigPool to reboot the nodes. The following example forces a reboot on all nodes with the role worker.

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
labels:
machineconfiguration.openshift.io/role: worker
name: 10-force-reboot-worker
spec:
config:
ignition:
version: 3.2.0
storage:
files:
- contents:
source: data:text/plain;charset=utf-8;base64,
mode: 0644
overwrite: true
path: /etc/force-reboot

Scenario B: During Installation

Here the configuration will be applied during the installation process.

Run openshift-install create manifests to create the manifest files.
Copy the YAML file above into the openshift/ directory.
Run openshift-install create ignition-configs.
Deploy your nodes. The configuration will be applied on the very first boot.

Verifying the Configuration

Once the node is online, it is crucial to verify that the interfaces came up correctly and that traffic is flowing through the bond.

# Check bond status
oc debug node/<node-name> -- chroot /host cat /proc/net/bonding/bond-public
# Check the state of the specific interfaces
nmstatectl show br-ex bond-public
# Look for state: up on both interfaces and ensure bond-public lists the correct ports (interface1 and interface2).
# Verify br-ex bridge
oc debug node/<node-name> -- chroot /host ovs-vsctl show
#Expected Output:
#
#Plaintext
# Bridge br-ex
# Port br-ex
# Interface br-ex
# type: internal
# Port bond-public
# Interface bond-public
# type: bond

Test Failover

To test the active-backup bonding:

Start a continuous ping to an external IP, for example: ping 8.8.8.8
Physically disconnect (or administratively down) the active interface (e.g., interface1).
Observe the ping. You should see minimal packet loss (usually just 1 or 2 packets) as traffic fails over to interface2.

Troubleshooting

Common Issues

Bond not forming:

Verify both interfaces are physically connected
Check switch port configuration matches bonding mode
Review NMState logs: oc logs -n openshift-nmstate deployment/nmstate-handler

Traffic not flowing:

Verify br-ex is properly attached to the bond
Check OVN-Kubernetes pods are running
Review node network configuration: oc debug node/<node-name> — chroot /host nmcli connection show

Rollback Procedure

If the configuration fails and the node becomes unreachable:

Boot the node into single-user mode or use console access
Remove the NMState configuration file: rm /etc/nmstate/openshift/<hostname>.yml
Restart NetworkManager: systemctl restart NetworkManager
The node should revert to the default network configuration

Alternatively, delete the MachineConfig from the cluster (from a working node):

oc delete machineconfig 10-br-ex-wrk01

This will trigger a rollback on the affected nodes during the next reboot.

Conclusion

By using MachineConfig to inject an NMState policy at /etc/nmstate/openshift/, we successfully bypassed the default network scripts. We created a robust OVS bridge (br-ex) backed by an active-backup bond (bond-public), ensuring that our OpenShift worker node (wrk01) is both redundant and compliant with OVN-Kubernetes networking requirements.

Configuring br-ex with bonding provides essential redundancy and can improve network throughput for your OpenShift cluster. Choose the appropriate bonding mode based on your switch capabilities and requirements:

Use active-backup for simple failover without switch configuration
Use 802.3ad (LACP) for optimal redundancy and load balancing when switches support it
Use balance-alb when you need load balancing but lack switch support

Always test your configuration in a non-production environment before applying to production clusters.