Introducing AdminNetworkPolicies

OpenShift Security Network

- Thomas Jungbauer Thomas Jungbauer ( Lastmod: 2024-11-09 ) - 9 min read

Classic Kubernetes/OpenShift offer a feature called NetworkPolicy that allows users to control the traffic to and from their assigned Namespace. NetworkPolicies are designed to give project owners or tenants the ability to protect their own namespace. Sometimes, however, I worked with customers where the cluster administrators or a dedicated (network) team need to enforce these policies.

Since the NetworkPolicy API is namespace-scoped, it is not possible to enforce policies across namespaces. The only solution was to create custom (project) admin and edit roles, and remove the ability of creating, modifying or deleting NetworkPolicy objects. Technically, this is possible and easily done. But shifts the whole network security to cluster administrators.

Luckily, this is where AdminNetworkPolicy (ANP) and BaselineAdminNetworkPolicy (BANP) comes into play.

AdminNetworkPolicy (ANP) and BaselineAdminNetworkPolicy (BANP)

This article demonstrates the configuration of the new AdminNetworkPolicy and BaselineAdminNetworkPolicy objects using the Helm Chart admin-networkpolicies. The NetworkPolicy object is not covered in this article.

ANP and BANP are designed for cluster administrators to protect the entire cluster by creating cluster-scoped policies. They are not replacing NetworkPolicies, but instead create a tier model and can be used together. Administrators can use ANPs to enforce non-overridable policies that take precedence over NetworkPolicy objects. Administrators can use BANP to set up and enforce optional cluster-scoped network policy rules that are overridable by users using NetworkPolicy objects when necessary. When used together, ANP, BANP, and network policy can achieve full multi-tenant isolation that administrators can use to secure their cluster.

The three resources create a 3-Tier Access Control List (ACL) that is evaluated in descending order:

  • Tier 1 - AdminNetworkPolicy (ANP): If the traffic matches an allow or deny rule, then any existing NetworkPolicy and BaselineAdminNetworkPolicy (BANP) objects in the cluster are skipped from evaluation. If a pass rule is matched, then the evaluation is handed over to the next tier (NetworkPolicy). This means, that Cluster Administrators can enforce policies that cannot be overwritten by users (allow/deny rules) or pass the evaluation to the Network Policy, where the project owners can decide further.

  • Tier 2 - NetworkPolicy (NP): If the traffic passed the ANP then the NetworkPolicy is evaluating the traffic. The NetworkPolicy resources are controlled by the project owners by default.

  • Tier 3 - BaselineAdminNetworkPolicy (BANP): If the traffic passed the ANP and the NetworkPolicy, then the BANP is evaluating the traffic. These objects are controlled by the cluster administrators again and are cluster scoped. There can only be one BANP (named "default") configured on the cluster.

AdminNetworkPolicy

An AdminNetworkPolicy (ANP) is a cluster-scoped resource, that allow cluster administrators to secure the network traffic before NetworkPolicies in the namespaces are evaluated. These rules cannot be overwritten by project owners or developers and allow the administrators to enforce the security. Use cases could be, for example:

  • You want to enforce only specific egress endpoints (e.g. only allow traffic to the specific database servers)

  • You want to be sure that traffic from OpenShift monitoring is always allowed

  • You want to allow the management of NetworkPolicies to project owners and do not want to take care for them or during the onboarding.

The ANP allows cluster administrators to define:

  • A priority value that determines the order of its evaluation. The lower the value the higher the precedence.

  • A set of pods that consists of a set of namespaces or namespace on which the policy is applied.

  • A list of ingress rules to be applied for all ingress traffic towards the subject.

  • A list of egress rules to be applied for all egress traffic from the subject.

AdminNetworkPolicy Actions for Rules

The AdminNetworkPolicy allows three actions for the rules:

  • Allow: The traffic is allowed, and no further rules are evaluated.

  • Deny: The traffic is denied, and no further rules are evaluated.

  • Pass: The traffic is passed to the next tier (NetworkPolicy).

Subject of a Policy

In any ANP (or BANP) a subject can be defined and they specify the pods to which this AdminNetworkPolicy applies. (Note that host-networked pods are not included in subject.selection.) There are two ways to define the subject:

  1. namespaces: The namespaces block is used to select pods via namespace selectors. Here, matchLabels or matchExpressions can be used to limit the namespaces.

  2. pods: The Pods-Array is used to select pods via namespace AND pod selectors. Here namespaceSelector and podSelector can be set to limit the Pods.

If subject is not defined, the policy applies to all pods and namespaces in the cluster.

In my Helm chart the options are supported like the following snippets show:

Select Namespaces with matchExpressions

Values in the Helm Chart:

anp:
  - name: sample-anp-rule-1
    enabled: true
    priority: 50

    subject:
      matchNamespaces: (1)
        matchExpressions: (2)
          - key: kubernetes.io/metadata.name
            operator: NotIn
            values:
              - kube-system
              - openshift*
              - default
              - kubde-info
1matchNamespaces is used to select namespaces
2matchExpressions is used to select namespaces with matchExpressions. In this example all namespaces that do not match (operator == NotIn) the values, so all namespaces except "kube-system, kube-info, default and openshift*" are selected.

This will result in the following AdminNetworkPolicy snippet:

  subject:
    namespaces:
      matchExpressions:
        - key: kubernetes.io/metadata.name
          operator: NotIn
          values:
            - "kube-system"
            - "openshift*"
            - "default"
            - "kubde-info"

Select Namespaces with matchLabels

Values in the Helm Chart:

anp:
  - name: sample-anp-rule-1
    enabled: true
    priority: 5

    subject:
      matchNamespaces: (1)
        matchLabels: (2)
          apps: my-apps
          tenant: my-tenant
1matchNamespaces is used to select namespaces
2matchLabels is used to select namespaces based on labels. In this example, all namespaces that have the labels "apps: my-apps" and "tenant: my-tenant" are selected.

This will result in the following AdminNetworkPolicy snippet:

spec:
  priority: 5
  subject:
    namespaces:
      matchLabels:
        apps: "my-apps"
        tenant: "my-tenant"

Select Pods with podSelectors and namespaceSelectors

Values in the Helm Chart:

anp:
  - name: sample-anp-rule-1
    enabled: true
    priority: 5

    subject:
      matchPods:
        - pods: (1)
            namespaceSelector: (2)
              labels:
                kubernetes.io/metadata.name: openshift-dns
            podSelector: (3)
              labels:
                app: dns
1matchPods is used to select pods. Here a list of pods can be defined.
2namespaceSelector is used to select namespaces based on labels. In this example all namespaces that have the label "kubernetes.io/metadata.name: openshift-dns" are selected.
3podSelector is used to select pods based on labels. In this example all pods that have the label "app: dns" are selected.

This will result in the following AdminNetworkPolicy snippet:

  subject:
    - pods:
        namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: openshift-dns
        podSelector:
          matchLabels:
            app: dns

BaselineAdminNetworkPolicy

BaselineAdminNetworkPolicy (BANP) is a cluster-scoped resource, that allow cluster administrators to secure the network traffic after NetworkPolicies in the namespaces have been evaluated. These rules can be overwritten by project owners or developers using NetworkPolicies.

BANP is a singleton resource, meaning it can be defined only one time. Therefore, its name must be default. Moreover, the priority field is not required here.

Use cases could be, for example:

  • Creating default rules, such as blocking any intra-cluster traffic by default. Users will need to explicitly use NetworkPolicy objects to allow known traffic.

A BANP allows administrators to specify:

  • A subject that consists of a set of namespaces or namespace.

  • A list of ingress rules to be applied for all ingress traffic towards the subject.

  • A list of egress rules to be applied for all egress traffic from the subject.

BaselineAdminNetworkPolicy Actions for Rules

The BaselineAdminNetworkPolicy allows two actions for the rules. They are like the AdminNetworkPolicy, except for the pass action, which does not make sense here as BANP is the last tier (nowhere to pass).

  • Allow: The traffic is allowed, and no further rules are evaluated.

  • Deny: The traffic is denied, and no further rules are evaluated.

Examples Examples Examples

The following examples are taken directly from Kubernetes Blog: Getting started with the AdminNetworkPolicy API and Official OpenShift Documentation. Verify the values-file of the Helm Chart for the further examples.

I will show, how to configure them using the Helm Chart admin-networkpolicies and the actual result. The chart is already configured with these examples and prepared to be used with GitOps/Argo CD.

Example 1: Allow all traffic from the OpenShift monitoring namespace

Typically, it makes sense to allow the traffic from OpenShift Monitoring to all namespaces. After all, monitoring is useful :) The following example shows the possible configuration for the Helm Chart, which will render a valid ANP resource for us. It will allow ALL (including OpenShift internal Namespaces) traffic from the OpenShift monitoring namespace (labeled as kubernetes.io/metadata.name: monitoring).

---
anp:
  - name: sample-anp-rule-1 (1)
    enabled: true (2)
    syncwave: 10
    priority: 5 (3)

    subject: {} (4)
    ingress: (5)
      - name: allow-ingress-from-monitoring (6)
        enabled: true (7)
        action: Allow (8)
        peers: (9)
          - type: namespaces
            labels:
              kubernetes.io/metadata.name: monitoring
1Name of the ANP
2Enable or disable the ANP. If disabled, the ANP will not be created. (Default is false)
3Priority of the ANP. The lower the value the higher the precedence. (Default is 50)
4Subject of the ANP. In this case, it is empty, which means all namespaces including OpenShift internal namespaces.
5Ingress rules of the ANP. Here a list of ingress rules for this ANP can be defined
6Name of the ingress rule
7Enable or disable the ingress rule. If disabled, the particular ingress rule will not be created. (Default is false)
8Action of the ingress rule. In this case, it is Allow, which means all traffic from the OpenShift monitoring namespace will be allowed. Other options are described at AdminNetworkPolicy Actions for Rules.
9Peers of the ingress rule. In this case, all namespaces labeled as kubernetes.io/metadata.name: monitoring are allowed to access all namespaces.

The ANP that will be created is the following. It is a valid ANP resource and can be applied to the cluster. (Typically applied by Argo CD) As described above it will allow incoming access from the OpenShift monitoring namespace to all namespaces.

---
apiVersion: policy.networking.k8s.io/v1alpha1
kind: AdminNetworkPolicy
metadata:
  name: "sample-anp-rule-1"
  labels:
    helm.sh/chart: admin-networkpolicies-1.0.2
    app.kubernetes.io/name: admin-networkpolicies
    app.kubernetes.io/instance: release-name
    app.kubernetes.io/managed-by: Helm
  annotations:
    argocd.argoproj.io/sync-wave: "10"
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
  priority: 5
  subject:
    namespaces: {}
  ingress:
    - name: "allow-ingress-from-monitoring"
      action: "Allow"
      from:
      - namespaces:
          matchLabels:
            kubernetes.io/metadata.name: "monitoring"

Example 2: Allow all traffic from labeled namespaces

As a second example, we want to allow all traffic from namespaces that are labeled with tenant: restricted to all namespaces that are labeled with anp: cluster-control-anp. This is useful, if you want to restrict access to certain namespaces. However, the rule action is configured as Pass which means that the traffic will be allowed but might be further restricted by a NetworkPolicy in the tenant namespace.

---
anp:
  - name: sample-anp-rule-2
    enabled: true
    priority: 5

    subject:
      matchNamespaces: (1)
        matchLabels:
          anp: cluster-control-anp (2)

    ingress:
      - name: pass-from-restricted-tenants
        enabled: true
        action: Pass (3)
        peers:
          - type: namespaces (4)
            labels:
              tenant: restricted
1Subject of the ANP. In this case, we select based on labels.
2Label selector for the namespaces. In this case, all namespaces that are labeled with anp: cluster-control-anp are subject of this ANP.
3Action of the ingress rule. In this case, it is Pass, which means the traffic is allowed, but might be restricted by NetworkPolicies in the tenant namespace. Other options are described at AdminNetworkPolicy Actions for Rules.
4Peers of the ingress rule. In this case, all namespaces labeled as tenant: restricted are allowed to access all namespaces.
---
apiVersion: policy.networking.k8s.io/v1alpha1
kind: AdminNetworkPolicy
metadata:
  name: "sample-anp-rule-2"
  labels:
    helm.sh/chart: admin-networkpolicies-1.0.2
    app.kubernetes.io/name: admin-networkpolicies
    app.kubernetes.io/instance: release-name
    app.kubernetes.io/managed-by: Helm
  annotations:
    argocd.argoproj.io/sync-wave: "10"
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
  priority: 5
  subject:
    namespaces:
      matchLabels:
        anp: "cluster-control-anp"
  ingress:
    - name: "pass-from-restricted-tenants"
      action: "Pass"
      from:
      - namespaces:
          matchLabels:
            tenant: "restricted"

Example 3: Show possible peers settings

The most important settings for the rules are the peers settings. The following examples show the snippets of possible peers. For further information, please refer to the example in the values file: Helm Chart Values with further examples

The following rules are examples of EGRESS rules.

  1. Allow egress traffic to namespaces labeled splunk on ports 80 and 443:

        peers:
          - type: namespaces
            labels:
              tenant: splunk
        ports:
          - protocol: TCP
            portNumber: 80
          - portName: https

  1. Allow egress traffic to nodes where the key "node-role.kubernetes.io/control-plane" exists on the port 6443:

        peers:
          - type: nodes
            expr:
              - key: node-role.kubernetes.io/control-plane
                operator: Exists
        ports:
          - protocol: TCP
            portNumber: 6443

  1. Allow egress traffic to pods labeled "app: dns" in the namespace openshift-dns on the port 53 and 5353:

        peers:
          - type: pods
            namespaceSelector:
              matchLabels:
                kubernetes.io/metadata.name: openshift-dns
            podSelector:
              matchLabels:
                app: dns

        ports:
          - protocol: TCP
            port: 5353
          - protocol: TCP
            port: 53
          - protocol: UDP
            port: 53
          - protocol: UDP
            port: 5353

  1. Allow egress traffic to a list of IPs:

        peers:
          - type: networks
            ips:
              - 172.29.0.0/30
              - 10.0.54.0/19
              - 10.0.56.38/32
              - 10.0.69.0/24

  1. Allows egress traffic to a list of domain names (*.kubernetes.io and kubernetes.io)

        peers:
          - type: domainNames
            domains:
              - '*.kubernetes.io'
              - kubernetes.io

  1. Deny all egress traffic. This should be the last rule when full egress traffic shall be disabled. This might also be put into the BANP.

      - name: default-deny
        enabled: true
        action: Deny
        peers:
          - type: networks
            ips:
              - 0.0.0.0/0

Example 4: BaselineAdminNetworkPolicy

The BANP is more or less identical to ANP, except that you cannot define a "name" and a "priority". The following example creates a BANP that allows incoming and outgoing traffic to namespaces labeled "tenant-1".

banp: (1)
  - enabled: true (2)
    syncwave: 10

    subject:
      matchNamespaces:
        matchLabels:
          kubernetes.io/metadata.name: example.name

    ingress:
      - name: "deny-all-ingress-from-tenant-1"
        enabled: true
        action: Deny
        peers:
          - type: pods
            namespaceSelector:
              matchLabels:
                custom-banp: tenant-1

            podSelector:
              matchLabels:
                custom-banp: tenant-1

    egress:
      - name: allow-all-egress-to-tenant-1
        enabled: true
        action: Allow
        peers:
          - type: pods
            namespaceSelector:
              matchLabels:
                custom-banp: tenant-1

            podSelector:
              matchLabels:
                custom-banp: tenant-1
1Using the key banp (instead of anp)
2No name or priority are defined here.

Further Information

See Also:

  • Using Kustomize to post render a Helm Chart –

    Lately I came across several issues where a given Helm Chart must be modified after it has been rendered by Argo CD. Argo CD does a helm template to render a Chart. Sometimes, especially when you work with Subcharts or when a specific setting is not yet supported by the Chart, you need to modify it later …​ you need to post-render the Chart. In this very short article, I would like to demonstrate this on a real-live example I had to do. I would like to inject annotations to a Route objects, so that the certificate can be injected. This is done by the cert-utils operator. For the post-rendering the Argo CD repo pod will be extended with a sidecar container, that is watching for the repos and patches them if required.

  • Managing Certificates using GitOps approach –

    The article SSL Certificate Management for OpenShift on AWS explains how to use the Cert-Manager Operator to request and install a new SSL Certificate. This time, I would like to leverage the GitOps approach using the Helm Chart cert-manager I have prepared to deploy the Operator and order new Certificates. I will use an ACME Letsencrypt issuer with a DNS challenge. My domain is hosted at AWS Route 53. However, any other integration can be easily used.

  • Update Cluster Version using GitOps approach –

    During a GitOps journey at one point, the question arises, how to update a cluster? Nowadays it is very easy to update a cluster using CLI or WebUI, so why bother with GitOps in that case? The reason is simple: Using GitOps you can be sure that all clusters are updated to the correct, required version and the version of each cluster is also managed in Git. All you need is the channel you want to use and the desired cluster version. Optionally, you can define the exact image SHA. This might be required when you are operating in a restricted environment.

  • Multiple Sources for Applications in Argo CD –

    Argo CD or OpenShift GitOps uses Applications or ApplicationSets to define the relationship between a source (Git) and a cluster. Typically, this is a 1:1 link, which means one Application is using one source to compare the cluster status. This can be a limitation. For example, if you are working with Helm Charts and a Helm repository, you do not want to re-build (or re-release) the whole chart just because you made a small change in the values file that is packaged into the repository. You want to separate the configuration of the chart with the Helm package. The most common scenarios for multiple sources are (see: Argo CD documentation): Your organization wants to use an external/public Helm chart You want to override the Helm values with your own local values You don’t want to clone the Helm chart locally as well because that would lead to duplication and you would need to monitor it manually for upstream changes. This small article describes three different ways with a working example and tries to cover the advantages and disadvantages of each of them. They might be opinionated but some of them proved to be easier to use and manage.

  • Installing OpenShift Logging using GitOps –

    OpenShift Logging is one of the more complex things to install and configure on an OpenShift cluster. Not because the service or Operators are so complex to understand, but because of the dependencies logging has. Besides the logging operator itself, the Loki operator is required, the Loki operator requires access to an object storage, that might be configured or is already available. In this article, I would like to demonstrate the configuration of the full stack using an object storage from OpenShift Data Foundation. This means: Installing the logging operator into the namespace openshift-logging Installing the Loki operator into the namespace openshift-operators-redhat Creating a new BackingStore and BucketClass Generating the Secret for Loki to authenticate against the object storage Configuring the LokiStack resource Configuring the ClusterLogging resource All steps will be done automatically. In case you have S3 storage available, or you are not using OpenShift Data Foundation, the setup will be a bit different. For example, you do not need to create a BackingStore or the Loki authentication Secret.

Copyright © 2020 - 2024 Toni Schmidbauer & Thomas Jungbauer