Limit Egress/External Traffic

Service Mesh OpenShift

- Thomas Jungbauer Thomas Jungbauer ( Lastmod: 2024-05-05 ) - 5 min read

Sometimes services are only available from outside the OpenShift cluster (like external API) which must be reached. Part 7 of OpenShift 4 and Service Mesh takes care and explains how to control the egress or external traffic. All operations have been successdully tested on OpenShift 4.3.

Preparation

Before this tutorial can be started, ensure that 3 microservices are deployed (recommendation may have 2 versions) and that the objects Gateway and VirtualService are configured. The status should be like in Issue #4..6

You can verify this the following way:

export GATEWAY_URL=$(oc -n istio-system get route istio-ingressgateway -o jsonpath='{.spec.host}')
curl $GATEWAY_URL

which should simply print:

customer => preference => recommendation v1 from 'f11b097f1dd0': 7123

Setup recommendation-v3

We need to deploy version 3 of our recommendation microservice. This will perform an external API call to http://worldclockapi.com to retrieve the current time.

To deploy the Deployment v3:

cd ~/istio-tutorial/recommendation
oc apply -f kubernetes/Deployment-v3.yml -n tutorial
If you list the pods at this moment, you will see that only one container (Ready 1/1) is started. This happens because the Deployment yaml file is missing an annotation.

Fixing missing proxy sidecar container

After you applied the Deployment-v3.yml, only 1 container is started. The proxy sidecar is not injected, because an annotation is missing in the configuration for the Deployment.

To fix this use the following command:

oc patch deployment recommendation-v3 -n tutorial -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar.istio.io/inject":"true"}}}}}'

This will automatically restart the pod with 2 containers.

Create DestinationRule and VirtualService

Use the following definition to create (overwrite) the DestinationRule for recommendation-v3.

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: recommendation
spec:
  host: recommendation
  subsets:
  - labels:
      version: v3
    name: version-v3
Only version 3 is used for now. The other versions are still there, but ignored for our tests.

Apply the change

oc apply -f DestinationRule_v3.yaml

Define the VirtualService and send 100% of the traffic to v3 of the recommendation microservice.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: recommendation
spec:
  hosts:
  - recommendation
  http:
  - route:
    - destination:
        host: recommendation
        subset: version-v3
      weight: 100
As an alternative, you can also edit the existing VirtualService and add the section for version-v3 with a weight of 100, while changing the weight of v1 and v2 to 0.

Test egress traffic

As usual we test our application by sending traffic to it. The following command should print successful connection requests:

sh ~/run.sh 1000 $GATEWAY_URL
# 0: customer => preference => recommendation v3 2020-04-06T18:31+02:00 from '83bbb6d11a7e': 1
# 1: customer => preference => recommendation v3 2020-04-06T18:31+02:00 from '83bbb6d11a7e': 2
# 2: customer => preference => recommendation v3 2020-04-06T18:31+02:00 from '83bbb6d11a7e': 3
# 3: customer => preference => recommendation v3 2020-04-06T18:31+02:00 from '83bbb6d11a7e': 4
# 4: customer => preference => recommendation v3 2020-04-06T18:31+02:00 from '83bbb6d11a7e': 5

As you can see 100% of the traffic is sent to v3 AND a new field enters the output. The current time is now shown as well. The information for this field is fetched with an external API call to http://worldclockapi.com.

The traffic is simply sent to an external destination. There is not limit yet. Readers of the Istio documentation will miss the object ServiceEntry which somebody should think is required. However, Openshift is currently(?) configured in a way to simply allow ANY traffic. This is defined in a ConfigMap which might be changed to modify the default behavior. However, as soon as ServiceEntry and the appropriate VirtualService is configured, the traffic will be limited as well.

Limit/Control external access

As you can see above you can simply send egress traffic without any control about what is allowed or not. In order to limit your outgoing traffic a new object called ServiceEntry must be defined as well as a change in your VirtualService will be required.

Define the ServiceEntry and apply it to your cluster:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: worldclockapi-egress-rule
spec:
  hosts:
  - worldclockapi.com
  ports:
  - name: http-80
    number: 81 (1)
    protocol: http
1Wrong port 81 is set on purpose for demonstration
The port number: 81 is set on purpose, to prove that the traffic will not work with a wrong ServiceEntry.
oc create -f ServiceEntry.yaml

To actually limit the traffic a link between the ServiceEntry and a VirtualService, which defines the external destination, must be created. Moreover, a timeout is set for possible connection errors, to keep the application responding even when the external API is down.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: worldclockapi-timeout (1)
spec:
  hosts:
    - worldclockapi.com (2)
  http:
  - timeout: 3s (3)
    route:
      - destination:
          host: worldclockapi.com
        weight: 100 (4)
1The name of the object
2The external hostname we want to reach
3The timeout setting in seconds
4The destination route, which is sending 100% of the external traffic to the host above
oc apply -f VirtualService-worldclockapi.yaml

If you now run a connection test you will still get an error.

sh ~/run.sh 1 $GATEWAY_URL

# customer => Error: 503 - preference => Error: 500 ...

Fix ServiceEntry

This happens, because we misconfigured the ServiceEntry on purpose to demonstrate that the traffic is sent to worldclockapi.com:80.

Fix the ServiceEntry object and apply to your cluster:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: worldclockapi-egress-rule
spec:
  hosts:
  - worldclockapi.com
  ports:
  - name: http-80
    number: 80 (1)
    protocol: http
1Changed from 81 to 80
oc apply -f ServiceEntry.yaml

Now the traffic should work and gives you back a connection to microservice and a current time:

sh ~/run.sh 10 $GATEWAY_URL

# 0: customer => preference => recommendation v3 2020-04-07T07:47+02:00 from '83bbb6d11a7e': 138
# 1: customer => preference => recommendation v3 2020-04-07T07:47+02:00 from '83bbb6d11a7e': 139
# 2: customer => preference => recommendation v3 2020-04-07T07:47+02:00 from '83bbb6d11a7e': 140
# 3: customer => preference => recommendation v3 2020-04-07T07:47+02:00 from '83bbb6d11a7e': 141
# 4: customer => preference => recommendation v3 2020-04-07T07:47+02:00 from '83bbb6d11a7e': 142
# 5: customer => preference => recommendation v3 2020-04-07T07:47+02:00 from '83bbb6d11a7e': 143
# 6: customer => preference => recommendation v3 2020-04-07T07:47+02:00 from '83bbb6d11a7e': 144

Verify Kiali

Kiali with external service
Figure 1. Kiali shows traffic to the external service

OPTIONAL: Disallow ANY connections

This is a change in the default ConfigMap of the ServiceMesh. Do this on your own risk and always consult the latest documentation of OCP.

As explained above, we are able to connect to an external service without any limitation. The ServiceEntry object together with the VirtualService define the actual destination and would disallow traffic if they are wrongly configured, but if you forget these entries, it would still be possible to establish an egress connection.

In OpenShift a ConfigMap in the istio-system namespace defines the default behavior. There are two possibilities:

  • ALLOW_ANY - outbound traffic to unknown destinations will be allowed, in case there are no services or ServiceEntries for the destination port

  • REGISTRY_ONLY - restrict outbound traffic to services defined in the service registry as well

    1. Let’s Cleanup the ServiceEntry and the VirtualService which have been created above

      oc delete serviceentry worldclockapi-egress-rule
serviceentry.networking.istio.io "worldclockapi-egress-rule" deleted

oc delete virtualservice worldclockapi-timeout
virtualservice.networking.istio.io "worldclockapi-timeout" deleted
      Now traffic to the external service will be allowed again

    2. Modify the ConfigMap istio in the namespace istio-system

      oc get configmap istio -n istio-system -o yaml | sed 's/mode: ALLOW_ANY/mode: REGISTRY_ONLY/g' | oc replace -n istio-system -f -

Wait a few seconds and try to connect. You will see that the connection is not possible anymore.

If you now re-create the ServiceEntry the connection will be possible again, since the service is registered to the Service Mesh.

See Also:

  • Introducing AdminNetworkPolicies –

    Classic Kubernetes/OpenShift offer a feature called NetworkPolicy that allows users to control the traffic to and from their assigned Namespace. NetworkPolicies are designed to give project owners or tenants the ability to protect their own namespace. Sometimes, however, I worked with customers where the cluster administrators or a dedicated (network) team need to enforce these policies. Since the NetworkPolicy API is namespace-scoped, it is not possible to enforce policies across namespaces. The only solution was to create custom (project) admin and edit roles, and remove the ability of creating, modifying or deleting NetworkPolicy objects. Technically, this is possible and easily done. But shifts the whole network security to cluster administrators. Luckily, this is where AdminNetworkPolicy (ANP) and BaselineAdminNetworkPolicy (BANP) comes into play.

  • Using Kustomize to post render a Helm Chart –

    Lately I came across several issues where a given Helm Chart must be modified after it has been rendered by Argo CD. Argo CD does a helm template to render a Chart. Sometimes, especially when you work with Subcharts or when a specific setting is not yet supported by the Chart, you need to modify it later …​ you need to post-render the Chart. In this very short article, I would like to demonstrate this on a real-live example I had to do. I would like to inject annotations to a Route objects, so that the certificate can be injected. This is done by the cert-utils operator. For the post-rendering the Argo CD repo pod will be extended with a sidecar container, that is watching for the repos and patches them if required.

  • Managing Certificates using GitOps approach –

    The article SSL Certificate Management for OpenShift on AWS explains how to use the Cert-Manager Operator to request and install a new SSL Certificate. This time, I would like to leverage the GitOps approach using the Helm Chart cert-manager I have prepared to deploy the Operator and order new Certificates. I will use an ACME Letsencrypt issuer with a DNS challenge. My domain is hosted at AWS Route 53. However, any other integration can be easily used.

  • Update Cluster Version using GitOps approach –

    During a GitOps journey at one point, the question arises, how to update a cluster? Nowadays it is very easy to update a cluster using CLI or WebUI, so why bother with GitOps in that case? The reason is simple: Using GitOps you can be sure that all clusters are updated to the correct, required version and the version of each cluster is also managed in Git. All you need is the channel you want to use and the desired cluster version. Optionally, you can define the exact image SHA. This might be required when you are operating in a restricted environment.

  • Multiple Sources for Applications in Argo CD –

    Argo CD or OpenShift GitOps uses Applications or ApplicationSets to define the relationship between a source (Git) and a cluster. Typically, this is a 1:1 link, which means one Application is using one source to compare the cluster status. This can be a limitation. For example, if you are working with Helm Charts and a Helm repository, you do not want to re-build (or re-release) the whole chart just because you made a small change in the values file that is packaged into the repository. You want to separate the configuration of the chart with the Helm package. The most common scenarios for multiple sources are (see: Argo CD documentation): Your organization wants to use an external/public Helm chart You want to override the Helm values with your own local values You don’t want to clone the Helm chart locally as well because that would lead to duplication and you would need to monitor it manually for upstream changes. This small article describes three different ways with a working example and tries to cover the advantages and disadvantages of each of them. They might be opinionated but some of them proved to be easier to use and manage.

Copyright © 2020 - 2024 Toni Schmidbauer & Thomas Jungbauer