YAUB Yet Another Useless Blog> Service Mesh> Limit Egress/External Traffic

Limit Egress/External Traffic

April 6, 2020 - By: Thomas Jungbauer ( Lastmod: 2021-08-14 ) - 5 min read

Sometimes services are only available from outside the OpenShift cluster (like external API) which must be reached. Part 7 of OpenShift 4 and Service Mesh takes care and explains how to control the egress or external traffic. All operations have been successdully tested on OpenShift 4.3.

Preparation

Before this tutorial can be started, ensure that 3 microservices are deployed (recommendation may have 2 versions) and that the objects Gateway and VirtualService are configured. The status should be like in Issue #4..6

You can verify this the following way:

export GATEWAY_URL=$(oc -n istio-system get route istio-ingressgateway -o jsonpath='{.spec.host}')
curl $GATEWAY_URL

which should simply print:

customer => preference => recommendation v1 from 'f11b097f1dd0': 7123

Setup recommendation-v3

We need to deploy version 3 of our recommendation microservice. This will perform an external API call to http://worldclockapi.com to retrieve the current time.

To deploy the Deployment v3:

cd ~/istio-tutorial/recommendation
oc apply -f kubernetes/Deployment-v3.yml -n tutorial

If you list the pods at this moment, you will see that only one container (Ready 1/1) is started. This happens because the Deployment yaml file is missing an annotation.

Fixing missing proxy sidecar container

After you applied the Deployment-v3.yml, only 1 container is started. The proxy sidecar is not injected, because an annotation is missing in the configuration for the Deployment.

To fix this use the following command:

oc patch deployment recommendation-v3 -n tutorial -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar.istio.io/inject":"true"}}}}}'

This will automatically restart the pod with 2 containers.

Create DestinationRule and VirtualService

Use the following definition to create (overwrite) the DestinationRule for recommendation-v3.

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: recommendation
spec:
  host: recommendation
  subsets:
  - labels:
      version: v3
    name: version-v3

Only version 3 is used for now. The other versions are still there, but ignored for our tests.

Apply the change

oc apply -f DestinationRule_v3.yaml

Define the VirtualService and send 100% of the traffic to v3 of the recommendation microservice.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: recommendation
spec:
  hosts:
  - recommendation
  http:
  - route:
    - destination:
        host: recommendation
        subset: version-v3
      weight: 100

As an alternative, you can also edit the existing VirtualService and add the section for version-v3 with a weight of 100, while changing the weight of v1 and v2 to 0.

Test egress traffic

As usual we test our application by sending traffic to it. The following command should print successful connection requests:

sh ~/run.sh 1000 $GATEWAY_URL

# 0: customer => preference => recommendation v3 2020-04-06T18:31+02:00 from '83bbb6d11a7e': 1
# 1: customer => preference => recommendation v3 2020-04-06T18:31+02:00 from '83bbb6d11a7e': 2
# 2: customer => preference => recommendation v3 2020-04-06T18:31+02:00 from '83bbb6d11a7e': 3
# 3: customer => preference => recommendation v3 2020-04-06T18:31+02:00 from '83bbb6d11a7e': 4
# 4: customer => preference => recommendation v3 2020-04-06T18:31+02:00 from '83bbb6d11a7e': 5

As you can see 100% of the traffic is sent to v3 AND a new field enters the output. The current time is now shown as well. The information for this field is fetched with an external API call to http://worldclockapi.com.

The traffic is simply sent to an external destination. There is not limit yet. Readers of the Istio documentation will miss the object ServiceEntry which somebody should think is required. However, Openshift is currently(?) configured in a way to simply allow ANY traffic. This is defined in a ConfigMap which might be changed to modify the default behavior. However, as soon as ServiceEntry and the appropriate VirtualService is configured, the traffic will be limited as well.

Limit/Control external access

As you can see above you can simply send egress traffic without any control about what is allowed or not. In order to limit your outgoing traffic a new object called ServiceEntry must be defined as well as a change in your VirtualService will be required.

Define the ServiceEntry and apply it to your cluster:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: worldclockapi-egress-rule
spec:
  hosts:
  - worldclockapi.com
  ports:
  - name: http-80
    number: 81 (1)
    protocol: http

1	Wrong port 81 is set on purpose for demonstration

The port number: 81 is set on purpose, to prove that the traffic will not work with a wrong ServiceEntry.

oc create -f ServiceEntry.yaml

To actually limit the traffic a link between the ServiceEntry and a VirtualService, which defines the external destination, must be created. Moreover, a timeout is set for possible connection errors, to keep the application responding even when the external API is down.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: worldclockapi-timeout (1)
spec:
  hosts:
    - worldclockapi.com (2)
  http:
  - timeout: 3s (3)
    route:
      - destination:
          host: worldclockapi.com
        weight: 100 (4)

1	The name of the object
2	The external hostname we want to reach
3	The timeout setting in seconds
4	The destination route, which is sending 100% of the external traffic to the host above

oc apply -f VirtualService-worldclockapi.yaml

If you now run a connection test you will still get an error.

sh ~/run.sh 1 $GATEWAY_URL

# customer => Error: 503 - preference => Error: 500 ...

Fix ServiceEntry

This happens, because we misconfigured the ServiceEntry on purpose to demonstrate that the traffic is sent to worldclockapi.com:80.

Fix the ServiceEntry object and apply to your cluster:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: worldclockapi-egress-rule
spec:
  hosts:
  - worldclockapi.com
  ports:
  - name: http-80
    number: 80 (1)
    protocol: http

1	Changed from 81 to 80

oc apply -f ServiceEntry.yaml

Now the traffic should work and gives you back a connection to microservice and a current time:

sh ~/run.sh 10 $GATEWAY_URL

# 0: customer => preference => recommendation v3 2020-04-07T07:47+02:00 from '83bbb6d11a7e': 138
# 1: customer => preference => recommendation v3 2020-04-07T07:47+02:00 from '83bbb6d11a7e': 139
# 2: customer => preference => recommendation v3 2020-04-07T07:47+02:00 from '83bbb6d11a7e': 140
# 3: customer => preference => recommendation v3 2020-04-07T07:47+02:00 from '83bbb6d11a7e': 141
# 4: customer => preference => recommendation v3 2020-04-07T07:47+02:00 from '83bbb6d11a7e': 142
# 5: customer => preference => recommendation v3 2020-04-07T07:47+02:00 from '83bbb6d11a7e': 143
# 6: customer => preference => recommendation v3 2020-04-07T07:47+02:00 from '83bbb6d11a7e': 144

Verify Kiali

Figure 1. Kiali shows traffic to the external service

OPTIONAL: Disallow ANY connections

This is a change in the default ConfigMap of the ServiceMesh. Do this on your own risk and always consult the latest documentation of OCP.

As explained above, we are able to connect to an external service without any limitation. The ServiceEntry object together with the VirtualService define the actual destination and would disallow traffic if they are wrongly configured, but if you forget these entries, it would still be possible to establish an egress connection.

In OpenShift a ConfigMap in the istio-system namespace defines the default behavior. There are two possibilities:

ALLOW_ANY - outbound traffic to unknown destinations will be allowed, in case there are no services or ServiceEntries for the destination port

REGISTRY_ONLY - restrict outbound traffic to services defined in the service registry as well

Let’s Cleanup the ServiceEntry and the VirtualService which have been created above

oc delete serviceentry worldclockapi-egress-rule
serviceentry.networking.istio.io "worldclockapi-egress-rule" deleted

oc delete virtualservice worldclockapi-timeout
virtualservice.networking.istio.io "worldclockapi-timeout" deleted

Now traffic to the external service will be allowed again

Modify the ConfigMap istio in the namespace istio-system

oc get configmap istio -n istio-system -o yaml | sed 's/mode: ALLOW_ANY/mode: REGISTRY_ONLY/g' | oc replace -n istio-system -f -

Wait a few seconds and try to connect. You will see that the connection is not possible anymore.

If you now re-create the ServiceEntry the connection will be possible again, since the service is registered to the Service Mesh.

Built with Hugo Learn Theme and Hugo