Service Mesh on TechBlog about OpenShift/Ansible/Satellite and much more

Enable Automatic Route Creation

Wed, 13 May 2020 00:00:00 +0000

Red Hat Service Mesh 1.1 allows you to enable a "Automatic Route Creation" which will take care about the routes for a specific Gateway. Instead of defining * for hosts, a list of domains can be defined. The Istio OpenShift Routing (ior) synchronizes the routes and creates them inside the Istio namespace. If a Gateway is deleted, the routes will also be removed again.

This new features makes the manual creation of the route obsolete, as it was explained here: Openshift 4 and Service Mesh 4 - Ingress with custom domain

Enable Automatic Route Creation

Before this feature can be used, it must be enabled. To do so the ServiceMeshContolPlace, typically found in the namespace istio-system must be modified. Add the line ior_enabled: true to the istio-ingressgate configuration.

...
spec:
istio:
gateways:
istio-egressgateway:
autoscaleEnabled: false
istio-ingressgateway:
autoscaleEnabled: false
ior_enabled: true
...

Verify current service

Let’s check our tutorial application, if it is still working.

oc project tutorial
export GATEWAY_URL=$(oc -n istio-system get route istio-ingressgateway -o jsonpath='{.spec.host}')
curl $GATEWAY_URL/customer
customer => preference => recommendation v1 from 'f11b097f1dd0': 30

Let’s review and remove the current used Gateway. As you can see the hosts is set to '*'

oc get istio-io
oc get gateway.networking.istio.io/customer-gateway -o yaml
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"networking.istio.io/v1alpha3","kind":"Gateway","metadata":{"annotations":{},"name":"customer-gateway","namespace":"tutorial"},"spec":{"selector":{"istio":"ingressgateway"},"servers":[{"hosts":["*"],"port":{"name":"http","number":80,"protocol":"HTTP"}}]}}
creationTimestamp: "2020-05-13T07:52:20Z"
generation: 1
name: customer-gateway
namespace: tutorial
resourceVersion: "41370056"
selfLink: /apis/networking.istio.io/v1alpha3/namespaces/tutorial/gateways/customer-gateway
uid: 96e82ed9-e870-493c-941f-bfa83c892b94
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- '*'
port:
name: http
number: 80
protocol: HTTP

Create a new Gateway

First let’s remove the current Gateway

oc delete gateway.networking.istio.io/customer-gateway

Now lets create a new Gateway, but this time we define some names for the hosts section:

cat <<'EOF' > Gateway-ior.yaml
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: customer-gateway
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- www.example.com
- svc.example.com
EOF
oc apply -f Gateway-ior.yaml -n tutorial

When you now check the routes, 2 new routes have been added:

oc get routes -n istio-system
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
...
tutorial-customer-gateway-kmqrl www.example.com istio-ingressgateway http2 None
tutorial-customer-gateway-ks7q7 svc.example.com istio-ingressgateway http2 None

To test the connectivity, you need to be sure that the hosts, used in the Gateway, are resolvable. If they are then you can access your service:

curl www.example.com/customer
customer => preference => recommendation v1 from 'f11b097f1dd0': 31
curl svc.example.com/customer
customer => preference => recommendation v1 from 'f11b097f1dd0': 32

Authorization (RBAC)

Tue, 12 May 2020 00:00:00 +0000

Per default all requests inside a Service Mesh are allowed, which can be a problem security-wise. To solve this, authorization, which verifies if the user is allowed to perform a certain action, is required. Istio’s authorization provides access control on mesh-level, namespace-level and workload-level.

With the resource AuthorizationPolicy granular policies can be defined. These policies are loaded to and verified by the Envoy Proxy which then authorizes a request.

Implicit enablement

To enable authorization the only thing you need is to do is to define the AuthorizationPolicy. If the resource is not defined, then no access control will be used, instead any traffic is allowed. If AuthorizationPolicy is applied to a workload, then by default any traffic is denied unless it is explicitly allowed.

This is applicable to Service Mesh version 1.1+

Preparing Environment

The following steps will configure an example Role Based Access Control (RBAC). It will start from scratch. If you just want to quickly configure the authorization and have anything else in place, you can start form here: Configure Authentication Policy

Create a new project
```
oc new-project tutorial
```

Be sure that a Service Mesh Member Roll exists for this new project

cat <<'EOF' > memberroll.yaml
apiVersion: maistra.io/v1
kind: ServiceMeshMemberRoll
metadata:
name: default
spec:
members:
- tutorial
EOF
oc apply -f memberroll.yaml -n istio-system

Clone and install the example application

git clone https://github.com/redhat-developer-demos/istio-tutorial/ istio-tutorial
oc apply -f istio-tutorial/customer/kubernetes/Deployment.yml -n tutorial
oc apply -f istio-tutorial/customer/kubernetes/Service.yml -n tutorial
oc expose service customer
oc apply -f istio-tutorial/preference/kubernetes/Deployment.yml -n tutorial
oc apply -f istio-tutorial/preference/kubernetes/Service.yml -n tutorial
oc apply -f istio-tutorial/recommendation/kubernetes/Deployment.yml -n tutorial
oc apply -f istio-tutorial/recommendation/kubernetes/Service.yml -n tutorial

Wait until all pods are running. There should be 2 containers for all pods:

oc get pods -w

Create Gateway and VirtualService

cat <<'EOF' > Gateway_VirtualService.yaml
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: customer-gateway
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: customer-gateway
spec:
hosts:
- "*"
gateways:
- customer-gateway
http:
- match:
- uri:
prefix: /customer
rewrite:
uri: /
route:
- destination:
host: customer
port:
number: 8080
EOF
oc apply -f Gateway_VirtualService.yaml -n tutorial

Verify if the application is working You can either use the run.sh from previous tutorials, or simply try the following curl

export GATEWAY_URL=$(oc -n istio-system get route istio-ingressgateway -o jsonpath='{.spec.host}'); echo $GATEWAY_URL
curl $GATEWAY_URL/customer

This should return the following line:

customer => preference => recommendation v1 from 'f11b097f1dd0': 1

Optionally check the connection from inside the customer container

get pods name and enter it:

oc get pods
NAME READY STATUS RESTARTS AGE
customer-6948b8b959-dhsm9 2/2 Running 0 177m
preference-v1-7fdb89c86b-dvzs9 2/2 Running 0 177m
recommendation-v1-69db8d6c48-cjcpn 2/2 Running 0 177m

Connect into the container pod and try to reach the different microservices

oc rsh customer-6948b8b959-dhsm9
sh-4.4$ curl customer:8080
customer => preference => recommendation v1 from 'f11b097f1dd0': 2
sh-4.4$ curl preference:8080
preference => recommendation v1 from 'f11b097f1dd0': 3
sh-4.4$ curl recommendation:8080
recommendation v1 from 'f11b097f1dd0': 4

Configure Authentication Policy

Enabling User-End authentication

cat <<'EOF' > authentication-policy.yaml
apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
name: "customerjwt"
spec:
targets:
- name: customer
- name: preference
- name: recommendation
origins:
- jwt:
issuer: "testing@secure.istio.io"
jwksUri: "https://gist.githubusercontent.com/lordofthejars/7dad589384612d7a6e18398ac0f10065/raw/ea0f8e7b729fb1df25d4dc60bf17dee409aad204/jwks.json"
principalBinding: USE_ORIGIN
EOF
oc apply -f authentication-policy.yaml -n tutorial

Access should be denied after a few seconds

curl $GATEWAY_URL/customer
Origin authentication failed.%

Use token to authenticate

token=$(curl https://gist.githubusercontent.com/lordofthejars/a02485d70c99eba70980e0a92b2c97ed/raw/f16b938464b01a2e721567217f672f11dc4ef565/token.simple.jwt -s)
curl -H "Authorization: Bearer $token" $GATEWAY_URL/customer

This will result in a correct response

customer => preference => recommendation v1 from 'f11b097f1dd0': 5

Configure Role Based Access Control (RBAC)

Create the resource AuthorizationPolicy

This is a new resources, supported since Service Mesh 1.1. It will allow GET method when the role equals to "customer"

cat <<'EOF' > AuthorizationPolicy.yaml
apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
name: "customer"
spec:
rules:
- to:
- operation:
methods: ["GET"]
when:
- key: request.auth.claims[role]
values: ["customer"]
EOF
oc apply -f AuthorizationPolicy.yaml -n tutorial

Get a token for the role and retry to connect to the service,

token=$(curl https://gist.githubusercontent.com/lordofthejars/f590c80b8d83ea1244febb2c73954739/raw/21ec0ba0184726444d99018761cf0cd0ece35971/token.role.jwt -s)
curl -H "Authorization: Bearer $token" $GATEWAY_URL/customer

This results in:

customer => preference => recommendation v1 from 'f11b097f1dd0': 8

Let’s verify the setting and change the AuthorizationPolicy. This will break the authorization, since the token provides roles=customer and we set the Policy to "whereistherole"

cat <<'EOF' > AuthorizationPolicy-Hack.yaml
apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
name: "customer"
spec:
rules:
- to:
- operation:
methods: ["GET"]
when:
- key: request.auth.claims[role]
values: ["whereistherole"]
EOF
oc replace -f AuthorizationPolicy-Hack.yaml -n tutorial

If you now try to access the service, with the token, which provides "customer" as role, it will lead to an error:

token=$(curl https://gist.githubusercontent.com/lordofthejars/f590c80b8d83ea1244febb2c73954739/raw/21ec0ba0184726444d99018761cf0cd0ece35971/token.role.jwt -s)
curl -H "Authorization: Bearer $token" $GATEWAY_URL/customer
RBAC: access denied

Deploy Example Bookinfo Application

Thu, 30 Apr 2020 00:00:00 +0000

To test a second application, a bookinfo application shall be deployed as an example.

The following section finds it’s origin at:

The Bookinfo application displays information about a book, similar to a single catalog entry of an online book store.
Displayed on the page is a description of the book, book details (ISBN, number of pages, and other information), and book reviews.
The Bookinfo application consists of these microservices:
* The productpage microservice calls the details and reviews microservices to populate the page.
* The details microservice contains book information.
* The reviews microservice contains book reviews. It also calls the ratings microservice.
* The ratings microservice contains book ranking information that accompanies a book review.
There are three versions of the reviews microservice:
* Version v1 does not call the ratings Service.
* Version v2 calls the ratings Service and displays each rating as one to five black stars.
* Version v3 calls the ratings Service and displays each rating as one to five red stars.
The end-to-end architecture of the application is shown below.

Figure 1. Bookinfo Application End2End Overview

To use the bookinfo application inside service mesh, no code changes are required. Instead an Envoy proxy is added as a sidecar container to all containers (product, review, details) which intercepts the traffic.

Installation

Let’s start right away:

Create a new project
```
oc new-project bookinfo
```

Add the new project to our Service Mesh

apiVersion: maistra.io/v1
kind: ServiceMeshMember
metadata:
name: default
namespace: bookinfo
spec:
controlPlaneRef:
name: basic-install
namespace: istio-system

Create the application

oc apply -n bookinfo -f https://raw.githubusercontent.com/Maistra/istio/maistra-1.1/samples/bookinfo/platform/kube/bookinfo.yaml

Create the Gateway and the VirtuaService

oc apply -n bookinfo -f https://raw.githubusercontent.com/Maistra/istio/maistra-1.1/samples/bookinfo/networking/bookinfo-gateway.yaml

Check if the services and pods are up and running

oc get svc,pods
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/details ClusterIP 172.30.178.172 <none> 9080/TCP 7m16s
service/productpage ClusterIP 172.30.78.96 <none> 9080/TCP 7m13s
service/ratings ClusterIP 172.30.154.12 <none> 9080/TCP 7m15s
service/reviews ClusterIP 172.30.138.174 <none> 9080/TCP 7m14s
NAME READY STATUS RESTARTS AGE
pod/details-v1-d7db4d55b-mwzsk 2/2 Running 0 7m14s
pod/productpage-v1-5f598fbbf4-svkbc 2/2 Running 0 7m11s
pod/ratings-v1-85957d89d8-v2lrs 2/2 Running 0 7m11s
pod/reviews-v1-67d9b4bcc-x6s2v 2/2 Running 0 7m11s
pod/reviews-v2-67b465c497-zpz6z 2/2 Running 0 7m11s
pod/reviews-v3-7bd659b757-j6rwn 2/2 Running 0 7m11s

Verify that application is accessible

Export the Gateway URL into a variable

export GATEWAY_URL=$(oc -n istio-system get route istio-ingressgateway -o jsonpath='{.spec.host}')

Verify if the productpage is accessible

curl -s http://${GATEWAY_URL}/productpage | grep -o "<title>.*</title>"
<title>Simple Bookstore App</title>

You can also access the Productpage in your browser. When you reload the page several times, you will see different results for the Reviews. This comes due to 3 different versions: one without any rating, one with black stars and one with red stars. http://${GATEWAY_URL}/productpage

Figure 2. Bookinfo Application

Adding default Destination Rule

oc apply -n bookinfo -f https://raw.githubusercontent.com/Maistra/istio/maistra-1.1/samples/bookinfo/networking/destination-rule-all-mtls.yaml

This will add default routing to all endpoints with same weight. As you can see in Kiali, the Reviews microservice is contacted equally.

Figure 3. Kiali: Bookinfo Application

Feel free to play with other DestinationRules to controll your traffic.

Service Mesh 1.1 released

Fri, 10 Apr 2020 00:00:00 +0000

April 10th 2020 Red Hat released Service Mesh version 1.1 which supports the following versions:

Istio - 1.4.6
Kiali - 1.12.7
Jaeger - 1.17.1

Update

To update an operator like Service Mesh, the Operator Life Cycle Manager takes care and automatically updates everything (unless it was configured differently).

For the Service Mesh 1.1 update consult Upgrading Red Hat OpenShift Service Mesh
It is important to add the version number to the ServiceMeshControlPlane object. The easiest way to do so is:

Log into OpenShift
Select the Namespace istio-system
Goto _"Installed Operators > Red Hat OpenShift Service Mesh > ServiceMeshControlPlanes > basic-install > YAML"
Under spec add the following:
```
spec:
version: v1.1
```

Notable Changes

ServiceMeshMember Object

With the ServiceMeshMember object it is now possible that a project administrator can add a service to the service mesh, instead relying on the cluster administrator to configure the ServiceMeshMemberRoll. To do so create the following object (i.e. under the namespace tutorial)

apiVersion: maistra.io/v1
kind: ServiceMeshMember
metadata:
name: default
namespace: tutorial
spec:
controlPlaneRef:
name: basic-install (1)
namespace: istio-system (2)

1	Name of the ServiceMeshControlPlane object
2	name of the service mesh namespace

Authentication JWT

Thu, 09 Apr 2020 00:00:00 +0000

Welcome to tutorial 10 of OpenShift 4 and Service Mesh, where we will discuss authentication with JWT. JSON Web Token (JWT) is an open standard that allows to transmit information between two parties securely as a JSON object. It is an authentication token, which is verified and signed and therefore trusted. The signing can be achieved by using a secret or a public/private key pair.

Service Mesh can be used to configure a policy which enables JWT for your services.

Preparation

Be sure that you have at least the Gateway and VirtualService configured:

oc get istio-io -n tutorial

Which should return the following:

NAME AGE
gateway.networking.istio.io/ingress-gateway-exampleapp 45h
NAME HOST AGE
destinationrule.networking.istio.io/recommendation recommendation 29h
NAME GATEWAYS HOSTS AGE
virtualservice.networking.istio.io/ingress-gateway-exampleapp [ingress-gateway-exampleapp] [*] 45h

Run some texample traffic, to be sure that our application is still working as expected

export GATEWAY_URL=$(oc -n istio-system get route istio-ingressgateway -o jsonpath='{.spec.host}')
sh ~/run.sh 1000 $GATEWAY_URL

# 0: customer => preference => recommendation v2 from '3cbba7a9cde5': 31622
# 1: customer => preference => recommendation v1 from 'f11b097f1dd0': 33056
# 2: customer => preference => recommendation v2 from '3cbba7a9cde5': 31623
# 3: customer => preference => recommendation v1 from 'f11b097f1dd0': 33057
# 4: customer => preference => recommendation v2 from '3cbba7a9cde5': 31624
# 5: customer => preference => recommendation v1 from 'f11b097f1dd0': 33058

Enabling End-User Authentication

To test this feature we will need a valid token (JWT). More details can be found at the Istio example

All we need to create a Policy object

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
name: "jwt-example"
spec:
targets:
- name: customer
origins:
- jwt:
issuer: "testing@secure.istio.io"
jwksUri: "https://raw.githubusercontent.com/istio/istio/release-1.2/security/tools/jwt/samples/jwks.json" (1)
principalBinding: USE_ORIGIN

1	Path to test a public key

After a few seconds the requests will fail with an "authentication failed" error:

sh ~/run.sh 1000 $GATEWAY_URL

# 0: Origin authentication failed.
# 1: Origin authentication failed.
# 2: Origin authentication failed.
# 3: Origin authentication failed.
# 4: Origin authentication failed.
# 5: Origin authentication failed.

In Kiali we see a 100% failure rate.

Figure 1. Kiali: failing because of authentication error.

To be able to connect to our application we first need to fetch a valid token and put this into the header while sending curl.

export GATEWAY_URL=$(oc -n istio-system get route istio-ingressgateway -o jsonpath='{.spec.host}')
export TOKEN=$(curl https://raw.githubusercontent.com/istio/istio/release-1.1/security/tools/jwt/samples/demo.jwt -s)
for x in $(seq 1 1000);
do curl --header "Authorization: Bearer $TOKEN" $GATEWAY_URL -s;
done

In Kiali the traffic is now working again and authenticated.

Figure 2. Kiali: Traffic authenticated.

Clean Up

Remove the policy again, to be ready for the next tutorial.

oc delete policy jwt-example -n tutorial

Mutual TLS Authentication

Wed, 08 Apr 2020 00:00:00 +0000

When more and more microservices are involved in an application, more and more traffic is sent on the network. It should be considered to secure this traffic, to prevent the possibility to inject malicious packets. Mutual TLS/mTLS authentication or two-way authentication offers a way to encrypt service traffic with certificates.

With Red Hat OpenShift Service Mesh, Mutual TLS can be used without the microservice knowing that it is happening. The TLS is managed completely by the Service Mesh Operator between two Envoy proxies using a defined mTLS policy.

Issue 9 of OpenShift 4 and Service Mesh will explain how to enable Mutual TLS inside the Service Mesh to secure the traffic between the different microservices.

How does it work?

If a microservice sends a request to a server, it must pass the local sidecar Envoy proxy first.
The proxy will intercept the outbound request and starts a mutual TLS handshake with the proxy at the server side. During this handshake the certificates are exchanged and loaded into the proxy containers by Service Mesh.
The client side Envoy starts a mutual TLS handshake with the server side Envoy.
The client proxy does a secure naming check on the server’s certificate to verify that the identity in the certificate is authorized.
A mutual TLS connection is established between the client and the server.
The Envoy proxy at the server sides decrypts the traffic and forwards it to the application through a local TCP connection.

Preparations

Before we can start be sure that the services are setup like in Issue #3.
In addition, be sure that the following DestinationRule already exists:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: recommendation
spec:
host: recommendation
subsets:
- labels:
version: v1
name: version-v1
- labels:
version: v2
name: version-v2

Now we will create a pod, which is running outside of the Service Mesh. It will not have a sidecar proxy and will simply curl our application.

Store the following yaml and create the object in our cluster.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
app: curl
version: v1
name: curl
spec:
replicas: 1
selector:
matchLabels:
app: curl
version: v1
template:
metadata:
labels:
app: curl
version: v1
annotations: (1)
sidecar.istio.io/proxyCPU: "500m"
sidecar.istio.io/proxyMemory: 400Mi
spec:
containers:
- image: quay.io/maistra_demos/curl:latest
command: ["/bin/sleep", "3650d"]
imagePullPolicy: Always
name: curl

1	since no sidecar is injected (sidecar.istio.io/inject: "true"), only 1 container will be started.

The traffic coming from the microservice customer AND from the external client curl must be simulated. To achieve this the following shell script can be used:

#!/bin/sh
export CURL_POD=$(oc get pods -n tutorial -l app=curl | grep curl | awk '{ print $1}' )
export CUSTOMER_POD=$(oc get pods -n tutorial -l app=customer | grep customer | awk '{ print $1}' )
echo "A load generating script is running in the next step. Ctrl+C to stop"
while :; do
echo "Executing curl in curl pod"
oc exec -n tutorial $CURL_POD -- curl -s http://preference:8080 > /dev/null
sleep 0.5
echo "Executing curl in customer pod"
oc exec -n tutorial $CUSTOMER_POD -c customer -- curl -s http://preference:8080 > /dev/null
sleep 0.5
done

By executing this, it will first execute a curl command out of the curl pod and then the same curl command out of the customer container. Kepp this script running

Enabling Mutual TLS

Lets execute the shell script above and verify Kiali. As you notice there are requests coming from the customer microservice and from the source called unknown, which is the curl-service running outside the Service Mesh.

Figure 1. Kiali: traffic coming from customer microserver and external pod

Enable the policy by creating the following object:

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
name: "preference-mutualtls"
spec:
targets:
- name: preference
peers:
- mtls:
mode: STRICT (1)

1	We are enforcing mtls for the target preference

After a few seconds the curl pod cannot reach the application anymore:

Executing curl in curl pod
command terminated with exit code 56
Executing curl in customer pod
Executing curl in curl pod
command terminated with exit code 56
Executing curl in customer pod
Executing curl in curl pod
command terminated with exit code 5

This is expected, since the preference service allows traffic over mutual TLS only. This was enforced by the Policy object (STRICT mode). The customer service, which is running inside the Service Mesh receives the error "5053 Service Unavalable" since it tries to send traffic, but it does not know yet to use mTLS.

In Kiali you will see the following:

Figure 2. Kiali: traffic is blocked

The curl pod is greyed out, since the traffic it tries to send, never reaches the preference service and is therefor not counted in the metric.

To make customer aware that mutual TLS shall be used, a DestinationRule must be configured:

apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
name: "preference-destination-rule"
spec:
host: "preference"
trafficPolicy:
tls:
mode: ISTIO_MUTUAL (1)

1	Let’s use mTLS

This defines that ISTIO_MUTUAL shall be used for the service preference. The customer service recognizes this and automatically enables mTLS. After a few minutes the traffic graph in Kiali will show "green" traffic from customer through preference to _recommendation:

Figure 3. Kiali: traffic for Service Mesh components is fine again.

Mutual TLS Migration

As you can see in the previous section, the curl pod cannot reach the application inside the Service Mesh. This happens because prefernce is strictly enforcing encrypted traffic, but curl only sends plain text. Luckily, Istio provides a method to gradually monitor the traffic and migrate to mTLS. Instead of STRICT mode PERMISSIVE can be used. Enabling permissive mode, preference will accept both, encrypted and plain-text traffic.

Replace the Policy object with the following configuration:

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
name: "preference-mutualtls"
spec:
targets:
- name: preference
peers:
- mtls:
mode: PERMISSIVE

oc replace -f Policy-permissive.yaml

Now let’s wait a few minutes and observe Kiali, which should end up with:

Figure 4. Kiali: Encrypted and Plain-Text traffic

As you can see with the lock icon, the traffic between cunstomer and preference is encrypted, while the traffic from unknown (which is our curl pod), is plain-text.

The errors you may see in Kiali happen due a known issue: https://issues.jboss.org/browse/MAISTRA-1000

Cleanup

Clean up your environment:

oc delete policy -n tutorial preference-mutualtls
oc delete destinationrule -n tutorial preference-destination-rule

Fault Injection

Tue, 07 Apr 2020 00:00:00 +0000

Tutorial 8 of OpenShift 4 and Service Mesh tries to cover Fault Injection by using Chaos testing method to verify if your application is running. This is done by adding the property HTTPFaultInjection to the VirtualService. The settings for this property can be for example: delay, to delay the access or abort, to completely abort the connection.

"Adopting microservices often means more dependencies, and more services you might not control. It also means more requests on the network, increasing the possibility for errors. For these reasons, it’s important to test your services’ behavior when upstream dependencies fail." [1]

Preparation

Before we start this tutorial, we need to clean up our cluster. This is especially important when you did the previous training Limit Egress/External Traffic.

oc delete deployment recommendation-v3
oc scale deployment recommendation-v2 --replicas=1
oc delete serviceentry worldclockapi-egress-rule
oc delete virtualservice worldclockapi-timeout

Verify that 2 pods for the recommendation services are running (with 2 containers)

oc get pods -l app=recommendation -n tutorial
NAME READY STATUS RESTARTS AGE
recommendation-v1-69db8d6c48-h8brv 2/2 Running 0 4d20h
recommendation-v2-6c5b86bbd8-jnk8b 2/2 Running 0 4d19h

Abort Connection with HTTP Error 503

For the first example, we will need to modify the VirtualService and the DestinationRule. The VirtualService must be extended with a http fault section, which will abort the traffic 50% of the time.

Create the VirtualService

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: recommendation
spec:
hosts:
- recommendation
http:
- fault:
abort:
httpStatus: 503
percent: 50
route:
- destination:
host: recommendation
subset: app-recommendation

Apply the change

oc replace -f VirtualService-abort.yaml

Existing VirtualService with the name recommendation will be overwritten.

Create the DestinationRule

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: recommendation
spec:
host: recommendation
subsets:
- labels:
app: recommendation
name: app-recommendation

Apply the change

oc replace -f destinationrule-faultinj.yaml

Existing Destination with the name recommendation will be overwritten.

Check the traffic and verify that 50% of the connections will end with a 503 error:

export INGRESS_GATEWAY=$(oc get route customer -n tutorial -o 'jsonpath={.spec.host}')
sh ~/run.sh 1000 $GATEWAY_URL

Clean Up

oc delete virtualservice recommendation

Test slow connection with Delay

More interesting, in my opinion, to test is a slow connection. This can be tested by adding the fixedDelay property into the VirtualService. Like in the example below, we will use a VirtualService. This time delay instead of abort is used. The fixDelay defines a delay of 7 seconds for 50% of the traffic.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: recommendation
spec:
hosts:
- recommendation
http:
- fault:
delay:
fixedDelay: 7.000s
percent: 50
route:
- destination:
host: recommendation
subset: app-recommendation

If you now send traffic into the application, you will see that some answers will have a delay of 7 seconds. Keep sending traffic in a loop.

Even more visible it will be, when you goto "Distributed Tracing" at the Kiali UI, select the service recommendation and a small lookback of maybe 5min. You will find that some requests are very fast, while other will tage about 7 seconds.

Figure 1. Jaeger with delayed traffic.

Retry on errors

If a microservice is answering with an error, Service Mesh/Istio will automatically try to reach another pod providing the service. These retries can be modified. In order to make everything visible, we will use Kiali to monitor the traffic.

We start by sending traffic into the application. This should be split evenly between v1 and v2 of the recommendation microservice

sh ~/run.sh 1000 $GATEWAY_URL

# 8329: customer => preference => recommendation v1 from 'f11b097f1dd0': 11145
# 8330: customer => preference => recommendation v2 from '3cbba7a9cde5': 9712
# 8331: customer => preference => recommendation v1 from 'f11b097f1dd0': 11146
# 8332: customer => preference => recommendation v2 from '3cbba7a9cde5': 9713
# 8333: customer => preference => recommendation v1 from 'f11b097f1dd0': 11147

In Kiali this ia visible in the Graphs, using the settings: "Versioned app graph" and "Requests percentage"

Figure 2. Traffic is split by 50% between recommendation v1 nd v2

As second step we need to enable the nasty mode for the microservice v2. This will simulate an outage, respoding with error 503 all the time. This change must be done inside the container:
```
oc exec -it $(oc get pods|grep recommendation-v2|awk '{ print $1 }'|head -1) -c recommendation /bin/bash
```
Inside the container use the following command and exit the container again
```
curl localhost:8080/misbehave
```
Kiali will now show that v1 will get 100% of the traffic, while v2 is shown as red. When you select the red square of v2 and then move the mouse over the red cross for the failing application, you will see that the pd itself is ready, but that 100% of the traffic is currently failing.

Figure 3. Traffic for v2 is failing
revert the change and fix v2 service
```
oc exec -it $(oc get pods|grep recommendation-v2|awk '{ print $1 }'|head -1) -c recommendation /bin/bash
```
```
curl localhost:8080/behave
```
Verify in Kiali that everything is "green" again and that the traffic is split by 50% between v1 and v2.

Sources

[1]: Istio By Example - Fault Injection

Limit Egress/External Traffic

Mon, 06 Apr 2020 00:00:00 +0000

Sometimes services are only available from outside the OpenShift cluster (like external API) which must be reached. Part 7 of OpenShift 4 and Service Mesh takes care and explains how to control the egress or external traffic. All operations have been successdully tested on OpenShift 4.3.

Preparation

Before this tutorial can be started, ensure that 3 microservices are deployed (recommendation may have 2 versions) and that the objects Gateway and VirtualService are configured. The status should be like in Issue #4..6

You can verify this the following way:

export GATEWAY_URL=$(oc -n istio-system get route istio-ingressgateway -o jsonpath='{.spec.host}')
curl $GATEWAY_URL

which should simply print:

customer => preference => recommendation v1 from 'f11b097f1dd0': 7123

Setup recommendation-v3

We need to deploy version 3 of our recommendation microservice. This will perform an external API call to http://worldclockapi.com to retrieve the current time.

To deploy the Deployment v3:

cd ~/istio-tutorial/recommendation
oc apply -f kubernetes/Deployment-v3.yml -n tutorial

If you list the pods at this moment, you will see that only one container (Ready 1/1) is started. This happens because the Deployment yaml file is missing an annotation.

Fixing missing proxy sidecar container

After you applied the Deployment-v3.yml, only 1 container is started. The proxy sidecar is not injected, because an annotation is missing in the configuration for the Deployment.

To fix this use the following command:

oc patch deployment recommendation-v3 -n tutorial -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar.istio.io/inject":"true"}}}}}'

This will automatically restart the pod with 2 containers.

Create DestinationRule and VirtualService

Use the following definition to create (overwrite) the DestinationRule for recommendation-v3.

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: recommendation
spec:
host: recommendation
subsets:
- labels:
version: v3
name: version-v3

Only version 3 is used for now. The other versions are still there, but ignored for our tests.

Apply the change

oc apply -f DestinationRule_v3.yaml

Define the VirtualService and send 100% of the traffic to v3 of the recommendation microservice.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: recommendation
spec:
hosts:
- recommendation
http:
- route:
- destination:
host: recommendation
subset: version-v3
weight: 100

As an alternative, you can also edit the existing VirtualService and add the section for version-v3 with a weight of 100, while changing the weight of v1 and v2 to 0.

Test egress traffic

As usual we test our application by sending traffic to it. The following command should print successful connection requests:

sh ~/run.sh 1000 $GATEWAY_URL

# 0: customer => preference => recommendation v3 2020-04-06T18:31+02:00 from '83bbb6d11a7e': 1
# 1: customer => preference => recommendation v3 2020-04-06T18:31+02:00 from '83bbb6d11a7e': 2
# 2: customer => preference => recommendation v3 2020-04-06T18:31+02:00 from '83bbb6d11a7e': 3
# 3: customer => preference => recommendation v3 2020-04-06T18:31+02:00 from '83bbb6d11a7e': 4
# 4: customer => preference => recommendation v3 2020-04-06T18:31+02:00 from '83bbb6d11a7e': 5

As you can see 100% of the traffic is sent to v3 AND a new field enters the output. The current time is now shown as well. The information for this field is fetched with an external API call to http://worldclockapi.com.

The traffic is simply sent to an external destination. There is not limit yet. Readers of the Istio documentation will miss the object ServiceEntry which somebody should think is required. However, Openshift is currently(?) configured in a way to simply allow ANY traffic. This is defined in a ConfigMap which might be changed to modify the default behavior. However, as soon as ServiceEntry and the appropriate VirtualService is configured, the traffic will be limited as well.

Limit/Control external access

As you can see above you can simply send egress traffic without any control about what is allowed or not. In order to limit your outgoing traffic a new object called ServiceEntry must be defined as well as a change in your VirtualService will be required.

Define the ServiceEntry and apply it to your cluster:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: worldclockapi-egress-rule
spec:
hosts:
- worldclockapi.com
ports:
- name: http-80
number: 81 (1)
protocol: http

1	Wrong port 81 is set on purpose for demonstration

The port number: 81 is set on purpose, to prove that the traffic will not work with a wrong ServiceEntry.

oc create -f ServiceEntry.yaml

To actually limit the traffic a link between the ServiceEntry and a VirtualService, which defines the external destination, must be created. Moreover, a timeout is set for possible connection errors, to keep the application responding even when the external API is down.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: worldclockapi-timeout (1)
spec:
hosts:
- worldclockapi.com (2)
http:
- timeout: 3s (3)
route:
- destination:
host: worldclockapi.com
weight: 100 (4)

1	The name of the object
2	The external hostname we want to reach
3	The timeout setting in seconds
4	The destination route, which is sending 100% of the external traffic to the host above

oc apply -f VirtualService-worldclockapi.yaml

If you now run a connection test you will still get an error.

sh ~/run.sh 1 $GATEWAY_URL
# customer => Error: 503 - preference => Error: 500 ...

Fix ServiceEntry

This happens, because we misconfigured the ServiceEntry on purpose to demonstrate that the traffic is sent to worldclockapi.com:80.

Fix the ServiceEntry object and apply to your cluster:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: worldclockapi-egress-rule
spec:
hosts:
- worldclockapi.com
ports:
- name: http-80
number: 80 (1)
protocol: http

1	Changed from 81 to 80

oc apply -f ServiceEntry.yaml

Now the traffic should work and gives you back a connection to microservice and a current time:

sh ~/run.sh 10 $GATEWAY_URL
# 0: customer => preference => recommendation v3 2020-04-07T07:47+02:00 from '83bbb6d11a7e': 138
# 1: customer => preference => recommendation v3 2020-04-07T07:47+02:00 from '83bbb6d11a7e': 139
# 2: customer => preference => recommendation v3 2020-04-07T07:47+02:00 from '83bbb6d11a7e': 140
# 3: customer => preference => recommendation v3 2020-04-07T07:47+02:00 from '83bbb6d11a7e': 141
# 4: customer => preference => recommendation v3 2020-04-07T07:47+02:00 from '83bbb6d11a7e': 142
# 5: customer => preference => recommendation v3 2020-04-07T07:47+02:00 from '83bbb6d11a7e': 143
# 6: customer => preference => recommendation v3 2020-04-07T07:47+02:00 from '83bbb6d11a7e': 144

Verify Kiali

Figure 1. Kiali shows traffic to the external service

OPTIONAL: Disallow ANY connections

This is a change in the default ConfigMap of the ServiceMesh. Do this on your own risk and always consult the latest documentation of OCP.

As explained above, we are able to connect to an external service without any limitation. The ServiceEntry object together with the VirtualService define the actual destination and would disallow traffic if they are wrongly configured, but if you forget these entries, it would still be possible to establish an egress connection.

In OpenShift a ConfigMap in the istio-system namespace defines the default behavior. There are two possibilities:

ALLOW_ANY - outbound traffic to unknown destinations will be allowed, in case there are no services or ServiceEntries for the destination port

REGISTRY_ONLY - restrict outbound traffic to services defined in the service registry as well

Let’s Cleanup the ServiceEntry and the VirtualService which have been created above

oc delete serviceentry worldclockapi-egress-rule
serviceentry.networking.istio.io "worldclockapi-egress-rule" deleted
oc delete virtualservice worldclockapi-timeout
virtualservice.networking.istio.io "worldclockapi-timeout" deleted

Now traffic to the external service will be allowed again

Modify the ConfigMap istio in the namespace istio-system

oc get configmap istio -n istio-system -o yaml | sed 's/mode: ALLOW_ANY/mode: REGISTRY_ONLY/g' | oc replace -n istio-system -f -

Wait a few seconds and try to connect. You will see that the connection is not possible anymore.

If you now re-create the ServiceEntry the connection will be possible again, since the service is registered to the Service Mesh.

Advanced Routing Example

Fri, 03 Apr 2020 00:00:00 +0000

Welcome to part 6 of OpenShift 4 and Service Mesh Advanced routing, like Canary Deployments, traffic mirroring and loadbalancing are discussed and tested. All operations have been successdully tested on OpenShift 4.3.

Advanced Routing

During Issue #5 some simple routing was implemented. The traffic was split by 100% to a new version (v2) of the recommendation microservice. This section shall give a brief overview of advanced routing possibilities.

Canary Deployments

A canary deployment is a strategy to roll out a new version of your service by using traffic splitting. A small amount of traffic (10%) will be sent to the new version, while most of the traffic will be sent to the old version still. The traffic to the new version can be analysed and if everything works as expected more and more traffic can be sent to the new version.

To enable split traffic, the VirtualService must be update:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: recommendation
spec:
hosts:
- recommendation
http:
- route:
- destination:
host: recommendation
subset: version-v1
weight: 90
- destination:
host: recommendation
subset: version-v2
weight: 10

Apply the change

oc apply -f VitualService_split_v1_and_v1.yaml

Test the traffic and verify that 10% will be sent to v2

sh ~/run.sh 100 $GATEWAY_URL
# 0: customer => preference => recommendation v1 from 'f11b097f1dd0': 1060
# 1: customer => preference => recommendation v1 from 'f11b097f1dd0': 1061
# 2: customer => preference => recommendation v2 from '3cbba7a9cde5': 2060
# 3: customer => preference => recommendation v1 from 'f11b097f1dd0': 1062
# 4: customer => preference => recommendation v1 from 'f11b097f1dd0': 1063
...

If an error is shown, then you most probably forget to configure the DestinationRule as described here.

Figure 1. Kiali split traffic 90/10

Routing based on user-agent header

It is possible to send traffic to different versions based on the browser type which is calling the application. In our test application the service customer is setting the header baggage-user-agent and propagates it to the other services.

>> headers.putSingle("baggage-user-agent", userAgent);

Create the following file

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: recommendation
spec:
hosts:
- recommendation
http:
- match:
- headers:
baggage-user-agent:
regex: .*Safari.*
route:
- destination:
host: recommendation
subset: version-v2
- route:
- destination:
host: recommendation
subset: version-v1

and apply the change

oc apply -f VitualService_safari.yaml

In order to test the result, either use the appropriate browser or use curl to set the user-agent. As expected, request from Safari are sent to v2, other are sent to v1.

Safari

curl -v -A Safari $GATEWAY_URL
[...]
> User-Agent: Safari
[...]
customer => preference => recommendation v2 from '3cbba7a9cde5': 2365

Firefox

curl -v -A Firefox $GATEWAY_URL
[...]
> User-Agent: Firefox
[...]
customer => preference => recommendation v1 from 'f11b097f1dd0': 3762

Mirroring Traffic

Mirroring Traffic, aka Dark Launch, will duplicate the traffic to another service, allowing you to analyse it before sending production data to it. Responses of the mirrored requests are ignored.

Run the following command and be sure that recommendation-v1 and recommendation-v2 are both running:

oc get pod -n tutorial| grep recommendation
recommendation-v1-69db8d6c48-h8brv 2/2 Running 0 24h
recommendation-v2-6c5b86bbd8-jnk8b 2/2 Running 0 23h

Update the VirtualService, so that version v2 will receive mirrored traffic, while the actual request will be sent to v1:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: recommendation
spec:
hosts:
- recommendation
http:
- route:
- destination:
host: recommendation
subset: version-v1
mirror: (1)
host: recommendation
subset: version-v2

1	This must be set to 'mirror'

Apply the change

oc apply -f VitualService_mirrored-traffic.yaml

Now lets open and follow the logs of recommandation-v2 in order to see that traffic will reach this service, but responses are ignored:

oc logs -f $(oc get pods|grep recommendation-v2|awk '{ print $1 }') -c recommendation

In a second terminal window send some traffic to our service.

sh ~/run.sh 100 $GATEWAY_URL

You will see that only v1 answers, while in the 2nd window, v2 gets the same traffic.

Load Balancing

In the default OpenShift environment the kube-proxy forwards all requests to pods randomly. With Red Hat ServiceMesh it is possible to add more complexity and let the Envoy proxy handle load balancing for your services.

Three methods are supported:

random
round-robin
least connection

The round robin function is used by default, when there is no DestinationRule configured. We can use the DestinationRule to use the least connection option to see how the traffic is sent.

Before we start we need to delete the VirtualService for the recommendation microservice

oc delete virtualservice recommendation

The we scale version v2 to 3:

oc scale deployment recommendation-v2 --replicas=3

After a few seconds the folling pods should run now:

NAME READY STATUS RESTARTS AGE
customer-6948b8b959-jdjlg 2/2 Running 1 25h
preference-v1-7fdb89c86b-nktqn 2/2 Running 0 25h
recommendation-v1-69db8d6c48-h8brv 2/2 Running 0 25h
recommendation-v2-6c5b86bbd8-6lgz6 2/2 Running 0 91s
recommendation-v2-6c5b86bbd8-dnc8b 2/2 Running 0 91s
recommendation-v2-6c5b86bbd8-jnk8b 2/2 Running 0 24h

If you send traffic to the application, you would see that 3 quarter are sent to v1 and one is sent to v1.

With the following DestinationRule the traffic will be sent randomly to the application

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: recommendation
spec:
host: recommendation
trafficPolicy:
loadBalancer:
simple: RANDOM

If you now sent traffic to the service, you will see that the traffic is sent randomly to the versions. (verify the serial number)

sh ~/run.sh 100 $GATEWAY_URL
# 140: customer => preference => recommendation v2 from '3cbba7a9cde5': 5729
# 141: customer => preference => recommendation v1 from 'f11b097f1dd0': 7119
# 142: customer => preference => recommendation v2 from '3cbba7a9cde5': 361
# 143: customer => preference => recommendation v2 from '3cbba7a9cde5': 362
# 144: customer => preference => recommendation v2 from '3cbba7a9cde5': 5730
# 145: customer => preference => recommendation v2 from '3cbba7a9cde5': 362
# 146: customer => preference => recommendation v1 from 'f11b097f1dd0': 7120
# 147: customer => preference => recommendation v1 from 'f11b097f1dd0': 7121
# 148: customer => preference => recommendation v2 from '3cbba7a9cde5': 363
...

Routing Example

Wed, 01 Apr 2020 00:00:00 +0000

In part 5 of the OpenShift 4 and Service Mesh tutorials, basic routing, using the objects VirtualService and DesitnationRule, are described. All operations have been successfully tested on OpenShift 4.3.

Some Theory

In this section another version of the recommendation microservice will be deployed. The traffic to the new version will be controlled with different settings of the VirtualService. Multiple scenarios can be realized with ServiceMesh. In general these are defined as follows ([1]):

Blue-Green Deployments

In a Blue-Green deployment the old version (green) is kept running, while a new version (blue) is deployed and tested. When testing is successful, the 100% of the traffic is switched to the new version. If there is any error, the traffic could be switched back to the green version.

A/B Deployments

A/B deployments, in difference to Blue-Green deployments, will enable you to try a new version of the application in a limited way in the production environment. It is possible to specify that the production version gets most of the user requests, while a limited number of requests is sent to the new version. This could be specified by location of the user for example, so that all users from Vienna are sent to the new version, while all others are still using the old version.

Canary Deployments

Canary releases can be used to allow a small, minimum amount of traffic to the new version of your application. This traffic can be increased gradually until all traffic is sent to the new version. If any issues are found, you can roll back and send the traffic to the old version.

Prerequisites

It is assumed that an OpenShift environment is up and running and that Issues #1 - #3 are done at least:

Prepare Simple Routing

OPTIONAL: Build the recommendation microservice

If you want to locally build the microservice, you must change the source code from version v1 to v2 the following way:

Open the file:

istio-tutorial/recommendation/java/vertx/src/main/java/com/redhat/developer/demos/recommendation/RecommendationVerticle.java

and change the following line from v1 to v2

private static final String RESPONSE_STRING_FORMAT = "recommendation v2 from '%s': %d\n";

Now you can build the image:

cd istio-tutorial/recommendation/java/vertx
mvn package
podman build -t example/recommendation:v2 . (1)

1	Note the v2 tag

Create second deployment with version2

A deployment with our recommendation:v2 microservice must be created. A service object must not be created this time, as it already exists.

cd ~/istio-tutorial/recommendation/
oc apply -f kubernetes/Deployment-v2.yml -n tutorial
oc get pods -w

If you want to diff v1 and v2 deployment, you will notice that the main change is the image which gets pulled.

diff recommendation/kubernetes/Deployment-v2.yml recommendation/kubernetes/Deployment.yml
6,7c6,7
< version: v2
< name: recommendation-v2
---
> version: v1
> name: recommendation-v1
13c13
< version: v2
---
> version: v1
18c18
< version: v2
---
> version: v1
27c27
< image: quay.io/rhdevelopers/istio-tutorial-recommendation:v2.1
---
> image: quay.io/rhdevelopers/istio-tutorial-recommendation:v1.1

Call application

Execute the test command to access the application. Since no rules are defined yet, the traffic is split by 50% to version 1 and version 2 (round robin):

sh ~/run.sh 10 $GATEWAY_URL
# 0: customer => preference => recommendation v2 from '3cbba7a9cde5': 27
# 1: customer => preference => recommendation v1 from 'f11b097f1dd0': 27
# 2: customer => preference => recommendation v2 from '3cbba7a9cde5': 28
# 3: customer => preference => recommendation v1 from 'f11b097f1dd0': 28
...

In Kiali presents this as well:

Figure 1. Kiali sends 50% to v1 and v2

Send all traffic to recommendation:v2

To route the traffic accordingly a DestinationRule and a VirtualService must be created for recommendation. While the DesinationRule will add a name to each version, VirtualService specifies the actual destination of the traffic.

Define DestinationRule for recommendation

The object DestinationRule will define the versions in subsets.

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: recommendation
spec:
host: recommendation
subsets:
- labels:
version: v1
name: version-v1
- labels:
version: v2
name: version-v2

Create the object with the command: oc create -f <filename>

Define VirtualService for recommendation

The VirtualService defines that 100% (weight) of the traffic for recomendation (host) will be sent to the subset (version-v2), which is defined in the DefinationRule

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: recommendation
spec:
hosts:
- recommendation
http:
- route:
- destination:
host: recommendation
subset: version-v2
weight: 100

Create the object with the command: oc create -f <filename>

Call application

If you now call the application, only traffic to v2 should be shown:

sh ~/run.sh 1000 $GATEWAY_URL
# 0: customer => preference => recommendation v2 from '3cbba7a9cde5': 27
# 1: customer => preference => recommendation v1 from 'f11b097f1dd0': 27
# 2: customer => preference => recommendation v2 from '3cbba7a9cde5': 28
# 3: customer => preference => recommendation v1 from 'f11b097f1dd0': 28
...

In Kiali presents this as well and send 100% of the traffic to recommendation:v2:

Figure 2. Kiali sends 100% to v2

Sources

[1]: DZone: Traffic Management With Istio

Ingress with custom domain

Tue, 31 Mar 2020 00:00:00 +0000

Since Service Mesh 1.1, there is a better way to achieve the following. Especially the manual creation of the route is not required anymore. Check the following article to Enable Automatic Route Creation.

Often the question is how to get traffic into the Service Mesh when using a custom domains. Part 4 our our tutorials series OpenShift 4 and Service Mesh will use a dummy domain "hello-world.com" and explains the required settings which must be done.

Modify Gateway and VirtualService

Issue #3 explains how to get ingress traffic into the Service Mesh, by defining the Gateway and the VirtualService. We are currently using the default ingress route defined in the istio_system project.
But what if a custom domain shall be used?
In such case another route must be defined in the istio-system project and small configuration changes must be applied.

First lets create a slightly modified Gateway.yaml:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: ingress-gateway-exampleapp
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "hello-world.com" (1)

1	add you custom domain here The only difference is at the hosts which was changed from '*' to 'hello-world.com'

As second change, the VirtualService must be modified as well with the custom domain:

VirtualService.yaml:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: ingress-gateway-exampleapp
spec:
hosts:
- "hello-world.com" (1)
gateways:
- ingress-gateway-exampleapp
http:
- match:
- uri:
exact: /
route:
- destination:
host: customer
port:
number: 8080

1	add you custom domain here

Replace current objects in OpenShift:

oc replace -f Gateway.yaml -n tutorial
oc replace -f VirtualService.yaml -n tutorial

Create a new route under the project istio-system:

apiVersion: route.openshift.io/v1
kind: Route
metadata:
name: hello-world.com (1)
namespace: istio-system (2)
spec:
host: hello-world.com (3)
to:
kind: Service
name: istio-ingressgateway (4)
port:
targetPort: 8080

1	add you custom domain here
2	the route must be created at istio-system
3	add you custom domain here
4	this is the service as it was created by the operator

OPTIONAL: Add custom domain to local hosts file

The custom domain hello-world.com must be resolvable somehow, pointing to the ingress router of OpenShift. This can be done, by adding the domain into the local hosts file (with all limitations this brings with it)

# Get IP address of:
oc -n istio-system get route istio-ingressgateway
echo "x.x.x.x hello-world.com" >> /etc/hosts

Create some example traffic

We will reuse the script of Issue #3 to simulate traffic. Since we changed the domain, the connection will go to hello-world.com

sh run-check.sh 1000 hello-world.com

This will send 1000 requests to our application:

# 0: customer => preference => recommendation v1 from 'f11b097f1dd0': 6626
# 1: customer => preference => recommendation v1 from 'f11b097f1dd0': 6627
# 2: customer => preference => recommendation v1 from 'f11b097f1dd0': 6628
# 3: customer => preference => recommendation v1 from 'f11b097f1dd0': 6629
# 4: customer => preference => recommendation v1 from 'f11b097f1dd0': 6630
# 5: customer => preference => recommendation v1 from 'f11b097f1dd0': 6631
...

Ingress Traffic

Mon, 30 Mar 2020 00:00:00 +0000

Part 3 of tutorial series OpenShift 4 and Service Mesh will show you how to create a Gateway and a VirtualService, so external traffic actually reaches your Mesh. It also provides an example script to run some curl in a loop.

Configure Gateway and VirtualService Example

With the microservices deployed during Issue #2, it makes sense to test the access somehow. In order to bring traffic into the application a Gateway object and a VirtualService object must be created.

The Gateway will be the entry point which forward the traffic to the istio ingressgateway

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: ingress-gateway-exampleapp
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"

As 2nd object a VirtualService must be created:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: ingress-gateway-exampleapp
spec:
hosts:
- "*"
gateways:
- ingress-gateway-exampleapp
http:
- match:
- uri:
exact: /
route:
- destination:
host: customer
port:
number: 8080

Get all istio-io related objects of your project. These objects represent the network objects of Service Mesh, like Gateway, VirtualService and DestinationRule (explained later)

oc get istio-io -n tutorial
NAME HOST AGE
destinationrule.networking.istio.io/recommendation recommendation 3d21h
NAME AGE
gateway.networking.istio.io/ingress-gateway 4d15h
NAME GATEWAYS HOSTS AGE
virtualservice.networking.istio.io/ingress-gateway [ingress-gateway] [*] 4d15h

Create some example traffic

Before we start, lets fetch the default route of our Service Mesh:

export GATEWAY_URL=$(oc -n istio-system get route istio-ingressgateway -o jsonpath='{.spec.host}')

This should return: istio-ingressgateway-istio-system.apps.<clustername>

Now, let’s create a shell script to run some curl commands in a loop and can be easily reused for other scenarios:

#!/bin/bash
numberOfRequests=$1
host2check=$2
if [ $# -eq 0 ]; then
echo "better define: <script> #ofrequests hostname2check"
echo "Example: run.sh 100 hello.com"
let "numberOfRequests=100"
else
let "i = 0"
while [ $i -lt $numberOfRequests ]; do
echo -n "# $i: "; curl $2
let "i=$((i + 1))"
done
fi

Run the script and check the output:

sh run-check.sh 1000 $GATEWAY_URL

This will send 1000 requests to our application:

# 0: customer => preference => recommendation v1 from 'f11b097f1dd0': 3622
# 1: customer => preference => recommendation v1 from 'f11b097f1dd0': 3623
# 2: customer => preference => recommendation v1 from 'f11b097f1dd0': 3624
# 3: customer => preference => recommendation v1 from 'f11b097f1dd0': 3625
# 4: customer => preference => recommendation v1 from 'f11b097f1dd0': 3626
# 5: customer => preference => recommendation v1 from 'f11b097f1dd0': 3627
...

Verify in Kiali

To verify in Kiali our application, open the URL in your browser and login using your OpenShift credentials.

If you do not know the URL for Kiali, execute the following command
oc get route kiali -n istio-system

Switch the the Graph view and you should see the following picture:

Figure 1. Kiali Graph

Deploy Microservices

Sun, 29 Mar 2020 00:00:00 +0000

The second tutorials explains how to install an example application containing thee microservices. All operations have been successfully tested on OpenShift 4.3.

Introduction

As quickly explained at Issue #1, OpenShift 4.3 and the Service Mesh shall be installed already. At this point you should have all 4 operators installed and ready. The test application used in this scenario is based on Java and contains 3 microservices in the following traffic flow:

Customer ⇒ Preference ⇒ Recomandation

The microservices are the same as used at the Interactive Learning Portal.

Deploy microservices

First lets create a new project for our tests
```
oc new-project tutorial
```
EITHER: Add tutorial to ServiceMeshMemberRoll

Service Mesh 1.1 now supports the object ServiceMember which can created under the application namespace and which automatically configures the ServiceMemberRoll. However, below description still works.

+ OpenShift will auto-inject the sidecar proxy to pods of a namespace, if the namespace is configured in the ServiceMeshMemberRoll object which was created for the operator.

Create the file memberroll.yaml:

cat <<'EOF' > memberroll.yaml
apiVersion: maistra.io/v1
kind: ServiceMeshMemberRoll
metadata:
name: default
spec:
members:
- tutorial
EOF

Add the namespace to the member roll:

oc apply -f memberroll.yaml -n istio-system

If you already have namespaces configured for ServiceMeshMemberRoll, better modify the object manually. Custom Resource Definitions (CRD) do not like to be modified on the fly (currently?)

OR: Create ServiceMeshMember object: Available since ServiceMesh 1.1

apiVersion: maistra.io/v1
kind: ServiceMeshMember
metadata:
name: default
namespace: tutorial
spec:
controlPlaneRef:
name: basic-install
namespace: istio-system

Download the tutorial application locally

git clone https://github.com/redhat-developer-demos/istio-tutorial/

Deploy microservice: customer

To deploy the first microservice 2 objects (Deployment and Service) must be created. Moreover, the service will be exposed, since the service customer is our entry point.
Verify ~/istio-tutorial/customer/kubernetes/Deployment.yml to check where the actual image is coming from: quay.io/rhdevelopers/istio-tutorial-customer:v1.1

cd ~/istio-tutorial/customer
oc apply -f kubernetes/Deployment.yml -n tutorial
oc apply -f kubernetes/Service.yml -n tutorial
oc expose service customer

Check if pods are running with two containers:

oc get pods -w

The result should look somehow like this:

NAME READY STATUS RESTARTS AGE
customer-6948b8b959-g77bs 2/2 Running 0 52m

If there is only 1 container running, indicated by READY = 1/1, then most likely the ServiceMeshMemberRoll was not updated with the name tutorial or contains a wrong project name.

Deploy microservice: preference

The deployment of the microservice preference is exactly like it is done for customer, except that no service must be exposed:

cd ~/istio-tutorial/preference/
oc apply -f kubernetes/Deployment.yml -n tutorial
oc apply -f kubernetes/Service.yml -n tutorial

Check if pods are running with two containers:

oc get pods -w

The result should look somehow like this:

NAME READY STATUS RESTARTS AGE
customer-6948b8b959-g77bs 2/2 Running 0 4d15h
preference-v1-7fdb89c86b-gkk5g 2/2 Running 0 4d14h

Deploy microservice: recommendation

The deployment of the microservice recommendation is exactly like it is done for preference:

cd ~/istio-tutorial/recommendation/
oc apply -f kubernetes/Deployment.yml -n tutorial
oc apply -f kubernetes/Service.yml -n tutorial

Check if pods are running with two containers:

oc get pods -w

The result should look somehow like this:

NAME READY STATUS RESTARTS AGE
customer-6948b8b959-g77bs 2/2 Running 0 4d15h
preference-v1-7fdb89c86b-gkk5g 2/2 Running 0 4d14h
recommendation-v1-69db8d6c48-p9w2b 2/2 Running 0 4d14h

Optional: build the images

It is possible (and probably a good training) to build the microservices locally to understand how this works. In order to achieve this the packages maven and podman must be installed.

Build customer

Go to the source folder of customer application and build it:

cd ~/projects/istio-tutorial/customer/java/springboot
mvn package

It will take a few seconds, but it should give "BUILD SUCCESS" as output, if everything worked.

Now the image will be built using podman

podman build -t example/customer .

Build preference

The image build process of the second microservice follows the same flow as customer:

cd ~/istio-tutorial/preference/java/springboot
mvn package
podman build -t example/preference:v1 .

the "v1" tag at the image name is important and must be used.

Build recommendation

The image build process of the third microservice follows the same flow as preference:

cd ~/projects/istio-tutorial/recommendation/java/vertx
mvn package
podman build -t example/recommendation:v1 .

the "v1" tag at the image name is important and must be used. Later other versions will be deployed.

Installation

Sat, 28 Mar 2020 00:00:00 +0000

Everything has a start, this blog as well as the following tutorials. This series of tutorials shall provide a brief and working overview about OpenShift Service Mesh. It is starting with the installation and the first steps, and will continue with advanced settings and configuration options.

OpenShift 4.x and ServiceMesh

UPDATE: At 10th April 2020 Red Hat released Service Mesh version 1.1 which supports:

Istio - 1.4.6
Kiali - 1.12.7
Jaeger - 1.17.1

The following tutorials for OpenShift Service Mesh are based on the official documentation: OpenShift 4.3 Service Mesh and on the Interactive Learning Portal. All operations have been successfully tested on OpenShift 4.3. Currently OpenShift supports Istio 1.4.6, which shall be updated in one of the future releases.

To learn the basics of Service Mesh, please consult the documentation as they are not repeated here.

It is assumed that OpenShift has access to external registries, like quay.io.

Other resource I can recommend are:

Istio By Example: A very good and brief overview of different topics by Megan O’Keefe.
Istio: The Istio documentation

Prerequisites

At the very beginning OpenShift must be installed. This tutorial is based on OpenShift 4.3 and a Lab installation on Hetzner was used. Moreover, it is assumed that the OpenShift Client and Git are installed on the local system.

During the tutorials an example application in the namespace tutorial will be deployed. This application will contain 3 microservices:

customer (the entry point)
preference
recommendation

This application is also used at the Interactive Learning Portal which can be tested there interactively. However, the training is still based on OpenShift version 3.

Install Red Hat Service Mesh

To deploy Service Mesh on Openshift 4 follow the guide at Installing Red Hat OpenShift Service Mesh.

In short, multiple operators must be installed:

Elasticsearch
Jaeger
Kiali
Service Mesh

Elasticsearch is a very memory intensive application. Per default it will request 16GB of memory which can be reduced on Lab environments.

Link Jaeger and Grafana to Kiali

In the lab environment it happened that Kiali was not able to auto detect Grafana or Jaeger. This is visible when the link Distributed Tracing is missing in the left menu.

To fix this the ServiceMeshControlPlane object in the istio-system namespace must be updated with 3 lines:

oc edit ServiceMeshControlPlane -n istio-system
kiali: # ADD THE FOLLOWING LINES
dashboard:
grafanaURL: https://grafana-istio-system.apps.<your clustername>
jaegerURL: https://jaeger-istio-system.apps.<your clustername>

This change will take a few minutes to be effective.