Observability on TechBlog about OpenShift/Ansible/Satellite and much more

The Hitchhiker's Guide to Observability - Limit Read Access to Traces - Part 8

Sat, 06 Dec 2025 00:00:00 +0000

In the previous articles, we deployed a distributed tracing infrastructure with TempoStack and OpenTelemetry Collector. We also deployed a Grafana instance to visualize the traces. The configuration was done in a way that allows everybody to read the traces. Every system:authenticated user is able to read ALL traces. This is usually not what you want. You want to limit trace access to only the appropriate namespace.

In this article, we’ll limit the read access to traces. The users of the team-a namespace will only be able to see their own traces.

Prerequisites

Before we begin, make sure you have:

TempoStack deployed and configured (from Part 2)
Team-a namespace with traces flowing (from Part 4)
A separate user for the team-a namespace.

Verify Trace Access

Let’s verify what a user can see when they are authenticated. We have user1 who is a member of the team-a namespace. Let’s log in as this user and verify what they can see.

Navigate to Observability > Traces and select the tempostack/simplest datasource:

You should see the traces for the team-a namespace. This is fine—that’s what we want.

But now let’s change the tenant to tenantB and verify what the user can see.

As you can see, the user can see traces from the tenantB namespace, although they are a member of the team-a namespace. This is not what we want. We want to limit trace access to only the appropriate namespace.

The Original RBAC Configuration

In Part 2, we created a ClusterRoleBinding to grant read access to traces for everybody who is authenticated.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: tempostack-traces-reader
subjects:
- kind: Group
apiGroup: rbac.authorization.k8s.io
name: 'system:authenticated'
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: tempostack-traces-reader

This ClusterRoleBinding allows system:authenticated users to read all traces from all tenants and is bound to the ClusterRole tempostack-traces-reader.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tempostack-traces-reader
rules:
- verbs:
- get
apiGroups:
- tempo.grafana.com
resources:
- tenantA
- tenantB
resourceNames:
- traces

This is the configuration we want to change. We want to limit read access to traces to only the appropriate namespace.

Change the RBAC Configuration

We will change the RBAC configuration to limit read access to traces. To do so, we will first create a new ClusterRole for the team-a namespace and bind it to users of that namespace.

Create the new ClusterRole:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tempostack-traces-reader-team-a (1)
rules:
- verbs:
- get
apiGroups:
- tempo.grafana.com
resources:
- tenantA (2)
resourceNames:
- traces

1	Name of the ClusterRole, now with the prefix team-a.
2	The Tenant that is allowed to read the traces. This time it is only the tenantA tenant.

Create the new ClusterRoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: tempostack-traces-reader-team-a (1)
subjects:
- kind: User (2)
apiGroup: rbac.authorization.k8s.io
name: user1
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: tempostack-traces-reader-team-a (3)

1	Name of the ClusterRoleBinding, now with the prefix team-a.
2	The User that is allowed to read the traces. In this example: user1.
3	The ClusterRole that is allowed to read the traces.

In this example, we are using a single user. In a real-world scenario, you would most likely have a group of users. In that case, you would use a Group instead of a User.

Modify the Original ClusterRole:

As a final step, we need to modify the original ClusterRole to remove tenantB from the list of tenants that are allowed to read the traces.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tempostack-traces-reader
rules:
- verbs:
- get
apiGroups:
- tempo.grafana.com
resources:
- tenantB (1)
resourceNames:
- traces

1	Only tenantA remains. Remove tenantB from the list of resources.

This removes the permission to read traces from the tenantB namespace for the system:authenticated group.

Eventually, the original ClusterRoleBinding might be deleted once every user has been assigned to a separate ClusterRole.

Verify the Changes

Let’s see what these changes do. As user1, you should still be able to see the traces from the tenantA namespace.

But if you change the tenant to tenantB, you should see an error message like this:

The user will still see the list of tenants. Hopefully, this will be fixed in a future version of TempoStack.

The Hitchhiker's Guide to Observability - Here Comes Grafana - Part 7

Thu, 04 Dec 2025 00:00:00 +0000

While we have been using the integrated tracing UI in OpenShift, it is time to summon Grafana. Grafana is a visualization powerhouse that allows teams to build custom dashboards, correlate traces with logs and metrics, and gain deep insights into their applications. In this article, we’ll deploy a dedicated Grafana instance for team-a in their namespace, configure a Tempo datasource, and create a dashboard to explore distributed traces.

Prerequisites

Before we begin, make sure you have:

The Grafana Operator installed cluster-wide (we’ll cover this first)
TempoStack deployed and configured (from Part 2)
Team-a namespace with traces flowing (from Part 4)

The Grafana Operator

The Grafana Operator provides Custom Resource Definitions (CRDs) for managing Grafana instances, datasources, and dashboards declaratively.

Be sure that the Operator is installed. Typically, it is installed in the openshift-operators namespace. If you keep that namespace, all you need to do is create a Subscription. Otherwise you will also need to create an OperatorGroup.

If you select a different namespace, be sure to verify possible RBAC bindings, that you might need to set up.

Install Subscription

The Grafana Operator is available from the OperatorHub. Either use the UI to install it, or simply create a Subscription to install it:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: grafana-operator
namespace: openshift-operators (1)
spec:
channel: v5 (2)
installPlanApproval: Automatic
name: grafana-operator
source: community-operators (3)
sourceNamespace: openshift-marketplace

1	Installing in `openshift-operators` makes the operator available cluster-wide
2	Use the v5 channel for the operator. This is the only available channel currently.
3	The source is the OperatorHub catalog. Grafana is a community operator.

Verify the operator is running:

oc get pods -n openshift-operators -l app.kubernetes.io/name=grafana-operator

You should see output similar to:

NAME READY STATUS RESTARTS AGE
grafana-operator-7d8f9c6b5-xyz12 1/1 Running 0 2m

Create Grafana Instance for Team-A

Now let’s deploy a Grafana instance in the team-a namespace. This gives the team full control over their dashboards while isolating them from other teams.

We are logging in as the project admin user of the team-a namespace. So everything we do in this namespace will be done as this user.

Create the Grafana Admin Secret

First, we need to create a secret containing the Grafana admin credentials, since we do not want to store passwords in plain text in our manifests!

Never store passwords in plain text in your manifests and use a secrets management solution like Sealed Secrets or External Secrets Operator.

Create the following example. Replace the password with your own strong password.

apiVersion: v1
kind: Secret
metadata:
name: grafana-admin-credentials
namespace: team-a
type: Opaque
stringData:
GF_SECURITY_ADMIN_USER: admin (1)
GF_SECURITY_ADMIN_PASSWORD: <your-secure-password> (2)

1	The admin username for Grafana
2	Replace with a strong password

Deploy the Grafana Instance

The Grafana Operator provides a Custom Resource Definition (CRD) for deploying Grafana instances. We can use this to deploy a Grafana instance in the team-a namespace. This instance is labelled with "dashboards: "grafana-team-a"" to make it easier to target it with GrafanaDashboard resources.

apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
name: grafana
namespace: team-a
labels:
dashboards: "grafana-team-a" (1)
spec:
config:
log:
mode: "console"
auth:
disable_login_form: "false"
security:
admin_user: ${GF_SECURITY_ADMIN_USER} (2)
admin_password: ${GF_SECURITY_ADMIN_PASSWORD}
deployment:
spec:
replicas: 1
template:
spec:
containers:
- name: grafana
envFrom:
- secretRef:
name: grafana-admin-credentials (3)
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
route:
spec:
tls:
termination: edge (4)
disableDefaultAdminSecret: true (5)

1	Label used by GrafanaDashboard resources to target this instance. Required so that the datasource can find the instance.
2	References environment variables from the secret
3	Mounts the secret as environment variables
4	Creates an OpenShift Route with TLS edge termination
5	Disables the default admin secret created by the Grafana Operator. We will use our own secret and this setting prevents that the operator overwrites it.

After a few moments, the Grafana pod should be ready.

oc get pods -n team-a -l app=grafana -w

Get the Route URL:

oc get route grafana-route -n team-a -o jsonpath='{.spec.host}'

Use this route and the credentials from the secret to log into Grafana.

Configure Tempo Datasource

The datasource connects Grafana to TempoStack (or any other datasource like Loki, Prometheus, etc.), allowing you to query and visualize traces. We need to authenticate with a client certificate to TempoStack and be sure the send the tenant ID of tenantA in the header of the queries.

When you read the previous part of this series, you saw that we created a ClusterRole to grant read access to the traces. The Binding for this role allows read access to everybody who is authenticated against the system. Therefore, we do not need to take care of permissions… at this point.

TempoStack Client Certificate Authentication

To query the TempoStack instance, we need to authenticate with it. This is done by using a client certificate. Therefore, we need to create a secret with the client certificate and key. In addition, this secret will also contain the tenant ID, which must be sent to the TempoStack instance as a header.

First, get the client certificate and key from the TempoStack instance:

oc get secret -n tempostack | grep gateway-mtls

You should find something like this:

tempo-simplest-gateway-mtls kubernetes.io/tls 2 19d

Fetch the client certificate and key and store them in a file locally:

oc get secret -n tempostack tempo-simplest-gateway-mtls -o jsonpath='{.data.tls\.crt}' | base64 -d > tls.crt
oc get secret -n tempostack tempo-simplest-gateway-mtls -o jsonpath='{.data.tls\.key}' | base64 -d > tls.key

Now, let’s get the tenant ID from the TempoStack instance:

oc get tempostack/simplest -n tempostack -o jsonpath='{.spec.tenants.authentication}'

You should find something like this (tenantA is the one we are interested in):

[
{
"tenantId": "1610b0c3-c509-4592-a256-a1871353dbfc", (1)
"tenantName": "tenantA"
},
{
"tenantId": "1610b0c3-c509-4592-a256-a1871353dbfd",
"tenantName": "tenantB"
}
]

1	The tenant ID of tenantA

Now, let’s create the secret with the client certificate and key and the tenant ID:

oc create secret generic tempo-auth -n team-a --from-file=tls.crt=tls.crt --from-file=tls.key=tls.key --from-literal=tenantA="1610b0c3-c509-4592-a256-a1871353dbfc"

That’s everything we need to authenticate with TempoStack for tenantA.

As you prefer, you can create a separate secret for each tenantID. Just be sure to update the "valuesFrom" section of the GrafanaDatasource resource.

Create the Tempo Datasource

Now we can create the GrafanaDatasource that connects to TempoStack:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
name: tempo-tenanta-datasource
namespace: team-a
spec:
allowCrossNamespaceImport: false
datasource:
access: proxy
editable: true
isDefault: true
jsonData:
httpHeaderName1: X-Scope-OrgID (1)
timeInterval: 5s
tlsAuth: true
tlsAuthWithCACert: false
tlsSkipVerify: true
name: tempo-tenanta (2)
secureJsonData: (3)
httpHeaderValue1: '${tenantA}'
tlsClientCert: '${tls.crt}'
tlsClientKey: '${tls.key}'
type: tempo (4)
url: 'https://tempo-simplest-query-frontend.tempostack.svc.cluster.local:3200' (5)
instanceSelector:
matchLabels:
dashboards: grafana-team-a (6)
resyncPeriod: 10m0s
valuesFrom: (7)
- targetPath: secureJsonData.tlsClientCert
valueFrom:
secretKeyRef:
key: tls.crt
name: tempo-auth
- targetPath: secureJsonData.tlsClientKey
valueFrom:
secretKeyRef:
key: tls.key
name: tempo-auth
- targetPath: secureJsonData.httpHeaderValue1
valueFrom:
secretKeyRef:
key: tenantA
name: tempo-auth

1	The header name for the tenant ID.
2	The name of the Tempo datasource.
3	The reference to the client certificate, key and tenantID. They are coming from the secret we created earlier.
4	The type of the datasource. This type must be available in Grafana. For Tempo, the type is `tempo`.
5	The URL of the Tempo query frontend. This is the URL of the Tempo query frontend service in the TempoStack namespace.
6	The label of the Grafana instance to target. This is the label we added to the Grafana instance when we created it.
7	The reference to the secrets and the keys inside the secret.

Verify the Datasource

Log into Grafana and navigate to Connection > Data sources. You should see the Tempo datasource listed.

Click on the Tempo datasource to see the details:

As depicted in the image above, the datasource is configured with TLS Client Authentication and a HTTP Header.

Any changes in the Grafana UI will not be synced back to the GrafanaDatasource resource. You need to update the resource manually.

To verify the datasource, we can create a test query. Navigate to Explore.

Enter the query {}, which will simply get all traces from the TempoStack instance from (by default) the last minute.

If you have multiple data sources configured already, be sure to select the correct one in the dropdown menu.

You should see the traces in the Grafana Explore UI.

You can select any trace and see the details in the Grafana Explore UI.

Create a Traces Dashboard

Now let’s create a dashboard to visualize and explore traces. The Grafana Operator allows us to define dashboards as Kubernetes resources.

Create the Dashboard ConfigMap

To have the first showable data, we need to create a dashboard. The Grafana Operator provides a Custom Resource Definition called GrafanaDashboard for deploying such dashboards.

Now, dashboards are their own beast. There are so many things to configure and set that probably a whole separate blog series is needed to cover all of it. I am by far not an expert in this field and I am happy that I got this one up and running, so I will just create a simple dashboard with a few panels.

Honestly, I am not sure if it is a good way to create dashboards using the CRD. It is probably better to use the Grafana UI and then export the JSON data if you want to keep it declarative.

This dashboard provides:

A trace search panel
Latency histogram -→ I have set this to "higher than 3ms" to at least see something.
Recent traces table

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
name: grafana-team-a-traces-dashboard
namespace: team-a
spec:
instanceSelector:
matchLabels:
dashboards: "grafana-team-a" (1)
json: | (2)
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "grafana",
"uid": "-- Grafana --"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"id": 1,
"links": [],
"panels": [
{
"fieldConfig": {
"defaults": {},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 24,
"x": 0,
"y": 0
},
"id": 1,
"options": {
"code": {
"language": "plaintext",
"showLineNumbers": false,
"showMiniMap": false
},
"content": "# Team-A Distributed Traces\n\nThis dashboard allows you to explore distributed traces from your applications.\n\n",
"mode": "markdown"
},
"pluginVersion": "12.1.0",
"title": "Welcome",
"type": "text"
},
{
"datasource": {
"type": "tempo",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"custom": {
"align": "auto",
"cellOptions": {
"type": "auto"
},
"inspect": false
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": 0
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 4
},
"id": 2,
"options": {
"cellHeight": "sm",
"footer": {
"countRows": false,
"fields": "",
"reducer": [
"sum"
],
"show": false
},
"showHeader": true
},
"pluginVersion": "12.1.0",
"targets": [
{
"datasource": {
"type": "tempo",
"uid": "${datasource}"
},
"filters": [
{
"id": "81ea3f00",
"operator": "=",
"scope": "span"
}
],
"limit": 20,
"metricsQueryType": "range",
"queryType": "traceqlSearch",
"refId": "A",
"tableType": "traces"
}
],
"title": "Trace Search",
"type": "table"
},
{
"datasource": {
"type": "tempo",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "bars",
"fillOpacity": 100,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": 0
},
{
"color": "red",
"value": 80
}
]
},
"unit": "ms"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 4
},
"id": 3,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "12.1.0",
"targets": [
{
"datasource": {
"type": "tempo",
"uid": "${datasource}"
},
"filters": [
{
"id": "6ff20d0d",
"operator": "=",
"scope": "span"
},
{
"id": "min-duration",
"operator": ">",
"tag": "duration",
"value": "0.3ms",
"valueType": "duration"
}
],
"limit": 20,
"metricsQueryType": "range",
"queryType": "traceqlSearch",
"refId": "A",
"tableType": "spans"
}
],
"title": "Span Duration Distribution",
"type": "timeseries"
},
{
"datasource": {
"type": "tempo",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"custom": {
"align": "auto",
"cellOptions": {
"type": "auto"
},
"inspect": false
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": 0
}
]
}
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "traceID"
},
"properties": [
{
"id": "links",
"value": [
{
"title": "View Trace",
"url": "/explore?orgId=1&left=%7B%22datasource%22:%22${datasource}%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22datasource%22:%7B%22type%22:%22tempo%22,%22uid%22:%22${datasource}%22%7D,%22queryType%22:%22traceql%22,%22limit%22:20,%22query%22:%22${__value.raw}%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D"
}
]
}
]
},
{
"matcher": {
"id": "byName",
"options": "duration"
},
"properties": [
{
"id": "unit",
"value": "ns"
}
]
}
]
},
"gridPos": {
"h": 10,
"w": 24,
"x": 0,
"y": 12
},
"id": 4,
"options": {
"cellHeight": "sm",
"footer": {
"countRows": false,
"fields": "",
"reducer": [
"sum"
],
"show": false
},
"showHeader": true,
"sortBy": [
{
"desc": true,
"displayName": "startTime"
}
]
},
"pluginVersion": "12.1.0",
"targets": [
{
"datasource": {
"type": "tempo",
"uid": "${datasource}"
},
"limit": 50,
"queryType": "traceqlSearch",
"refId": "A",
"tableType": "traces"
}
],
"title": "Recent Traces",
"type": "table"
}
],
"preload": false,
"refresh": "30s",
"schemaVersion": 41,
"tags": [
"tracing",
"tempo",
"team-a"
],
"templating": {
"list": [
{
"current": {
"text": "tempo-tenanta",
"value": "90b71ee3-693a-4c41-8cdf-624a3bb78e7a"
},
"includeAll": false,
"label": "Datasource",
"name": "datasource",
"options": [],
"query": "tempo",
"refresh": 1,
"regex": "",
"type": "datasource"
}
]
},
"time": {
"from": "now-1h",
"to": "now"
},
"timepicker": {},
"timezone": "",
"title": "Team-A Distributed Traces",
"uid": "team-a-traces",
"version": 3
}

1	The label of the Grafana instance to target. This is the label we added to the Grafana instance when we created it.
2	The JSON data of the dashboard (yes, it’s a monster!).

Now you will see a new dashboard in the Grafana UI. Navigate to Dashboards and find Team-A Distributed Traces.

Next Steps

Congratulations! The Cat has summoned Grafana .

You now have a functional Grafana instance with Tempo integration for team-a. Here are some ideas for extending this setup:

Add Loki datasource: Correlate traces with logs using derived fields
Add Prometheus datasource: Link metrics to traces for full observability
Create alerting rules: Set up alerts for high latency or error rates
Build more dashboards: Create service-specific dashboards for different applications

As mentioned above, Grafana and its dashboards are powerful tools for visualizing and exploring traces. There are tons of things to configure and set up. Maybe I will cover some of these things in a future article.

Happy tracing! 🚀

The Hitchhiker's Guide to Observability Introduction - Part 1

Sun, 23 Nov 2025 00:00:00 +0000

With this article I would like to summarize and, especially, remember my setup. This is Part 1 of a series of articles that I split up so it is easier to read and understand and not too long. Initially, there will be 6 parts, but I will add more as needed.

Introduction

In modern microservices architectures, understanding how requests flow through your distributed system is crucial for debugging, performance optimization, and maintaining system health. Distributed tracing provides visibility into these complex interactions by tracking requests as they traverse multiple services.

This whole guide demonstrates how to set up a distributed tracing infrastructure using

OpenShift (4.16+) as base platform
Red Hat Build of OpenTelemetry - The observability framework based on OpenTelemetry
TempoStack - Grafana’s distributed tracing backend for Kubernetes
Multi-tenant architecture - Isolating traces by team or environment
Cluster Observability Operator - For now this Operator is only used to extend the OpenShift UI with the tracing UI.

Thanks to

This article would not have been possible without the help of Michaela Lang. Check out her articles on LinkedIn mainly discussing Tracing and Service Mesh.

What is OpenTelemetry?

OpenTelemetry is an observability framework and toolkit which aims to provide unified, standardized, and vendor-neutral telemetry data collection for traces, metrics and logs for cloud-native software.

In this article we will focus on traces only.

When it comes to Red Hat and OpenShift, the supported installation is based on the Operator Red Hat Build of OpenTelemetry, which is based on the open source OpenTelemetry project and adds supportability.

Core Features:

Source: OpenShift OTEL Product Overview

The OpenTelemetry Collector can receive, process, and forward telemetry data in multiple formats, making it the ideal component for telemetry processing and interoperability between telemetry systems. The Collector provides a unified solution for collecting and processing metrics, traces, and logs.

The core features of the OpenTelemetry Collector include:

Data Collection and Processing Hub It acts as a central component that gathers telemetry data like metrics and traces from various sources. This data can be created from instrumented applications and infrastructure.
Customizable telemetry data pipeline The OpenTelemetry Collector is customizable and supports various processors, exporters, and receivers.
Auto-instrumentation features Automatic instrumentation simplifies the process of adding observability to applications. If used, developers do not need to manually instrument their code for basic telemetry data. (This depends a bit on the used coding language and framework, maybe this is worth a separate article)

Here are some of the use cases for the OpenTelemetry Collector:

Centralized data collection In a microservices architecture, the Collector can be deployed to aggregate data from multiple services.
Data enrichment and processing Before forwarding data to analysis tools, the Collector can enrich, filter, and process this data.
Multi-backend receiving and exporting The Collector can receive and send data to multiple monitoring and analysis platforms simultaneously. You can use Red Hat build of OpenTelemetry in combination with Red Hat OpenShift Distributed Tracing Platform.

What is Grafana Tempo?

Grafana Tempo is an open-source, easy-to-use, and high-scale distributed tracing backend. Tempo lets you search for traces, generate metrics from spans, and link your tracing data with logs and metrics. It is deeply integrated with Grafana, Prometheus and Loki and can ingest traces from various sources, such as OpenTelemetry, Jaeger, Zipkin and more.

Core Features:

Source: Grafana Tempo

Built for massive scale The only dependency is object storage which provides affordable long-term storage of traces.
Cost-effective Not indexing the traces makes it possible to store orders of magnitude more trace data for the same cost.
Strong integration with open source tools Compatible with open source tracing protocols.

In addition, it is deeply integrated with Grafana, allowing you to visualize the traces in a Grafana dashboard and link logs, metrics and traces together.

Use Case for this Article

The use case that was tested in this article was the following:

Several applications (team-a, team-b, …) are hosted on separate OpenShift namespaces. On each application namespace, a local OpenTelemetry Collector (OTC) is configured to collect the traces from the application. These local OpenTelemetry Collectors will export the traces to a central OpenTelemetry Collector (hosted in the namespace tempostack). The central Collector will then export the data to a TempoStack instance (also hosted in the namespace tempostack), which will store the traces in object storage. The storage itself is provided by S3-compatible storage, in this example OpenShift Data Foundation.

For a more detailed view see the next section.

Architecture Overview

As described the implementation follows a two-tier collector architecture with multi-tenancy support:

---
title: "Architecture Overview"
config:
theme: 'dark'
---
graph TB
subgraph app["Application"]
mockbin1["Mockbin #1<br/>(team-a namespace)<br/>(tenantA)"]
mockbin2["Mockbin #2<br/>(team-b namespace)<br/>(tenantB)"]
end
subgraph local["Local OTC"]
otc_a["OTC-team-a<br/>• Add namespace<br/>• Batch processing<br/>• Forward to central"]
otc_b["OTC-team-b<br/>• Add namespace<br/>• Batch processing<br/>• Forward to central"]
end
subgraph central["Central OTC (tempostack namespace)"]
otc_central["OTC-central<br/>• Receive from local collectors<br/>• Add K8s metadata (k8sattributes)<br/>• Route by namespace (routing connector)<br/>• Authenticate with bearer token<br/>• Forward to TempoStack with tenant ID"]
end
subgraph tempo["TempoStack (tempostack namespace)"]
tempostack["Multi-tenant Trace Storage<br/>• tenantA, tenantB, ...<br/>• S3 backend storage<br/>• 48-hour retention"]
end
mockbin1 -->|"OTLP"| otc_a
mockbin2 -->|"OTLP"| otc_b
otc_a -->|"OTLP(with namespace)"| otc_central
otc_b -->|"OTLP(with namespace)"| otc_central
otc_central -->|"OTLP<br/>(X-Scope-OrgID header)"| tempostack
classDef appStyle fill:#2f652a,stroke:#2f652a,stroke-width:2px
classDef localStyle fill:#425cc6,stroke:#425cc6,stroke-width:2px;
classDef centralStyle fill:#425cc6,stroke:#425cc6,stroke-width:2px;
classDef tempoStyle fill:#906403,stroke:#906403,stroke-width:2px
class mockbin1,mockbin2 appStyle
class otc_a,otc_b localStyle
class otc_central centralStyle
class tempostack tempoStyle

As quick summary Traces from the application Mockbin #1 are collected by the "OTC-team-a" and forwarded to the "Central OTC". From there the traces are further forwarded to Tempo.

Why 2-Tier Architecture?

You may ask yourself why there are two OpenTelemetry Collectors and if the application could not send directly to the Central OTC or if the Local OTC could not write directly into the Tempo storage. Both options are fine and will work, however, I tried to make it more secure. Only one OTC is allowed to perform write actions and applications can only send to the Local OTC, which forwards to the Central OTC where the traces are routed based on the source namespace. This way, nobody can interfere with other namespaces.

Therefore:

Separation of Concerns: Application namespaces handle local processing; central namespace handles routing and storage. The Central decides where and how to store. Application owners cannot overwrite this.
Resource Efficiency: Lightweight collectors in app namespaces, heavy processing centralized
Security: Applications don’t need direct access to TempoStack
Scalability: Each tier can scale independently
Multi-tenancy: Central collector routes traces to appropriate tenants

What now?

The next articles will cover the actual implementation. We will first deploy Tempo and the Central Collector. Then we will deploy example applications and the Local Collector. If everything works as planned, we will be able to see traces on the OpenShift UI.

The Hitchhiker's Guide to Observability - Grafana Tempo - Part 2

Mon, 24 Nov 2025 00:00:00 +0000

After covering the fundamentals and architecture in Part 1, it’s time to get our hands dirty! This article walks through the complete implementation of a distributed tracing infrastructure on OpenShift.

We’ll deploy and configure the Tempo Operator and a multi-tenant TempoStack instance. For S3 storage we will use the integrated OpenShift Data Foundation. However, you can use whatever S3-compatible storage you have available.

Grafana Tempo - Step-by-Step Implementation

Prerequisites

Before starting, ensure you have the following Systems or Operators installed (used Operator versions in this article):

OpenShift or Kubernetes cluster (OpenShift v4.20)
Red Hat build of OpenTelemetry installed (v0.135.0-1)
Tempo Operator installed (v0.18.0-2)
S3-compatible storage (for TempoStack, based on OpenShift Data Foundation)
Cluster Observability Operator (v1.3.0) - for now this Operator is only used to extend the OpenShift UI with the tracing UI.

For all configurations I also created a proper GitOps implementation (of course :)). However, first I would like to show the actual configuration. The GitOps implementation can be found at the section GitOps Deployment.

Step 1: Verify/Deploy Tempo Operator

Let’s first verify if the Tempo Operator is installed and ready to use. If everything is fine, then the Operator will be deployed in the namespace openshift-tempo-operator:

Step 2: Deploy TempoStack Resource

TempoStack is the central trace storage backend. We are using the Tempo Operator here, which provides the TempoStack resource and multi-tenancy capability. During my tests I deployed the TempoStack resource and also created a Helm Chart that is able to render this resource. However, the Operator itself also provides a TempoMonolith resource. This will put everything into one Pod, while the TempoStack rolls out the stack in separate containers (like ingester, gateway, etc.).

My Helm Chart currently does not support the TempoMonolith resource yet. If you require this, please ping me and I will try to add it.

Be sure that the S3 bucket is available and a Secret (here called tempo-s3) with the following keys exists (valid for OpenShift Data Foundation): access_key_id, access_key_secret, bucket, endpoint. The layout of the Secret will look slightly different depending on the S3 storage backend you are using.

Create the TempoStack instance:

The following TempoStack resource has been used

apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
name: simplest (1)
namespace: tempostack (2)
spec:
managementState: Managed
replicationFactor: 1 (3)
# Resource limits
resources: (4)
total:
limits:
cpu: "2"
memory: 2Gi
# Trace retention
retention: (5)
global:
traces: 48h0m0s
# S3 storage configuration
storage:
secret:
credentialMode: static (6)
name: tempo-s3 (7)
type: s3
tls:
enabled: false
storageSize: 500Gi
# Multi-tenancy configuration
tenants:
mode: openshift
authentication: (8)
- tenantId: 1610b0c3-c509-4592-a256-a1871353dbfa
tenantName: tenantA
- tenantId: 1610b0c3-c509-4592-a256-a1871353dbfb
tenantName: tenantB
- tenantId: 1610b0c3-c509-4592-a256-a1871353dbfc
tenantName: tenantC
# Gateway and UI
template: (9)
gateway:
enabled: true
component:
replicas: 1
ingress:
type: route
route:
termination: reencrypt
queryFrontend:
component:
replicas: 1
jaegerQuery:
enabled: true
servicesQueryDuration: 72h0m0s

1	Name of the TempoStack instance
2	Namespace of the TempoStack instance
3	Integer value for the number of ingesters that must acknowledge the data from the distributors before accepting a span.
4	Defines resources for the TempoStack instance. Default is (limit only) 2 CPU and 2Gi memory.
5	Configuration options for retention of traces. The default value is 48h.
6	Credential mode for the S3 storage. Depends how the storage will be integrated. Default is static.
7	Secret name for the S3 storage.
8	Configuration options for the tenants. In this example: tenantA, tenantB, tenantC. Consists of tenantName and tenantId, both can be defined by the user.
9	Configuration options for the different Tempo components. In this example: gateway, query-frontend. Other components could be: distributor, ingester, compactor or querier.

Key Configuration Points:

Multi-tenancy: Supports 3 tenants currently (tenantA, tenantB, tenantC). This part must be modified when a new tenant enters the realm.
Retention: Traces stored for 48 hours.
Storage: Uses S3-compatible backend (requires separate secret).
Gateway: Exposes OTLP endpoint with TLS.

Step 3: Configure RBAC for TempoStack Trace Access read/write

Set up ClusterRoles to control who can read and write traces.

The ClusterRoles must be updated whenever a new tenant is configured in TempoStack. The name of the tenant must be added in the resources array.

Traces Reader Role:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tempostack-traces-reader (1)
rules:
- verbs: (2)
- get
apiGroups:
- tempo.grafana.com
resources:
- tenantA (3)
- tenantB
- tenantC
resourceNames:
- traces

1	Name of the ClusterRole
2	Verbs for the ClusterRole. apiGroups: tempo.grafana.com resources: List of Tenants resourceNames: traces verbs: get
3	List of tenants that are allowed to read the traces.

Bind Reader Role to Authenticated Users:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: tempostack-traces-reader
subjects:
- kind: Group
apiGroup: rbac.authorization.k8s.io
name: 'system:authenticated' (1)
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: tempostack-traces-reader

1	The Group that is allowed to read the traces. In this example: system:authenticated. This means ALL authenticated users will be able to read all the traces (see warning below).

With this ClusterRoleBinding, anybody who is authenticated (system:authenticated) will be able to see the traces for the defined tenants (A, B, C). This is for an easy showcase in this article. For production environments, you should implement more granular RBAC controls per tenant.

Traces Writer Role:

This ClusterRole is used to write traces into TempoStack. Typically you will use this ClusterRole for the OpenTelemetry Collector, so it can write into TempoStack.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tempostack-traces-write (1)
rules:
- verbs:
- create (2)
apiGroups:
- tempo.grafana.com
resources: (3)
- tenantB
- tenantA
- tenantC
resourceNames:
- traces

1	Name of the ClusterRoleBinding
2	This time the verb is create. This means the user will be able to write new traces into TempoStack.
3	List of tenants.

Bind Writer Role to Central Collector:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: tempostack-traces
subjects:
- kind: ServiceAccount
name: otel-collector (1)
namespace: tempostack (2)
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: tempostack-traces-write

1	The ServiceAccount that is allowed to write the traces. In this example: otel-collector. We will create this Service Account in the next article.
2	The namespace of the ServiceAccount. In this example: tempostack.

GitOps Deployment

While the above is good for quick tests, it always makes sense to have a proper GitOps deployment. I have created a Chart and GitOps configuration that will:

Deploy the Tempo Operator
Configure the TempoStack instance
Configure the RBAC configurations for the TempoStack instance

The following sources will be used:

Helm Repository - to fetch the Helm Chart for the Tempo Operator including required Sub-Charts.
Setup Tempo Operator - To deploy and configure the Tempo Operator, configure object storage, magically create a Secret with the required keys, etc.

Feel free to clone or use whatever you need.

The following Sub-Charts are used:

helper-objectstore (version ~1.0.0) - Creates S3 Bucket
helper-odf-bucket-secret (version ~1.0.0) - Creates the Secret usable by the TempoStack instance
helper-operator (version ~1.0.18) - Installs the Tempo Operator
helper-status-checker (version ~4.0.0) - Verifies the status of the Tempo Operator
tempo-tracing (version ~1.0.0) - Installs the TempoStack instance
tpl (version ~1.0.0) - Template Library

The this Argo CD Application we can deploy the Tempo Operator:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: setup-tempo-operator
namespace: openshift-gitops
spec:
destination:
name: in-cluster (1)
namespace: openshift-tempo-operator (2)
info:
- name: Description
value: ApplicationSet that Deploys on Management Cluster Configuration (using Git Generator)
project: in-cluster (3)
source:
path: clusters/management-cluster/setup-tempo-operator (4)
repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops' (5)
targetRevision: main
syncPolicy:
retry:
backoff:
duration: 5s
factor: 2
maxDuration: 3m
limit: 5

1	Target cluster, here the local cluster
2	Namespace of the target cluster, here the Operator will be installed.
3	Project of the target cluster
4	Path to the Git repository
5	URL to the Git repository

Since many things are moving in the background, it will take a while until the Argo CD Application is synced (i.e. Creation of S3 Bucket)

This will create the Argo CD Application that can be synchronized with the cluster:

As seen above, many resources are created using a single Helm Chart (ok, with some Sub-Charts). The actual configuration is done in the values.yaml file. The full values file is quite long, so I will break it down to the different Sub-Charts and add the complete file at the end that latest file I am using for testing can be found at values.yaml.

Values File Snippets for GitOps

TempStack Settings

The following will configure the TempoStack resource. Verify the upstream Helm Chart tempo-tracing for detailed and additional information about the settings.

tempo-tracing:
tempostack: (1)
enabled: true
name: simplest
managementState: Managed
namespace: (2)
create: true
name: tempostack
descr: "Namespace for the TempoStack"
display: "TempoStack"
additionalAnnotations: {}
additionalLabels: {}
storage: (3)
secret:
name: tempo-s3
type: s3
credentialMode: static
storageSize: 500Gi
replicationFactor: 1
serviceAccount: tempo-simplest (4)
tenants: (5)
mode: openshift
enabled: true
authentication: (6)
- tenantName: 'tenantA'
tenantId: '1610b0c3-c509-4592-a256-a1871353dbfc'
permissions: (7)
- write
- read
- tenantName: 'tenantB'
tenantId: '1610b0c3-c509-4592-a256-a1871353dbfd'
permissions:
- write
- read
observability:
enabled: true
tracing:
jaeger_agent_endpoint: 'localhost:6831'
otlp_http_endpoint: 'http://localhost:4320'
template:
gateway:
enabled: true
rbac: false
ingress:
type: 'route'
termination: 'reencrypt'
component:
replicas: 1
queryFrontend:
jaegerQuery:
enabled: true
component:
replicas: 1

1	Basic settings, like name of the instance
2	Namespace for the TempoStack instance.
3	Storage configuration for the TempoStack instance. Here we use type s3 and the Secret called "tempo-s3" which will be generated.
4	ServiceAccount for the TempoStack instance.
5	Tenant configuration for the TempoStack instance.
6	List of tenants. This list must be extended when new tenants shall be configured.
7	RBAC permissions for this tenant. This will add the tenant as resource to the READ and/or WRITE ClusterRole

Tempo Operator Deployment

The following settings will deploy the Operator and verify the status of the Operator installation. Only when the Operator has been installed successfully will Argo CD continue with the synchronization.

helper-operator:
operators:
tempo-operator: (1)
enabled: true
namespace: (2)
name: openshift-tempo-operator
create: true
subscription: (3)
channel: stable
approval: Automatic
operatorName: tempo-product
source: redhat-operators
sourceNamespace: openshift-marketplace
operatorgroup:
create: true
notownnamespace: true
########################################
# SUBCHART: helper-status-checker
# Verify the status of a given operator.
########################################
helper-status-checker:
enabled: true
approver: false (4)
checks:
- operatorName: tempo-operator (5)
namespace:
name: openshift-tempo-operator
syncwave: 1
serviceAccount:
name: "status-checker-tempo"

1	Install the Operator
2	Namespace settings of the Operator
3	Settings of the Operator itself, like name, channel and approval strategy.
4	Approver settings for the status checker. Here disabled, because we are using automatic approval.
5	Operator name to be verified.

S3 Bucket Deployment

The following Sub-Chart can be used to automatically create a Bucket in OpenShift Data Foundation.

########################################
# SUBCHART: helper-objectstore
# A helper chart that simply creates another backingstore for logging.
# This is a chart in a very early state, and not everything can be customized for now.
# It will create the objects:
# - BackingStore
# - BackingClass
# - StorageClass
# NOTE: Currently only PV type is supported
########################################
helper-objectstore:
# -- Enable objectstore configuration
# @default -- false
enabled: true
# -- Syncwave for Argo CD
# @default - 1
syncwave: 1
# -- Name of the BackingStore
backingstore_name: tempo-backingstore
# -- Size of the BackingStore that each volume shall have.
backingstore_size: 400Gi (1)
# -- CPU Limit for the Noobaa Pod
# @default -- 500m
limits_cpu: 500m
# -- Memory Limit for the Noobaa Pod.
# @default -- 2Gi
limits_memory: 2Gi
pvPool:
# -- Number of volumes that shall be used
# @default -- 1
numOfVolumes: 1
# Type of BackingStore. Currently pv-pool is the only one supported by this Helm Chart.
# @default -- pv-pool
type: pv-pool
# -- The StorageClass the BackingStore is based on
baseStorageClass: gp3-csi (2)
# -- Name of the StorageClass that shall be created for the bucket.
storageclass_name: tempo-bucket-storage-class (3)
# Bucket that shall be created
bucket:
# -- Shall a new bucket be enabled?
# @default -- false
enabled: true
# -- Name of the bucket that shall be created
name: tempo-bucket (4)
# -- Target Namespace for that bucket.
namespace: tempostack (5)
# -- Syncwave for bucketclaim creation. This should be done very early, but it depends on ODF.
# @default -- 2
syncwave: 2
# -- Name of the storageclass for our bucket
# @default -- openshift-storage.noobaa.io
storageclass: tempo-bucket-storage-class (6)

1	Size of the bucket.
2	StorageClass the bucket is based on.
3	Name of the StorageClass that shall be created for the bucket.
4	Name of the bucket that shall be created.
5	Namespace for the bucket.
6	StorageClass for the bucket.

Automatic Secret Creation for TempoStack

TempoStack is expecting a Secret with the following keys:

access_key_id
access_key_secret
bucket
endpoint
region (only for specific settings)

OpenShift Data Foundation creates a secret with access_key_id and access_key_secret and a ConfigMap with endpoint, region and the name of the bucket. Unfortunately, this does not work for TempoStack. Therefore, we are using a helper-odf-bucket-secret chart that will create a new Secret with the required keys. This chart creates a Job that reads the required information and creates a new Secret with the required keys.

##############################################
# SUBCHART: helper-odf-bucket-secret
# Creates a Secret that Tempo requires
#
# A Kubernetes Job is created, that reads the
# data from the Secret and ConfigMap and
# creates a new secret for Tempo.
##############################################
helper-odf-bucket-secret:
# -- Enable Job to create a Secret for TempoStack.
# @default -- false
enabled: true
# -- Syncwave for Argo CD.
# @default -- 3
syncwave: 3
# -- Namespace where TempoStack is deployed and where the Secret shall be created.
namespace: tempostack
# -- Name of Secret that shall be created.
secretname: tempo-s3 (1)
# Bucket Configuration
bucket:
# -- Name of the Bucket shall has been created.
name: tempo-bucket (2)
# -- Keys that shall be used to create the Secret.
keys: (3)
# -- Overwrite access_key_id key.
# @default -- access_key_id
access_key_id: access_key_id
# -- Overwrite access_key_secret key.
# @default -- access_key_secret
access_key_secret: access_key_secret
# -- Overwrite bucket key.
# @default -- bucket
bucket: bucket
# -- Overwrite endpoint key.
# @default -- endpoint
endpoint: endpoint
# -- Overwrite region key. Region is only set if set_region is true.
# @default -- region
region: region
# -- Set region key.
# @default -- false
set_region: false (4)

1	Name of the Secret that shall be created.
2	Name of the bucket that shall be used.
3	Keys that shall be used to create the Secret.
4	Set region key. Here disabled, because we are not using a specific region.

Using OpenShift Data Foundation with TempoStack requires you to NOT set a region. Therefore, it is disabled above.

Complete values file

To see the whole file expand the code:

Expand me...

tempo: &channel-tempo stable
tempo-namespace: &tempo-namespace openshift-tempo-operator
bucketname: &bucketname tempo-bucket
tempo-secret: &tempo-secret-name tempo-s3
storageclassname: &storageclassname tempo-bucket-storage-class
tempostack-namespace: &tempostack-namespace tempostack
tempo-tracing:
tempostack:
enabled: true
name: simplest
managementState: Managed
namespace:
create: true
name: *tempostack-namespace
descr: "Namespace for the TempoStack"
display: "TempoStack"
additionalAnnotations: {}
additionalLabels: {}
storage:
secret:
name: tempo-s3
type: s3
credentialMode: static
storageSize: 500Gi
replicationFactor: 1
serviceAccount: tempo-simplest
tenants:
mode: openshift
enabled: true
authentication:
- tenantName: 'tenantA'
tenantId: '1610b0c3-c509-4592-a256-a1871353dbfc'
permissions:
- write
- read
- tenantName: 'tenantB'
tenantId: '1610b0c3-c509-4592-a256-a1871353dbfd'
permissions:
- write
- read
observability:
enabled: true
tracing:
jaeger_agent_endpoint: 'localhost:6831'
otlp_http_endpoint: 'http://localhost:4320'
template:
gateway:
enabled: true
rbac: false
ingress:
type: 'route'
termination: 'reencrypt'
component:
replicas: 1
queryFrontend:
jaegerQuery:
enabled: true
component:
replicas: 1
######################################
# SUBCHART: helper-operator
# Operators that shall be installed.
######################################
helper-operator:
operators:
tempo-operator:
enabled: true
namespace:
name: *tempo-namespace
create: true
subscription:
channel: *channel-tempo
approval: Automatic
operatorName: tempo-product
source: redhat-operators
sourceNamespace: openshift-marketplace
operatorgroup:
create: true
notownnamespace: true
########################################
# SUBCHART: helper-status-checker
# Verify the status of a given operator.
########################################
helper-status-checker:
enabled: true
approver: false
checks:
- operatorName: tempo-operator
namespace:
name: *tempo-namespace
syncwave: 1
serviceAccount:
name: "status-checker-tempo"
########################################
# SUBCHART: helper-objectstore
# A helper chart that simply creates another backingstore for logging.
# This is a chart in a very early state, and not everything can be customized for now.
# It will create the objects:
# - BackingStore
# - BackingClass
# - StorageClass
# NOTE: Currently only PV type is supported
########################################
helper-objectstore:
# -- Enable objectstore configuration
# @default -- false
enabled: true
# -- Syncwave for Argo CD
# @default - 1
syncwave: 1
# -- Name of the BackingStore
backingstore_name: tempo-backingstore
# -- Size of the BackingStore that each volume shall have.
backingstore_size: 400Gi
# -- CPU Limit for the Noobaa Pod
# @default -- 500m
limits_cpu: 500m
# -- Memory Limit for the Noobaa Pod.
# @default -- 2Gi
limits_memory: 2Gi
pvPool:
# -- Number of volumes that shall be used
# @default -- 1
numOfVolumes: 1
# Type of BackingStore. Currently pv-pool is the only one supported by this Helm Chart.
# @default -- pv-pool
type: pv-pool
# -- The StorageClass the BackingStore is based on
baseStorageClass: gp3-csi
# -- Name of the StorageClass that shall be created for the bucket.
storageclass_name: *storageclassname
# Bucket that shall be created
bucket:
# -- Shall a new bucket be enabled?
# @default -- false
enabled: true
# -- Name of the bucket that shall be created
name: *bucketname
# -- Target Namespace for that bucket.
namespace: *tempo-namespace
# -- Syncwave for bucketclaim creation. This should be done very early, but it depends on ODF.
# @default -- 2
syncwave: 2
# -- Name of the storageclass for our bucket
# @default -- openshift-storage.noobaa.io
storageclass: *storageclassname
##############################################
# SUBCHART: helper-odf-bucket-secret
# Creates a Secret that Tempo requires
#
# A Kubernetes Job is created, that reads the
# data from the Secret and ConfigMap and
# creates a new secret for Tempo.
##############################################
helper-odf-bucket-secret:
# -- Enable Job to create a Secret for TempoStack.
# @default -- false
enabled: true
# -- Syncwave for Argo CD.
# @default -- 3
syncwave: 3
# -- Namespace where TempoStack is deployed and where the Secret shall be created.
namespace: *tempostack-namespace
# -- Name of Secret that shall be created.
secretname: *tempo-secret-name
# Bucket Configuration
bucket:
# -- Name of the Bucket shall has been created.
name: *bucketname
# -- Keys that shall be used to create the Secret.
keys:
# -- Overwrite access_key_id key.
# @default -- access_key_id
access_key_id: access_key_id
# -- Overwrite access_key_secret key.
# @default -- access_key_secret
access_key_secret: access_key_secret
# -- Overwrite bucket key.
# @default -- bucket
bucket: bucket
# -- Overwrite endpoint key.
# @default -- endpoint
endpoint: endpoint
# -- Overwrite region key. Region is only set if set_region is true.
# @default -- region
region: region
# -- Set region key.
# @default -- false
set_region: false

The TempoStack

Let’s imagine all of the above works and we have a TempoStack instance running in the namespace tempostack with the name simplest. Several Pods are running, like the distributor, ingester, gateway, query-frontend, compactor and querier. I admit it is not highly available, but this can be easily changed in the values file above (replica count).

oc get pods -n tempostack | grep simplest
tempo-simplest-compactor-584689c78f-t7pxb 1/1 Running 0 3d2h
tempo-simplest-distributor-6fb5d7dc9d-wrzt4 1/1 Running 0 3d2h
tempo-simplest-gateway-bbcb774b9-p44lq 2/2 Running 0 11h
tempo-simplest-ingester-0 1/1 Running 0 3d2h
tempo-simplest-querier-6cf9d7b6d8-mvc9d 1/1 Running 0 3d2h
tempo-simplest-query-frontend-7d859f9f9f-xzj97 3/3 Running 0 3d2h

Now we can continue with the OpenTelemetry Collector deployment. But before we do this, let’s first discuss how to:

Add Tracing UI to OpenShift

Extend the OpenShift UI

While we have Tempo installed, it will not be visible in the OpenShift UI by default. Here a separate Operator has been created by Red Hat, that will take care of this extension: Cluster Observability Operator.

This Operator be installed:

The Operator can be deployed with the Chart helper-operator as well. However, I did not merge this together with the TempoStack deployment, because this Operator also serves different purposes.

We only require a very small bit of this Operator. Amongst other resources, it also provides a resource called UIPlugin.

The resource must be configured as follows:

apiVersion: observability.openshift.io/v1alpha1
kind: UIPlugincd
metadata:
name: distributed-tracing (1)
spec:
type: DistributedTracing (2)

1	The name MUST be distributed-tracing.
2	The type MUST be DistributedTracing.

This will extend the OpenShift UI with a new navigation link "Observe" > "Traces".

On the next Episode

The next article will cover the deployment of the Central OpenTelemetry Collector. We will configure the RBAC permissions required for the Central Collector to enrich traces with Kubernetes metadata and deploy the Central OpenTelemetry Collector with its complete configuration.

The Hitchhiker's Guide to Observability - Central Collector - Part 3

Tue, 25 Nov 2025 00:00:00 +0000

With the architecture defined in Part 1 and TempoStack deployed in Part 2, it’s time to tackle the heart of our distributed tracing system: the Central OpenTelemetry Collector. This is the critical component that sits between your application namespaces and TempoStack, orchestrating trace flow, metadata enrichment, and tenant routing.

In this article, we’ll configure the RBAC permissions required for the Central Collector to enrich traces with Kubernetes metadata and deploy the Central OpenTelemetry Collector with its complete configuration. You’ll learn how to set up receivers for accepting traces from local collectors, configure processors to enrich traces with Kubernetes and OpenShift metadata, and implement routing connectors to direct traces to the appropriate TempoStack tenants based on namespace.

To be honest, this was the most challenging part of the entire setup to get right, as you can easily miss a setting, or misconfigure a part of the configuration. But once you understand the configuration, it becomes straightforward to extend and modify.

Central OpenTelemetry Collector - Step-by-Step Implementation

Prerequisites

Before starting, ensure you have completed the previous parts and have:

Part 1 completed: Hitchhiker’s Guide to Observability with OpenTelemetry and Tempo
Part 2 completed: Deploy Grafana Tempo and TempoStack
OpenShift cluster (v4.20) with Red Hat build of OpenTelemetry Operator installed (v0.135.0-1)
Access Cluster-admin access

Step 1: Verify/Deploy Red Hat Build of OpenTelemetry Operator

Verify if the Operator Red Hat Build of OpenTelemetry is installed and ready. The Operator itself is deployed in the namespace openshift-opentelemetry-operator

Step 2: Configure RBAC for Central Collector

For the OpenTelemetry Collector to function properly with processors like k8sattributes and resourcedetection, it requires cluster-wide read access to Kubernetes resources. Depending on the configuration of the central collector, you might need to configure different RBAC settings to allow the collector to perform specific things. The central collector in our example uses the k8sattributes processor and resourcedetection processor to enrich traces with Kubernetes and OpenShift metadata. These processors require read access to cluster resources.

Why These Permissions Are Needed

The k8sattributes processor enriches telemetry data by querying the Kubernetes API to add metadata such as:

Pod information: Pod name, UID, start time
Namespace details: Namespace name and labels
Deployment context: ReplicaSet and Deployment names
Node information: Node name where the pod is running

The resourcedetection processor detects the OpenShift environment by querying:

Infrastructure resources: Cluster name, platform type, region (from config.openshift.io/infrastructures)

Without these permissions, traces would lack critical context needed for debugging and analysis.

Always check the latest documentation of the appropriate component to get the most up to date information. I prefer to use separate ClusterRoles for each processor to keep the permissions as granular as possible. However, that causes some overhead, so you might want to combine the permissions into a single ClusterRole.

Create ClusterRole for Kubernetes Attributes

The following ClusterRole has been created:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: k8sattributes-otel-collector
rules:
# Permissions for k8sattributes processor
# Allows the collector to read pod, namespace, and replicaset information
# to enrich traces with Kubernetes metadata
- verbs:
- get
- watch
- list
apiGroups:
- '*'
resources:
- pods
- namespaces
- replicasets
# Permissions for resourcedetection processor (OpenShift)
# Allows detection of OpenShift cluster information
# such as cluster name, platform type, and region
- verbs:
- get
- watch
- list
apiGroups:
- config.openshift.io
resources:
- infrastructures
- infrastructures/status

Key Points:

Read-Only Access: The collector only needs get, watch, and list verbs (no write permissions)
Cluster-Wide Scope: ClusterRole grants permissions across all namespaces, necessary for monitoring multi-tenant environments
Essential Resources:
- pods: Source of trace context (which pod generated the span)
- namespaces: Namespace metadata and labels for routing
- replicasets: Determine the owning Deployment for better trace attribution
OpenShift Infrastructure: Access to config.openshift.io/infrastructures allows detection of cluster-level properties

Create ClusterRoleBinding

Our OpenTelemetry Collector will use the ServiceAccount otel-collector (in the namespace tempostack) to read the Kubernetes resources. Thus, we need to create a ClusterRoleBinding to grant the necessary permissions to the ServiceAccount.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: k8sattributes-collector-tempo
subjects:
- kind: ServiceAccount
name: otel-collector (1)
namespace: tempostack (2)
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: k8sattributes-otel-collector (3)

1	The ServiceAccount that is allowed to write the traces. In this example: otel-collector.
2	The namespace of the ServiceAccount. In this example: tempostack.
3	The reference to the ClusterRole

Security Note

The above permissions in the ClusterRole follow the principle of least privilege:

Only read operations are granted (no create, update, or delete)
Access is limited to specific resource types needed for metadata enrichment
The ServiceAccount otel-collector is dedicated to the collector and not shared with other applications

Step 3: Create ServiceAccount for Central OpenTelemetry Collector

The ServiceAccount used by the OpenTelemetry Collector. This is the service account that will be used to authenticate to the TempoStack instance. Thus, the Bindings created earlier to write into TempoStack and to read from Kubernetes will be required.

In this article we are installing the central Collector into the same namespace as the TempoStack instance. However, you might want to install it in a different namespace to keep the namespaces separated. Keep an eye on possible Network Policies that might be required to allow the communication between the namespaces.

apiVersion: v1
kind: ServiceAccount
metadata:
name: otel-collector
namespace: tempostack

Step 4: Deploy Central Collector

The central collector receives traces from local collectors, enriches them with Kubernetes metadata, and routes them to appropriate TempoStack tenants. For the sake of simplicity, I have taken snippets from the whole Configuration manifest. At the end of this section you will find the whole manifest.

Basic Configuration

The very basic settings of the OpenTelemetry Collector, are the amount of replicas, the ServiceAccount to use and the deployment mode.

The Collector can be deployed in one of the following modes:

Deployment (default) - Creates a Deployment with the given numbers of replicas.
StatefulSet - Creates a StatefulSet with the given numbers of replicas. Useful for stateful workloads, for example when using the Collector’s File Storage Extension or Tail Sampling Processor.
DaemonSet - Creates a DaemonSet with the given numbers of replicas. Useful for scraping telemetry data from every node, for example by using the Collector’s Filelog Receiver to read container logs.
Sidecar - Injects the Collector as a sidecar into the pod. Useful for accessing log files inside a container, for example by using the Collector’s Filelog Receiver and a shared volume such as emptyDir.

In the examples in this series of articles we will use the modes: deployment and sidecar.

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
name: otel
namespace: tempostack
spec:
mode: deployment (1)
replicas: 1 (2)
serviceAccount: otel-collector (3)
[...]

1	The deployment mode to use.
2	The number of replicas to use.
3	The ServiceAccount to use, created in the previous step.

Receivers

The Receivers are the components that receive the traces from the local collectors. Receivers accept data in a specified format and translate it into the internal format. In our example we want to receive the traces from the local collectors. For our tests we are using the oltp Receiver, which is collecting traces, metrics and logs by using the OpenTelemetry Protocol (OTLP).

The easiest configuration is:

 receivers:
otlp: (1)
protocols:
grpc:
endpoint: 0.0.0.0:4317 (2)
http:
endpoint: 0.0.0.0:4318 (3)

1	The name of the receiver.
2	The gRPC endpoint to receive.
3	The http endpoint to receive.

Besides the otlp receiver, there are other receivers available. For example:

Jaeger
Kubernetes Object Receiver
Kubelet Stats Receiver
Prometheus Receiver
Filelog Receiver
Journald Receiver
Kubernetes Events Receiver
Kubernetes Cluster Receiver
OpenCensus Receiver
Zipkin Receiver
Kafka Receiver.

Please check the OpenShift OTEL Receivers documentation for more details.

Processors

The Processors are the components that process the data after it is received and before it is exported. Processors are completely optional, but they are useful to transform, enrich, or filter traces.

The order of processors matters.

The example configuration is using the following processors:

k8sattributes: Can add Kubernetes metadata to the traces.
resourcedetection: Can detect OpenShift/K8s environment info.
memory_limiter: Periodically checks the Collector’s memory usage and pauses data processing when the soft memory limit is reached.
batch: Batches the traces for efficiency. This is a very important processor to improve the performance of the Collector.

Additional processors are available. Please check the OpenShift OTEL Processors documentation for more details.

Some processors will requires additional ClusterRole configuration.

 # Processors - enrich and batch traces
processors:
# Add Kubernetes metadata
k8sattributes: {} (1)
# Detect OpenShift/K8s environment info
resourcedetection: (2)
detectors:
- openshift
timeout: 2s
# Memory protection
memory_limiter: (3)
check_interval: 1s
limit_percentage: 75
spike_limit_percentage: 15
# Batch for efficiency
batch: (4)
send_batch_size: 10000
timeout: 10s

1	The k8sattributes processor to add Kubernetes metadata to the traces.
2	The resourcedetection processor to detect OpenShift/K8s environment info.
3	The memory_limiter processor to protect the Collector’s memory usage.
4	The batch processor to batch the traces for efficiency.

Connectors

A Connector joins two pipelines together. It consumes data as an exporter at the end of one pipeline and emits data as a receiver at the start of another pipeline. It can consume and emit data of the same or different data type. It can generate and emit data to summarize the consumed data, or it can merely replicate or route data.

Several Connectors are available, for example:

Count Connector: Counts traces spans, trace span events, metrics, metric data points, and log records.
Routing Connector: Routes the traces to different pipelines.
Forward Connector: Merges two pipelines of the same type.
Spanmetrics Connector: Aggregates Request, Error, and Duration (R.E.D) OpenTelemetry metrics from span data.

The OpenShift OTEL Connectors documentation lists all available Connectors and their configuration options.

In our example we are using the routing Connector, which is able to route the traces to different pipelines based on the namespace. This helps to route the traces to the correct tenant, without the need to configure this in the local OpenTelemetry Collector. (In other words, the project cannot change this setting, because it is configured in the central OpenTelemetry Collector.)

In this example traces from the namespace "team-a" will be routed to the pipeline "tenantA", traces from the namespace "team-b" will be routed to the pipeline "tenantB", and so on.

 # Connectors - route traces to different pipelines
connectors:
routing/traces:
default_pipelines: (1)
- traces/Default
error_mode: ignore (2)
table:
# Route team-a namespace to tenantA
- statement: route() where attributes["k8s.namespace.name"] == "team-a" (3)
pipelines:
- traces/tenantA
# Route team-b namespace to tenantB
- statement: route() where attributes["k8s.namespace.name"] == "team-b"
pipelines:
- traces/tenantB
# Route team-c namespace to tenantC
- statement: route() where attributes["k8s.namespace.name"] == "team-c"
pipelines:
- traces/tenantC

1	Destination pipelines for routing the telemetry data for which no routing condition is satisfied.
2	Error-handling mode: Defines how the connector handles routing errors: `propagate`: Logs an error and drops the payload (stops processing) `ignore`: Logs the error but continues attempting to match subsequent routing rules `silent`: Same as `ignore` but without logging Default: `propagate`
3	Route traces from namespace team-a to the pipeline tenantA.

Exporters

The Exporters are the components that export the traces to a destination. Exporters accept data in a specified format and translate it into the destination format. In our example we want to export the traces to the TempoStack instance. The X-Scope-OrgID header is used to identify the tenant and is sent to the TempoStack instance. The authentication is done by using the ServiceAccount token.

Many different Exporters are available. The OpenShift OTEL Exporters documentation lists all available Exporters and their configuration options.

For our tests we are using the otlp Exporter, which will export using the OpenTelemetry Protocol (OTLP):

 exporters:
# Tenant A exporter
otlp/tenantA: (1)
endpoint: tempo-simplest-gateway:8090 (2)
auth:
authenticator: bearertokenauth
headers:
X-Scope-OrgID: tenantA (3)
tls:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
insecure_skip_verify: true (4)
server_name_override: tempo-simplest-gateway.tempostack.svc.cluster.local (5)

1	Protocol/name of the exporter.
2	The endpoint to export to. Here, the endpoint is the address of the TempoStack instance.
3	The scope org ID to use.
4	Whether to skip certificate verification. I did not bother with certificates in this example.
5	The server name override to use. This was just for testing purposes and can be omitted.

Extensions

The Extensions extend the Collector capabilities. In our example we are using the bearertokenauth Extension, which is used to authenticate to the TempoStack instance.

 # Extensions - authentication
extensions:
bearertokenauth:
filename: /var/run/secrets/kubernetes.io/serviceaccount/token

Service:

Components are enabled by adding them into a Pipeline. If a component is not configured in a pipeline, it is not enabled.

In this example we are using the following pipelines (snippets):

traces/in: The incoming traces pipeline.
traces/tenantA: The tenant A pipeline.

The traces/in pipeline is the incoming traces pipeline. It is used to receive the traces from the local collectors. It leverages the otlp Receiver to receive the traces. It uses the resourcedetection, k8sattributes, memory_limiter, and batch processors to process the traces. And finally, it uses the routing/traces Connector to route the traces.

The routing/traces Connector is used to route the traces to the correct tenant based on the namespace.

The traces/tenantA Pipeline will receive the traces from routing/traces and export the traces to otlp/tenantA which then sends everything to TempoStack to store the traces using the header X-Scope-OrgID: tenantA.

 # Service pipelines
service:
extensions:
- bearertokenauth
pipelines:
# Incoming traces pipeline
traces/in: (1)
receivers: (2)
- otlp
processors: (3)
- resourcedetection
- k8sattributes
- memory_limiter
- batch
exporters: (4)
- routing/traces
# Tenant A pipeline
traces/tenantA: (5)
receivers: (6)
- routing/traces
exporters: (7)
- otlp/tenantA

1	The incoming traces pipeline. It takes the traces from the otlp receiver.
2	The receivers to use.
3	The processors to use.
4	The exporters to use. It is exporting the data to the routing connector.
5	The tenant A pipeline.
6	The receivers to use.
7	The exporters to use. It is exporting to the exporter, that will send the data to TempoStack.

The service automatically creates Kubernetes Services for the receivers. The Central Collector will be accessible at otel-collector.tempostack.svc.cluster.local:4317 (gRPC) and :4318 (HTTP) for local collectors to send traces.

Data Flow Visualization

To better understand how traces flow through the Central Collector, the following Mermaid diagram visualizes the complete journey from application to storage using team-a as an example:

---
title: "Central Collector Data Flow (Wrapped Layout)"
config:
theme: 'dark'
---
flowchart LR
%% --------------------------
%% Column 1: Local Collector
%% --------------------------
subgraph local["(team-a namespace)"]
app["Application<br/>(team-a)"]
end
%% --------------------------
%% Column 2: Central Collector
%% --------------------------
subgraph central["Central Collector (tempostack namespace)"]
%% We force this specific column to stack vertically (Top-Bottom)
direction TB
%% --- ROW 1: Receiver + Processors ---
subgraph row1[" "]
direction LR
receiver["OTLP Receiver<br/>Port: 4317/4318"]
subgraph pipeline_in["traces/in Processors"]
proc1["resourcedetection<br/>(Detect OpenShift)"]
proc2["k8sattributes<br/>(Add K8s metadata)"]
proc3["memory_limiter<br/>(Protect memory)"]
proc4["batch<br/>(Batch traces)"]
end
end
%% --- ROW 2: Connector + Tenant Pipeline ---
subgraph row2[" "]
direction LR
exporter_routing["Exporter: to connector"]
connector["Export to routing/traces Connector<br/>(Route by namespace)"]
subgraph pipeline_tenant["Pipeline: traces/tenantA"]
receiver_tenant["Receiver:<br/>routing/traces"]
exporter_tenant["Exporter:<br/>otlp/tenantA"]
end
end
end
%% --------------------------
%% Column 3: TempoStack
%% --------------------------
subgraph tempo["TempoStack"]
gateway["Tempo Gateway<br/>(X-Scope-OrgID: tenantA)"]
storage["S3 Storage<br/>(tenantA traces)"]
end
%% --------------------------
%% Connections
%% --------------------------
app -->|"OTLP traces"| receiver
receiver --> proc1 --> proc2 --> proc3 --> proc4
%% The Wrap: Connect end of Row 1 to start of Row 2
proc4 --> exporter_routing
exporter_routing --> connector
connector -->|"Route by namespace=team-a<br/>→ tenantA"| receiver_tenant
receiver_tenant --> exporter_tenant
exporter_tenant -->|"OTLP + bearer token<br/>Header: X-Scope-OrgID"| gateway
gateway --> storage
%% --------------------------
%% Styles
%% --------------------------
classDef appStyle fill:#2f652a,stroke:#2f652a,stroke-width:2px
classDef receiverStyle fill:#425cc6,stroke:#425cc6,stroke-width:2px
classDef processorStyle fill:#4a90e2,stroke:#4a90e2,stroke-width:2px
classDef connectorStyle fill:#906403,stroke:#906403,stroke-width:2px
classDef exporterStyle fill:#7b4397,stroke:#7b4397,stroke-width:2px
classDef tempoStyle fill:#d35400,stroke:#d35400,stroke-width:2px
class app appStyle
class receiver,receiver_tenant receiverStyle
class proc1,proc2,proc3,proc4 processorStyle
class connector connectorStyle
class exporter_tenant exporterStyle
class exporter_routing exporterStyle
class gateway,storage tempoStyle
%% Hide the structural boxes for the rows so they look seamless
style row1 fill:none,stroke:none
style row2 fill:none,stroke:none

Key Points in the Flow:

Application in team-a namespace sends traces to local collector
Local Collector forwards to Central Collector’s OTLP receiver
Pipeline traces/in processes the traces sequentially:
- Detects OpenShift environment info
- Adds Kubernetes metadata (namespace, pod, labels)
- Applies memory limits
- Batches traces for efficiency
Routing Connector examines the namespace attribute and routes to the correct tenant pipeline
Pipeline traces/tenantA receives from connector and exports to TempoStack
Exporter adds authentication (bearer token) and tenant ID header (X-Scope-OrgID: tenantA)
TempoStack receives and stores traces in the appropriate tenant storage

This architecture ensures complete isolation between tenants while maintaining a single, centralized collection point.

Complete OpenTelemetry Collector Manifest

Let’s put everything together in the complete OpenTelemetry Collector Manifest. The following defines the Central OpenTelemetry Collector:

mode: The deployment mode to use.
replicas: The number of replicas to use.
serviceAccount: The ServiceAccount to use, created in the previous step.
config: The configuration of the OpenTelemetry Collector.
- receivers: The receivers to use.
- processors: The processors to use.
- connectors: The connectors to use.
- exporters: The exporters to use.
- extensions: The extensions to use.
- service: The service to use.

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
name: otel
namespace: tempostack
spec:
mode: deployment
replicas: 1
serviceAccount: otel-collector
config:
# Receivers - accept traces from local collectors
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
# Processors - enrich and batch traces
processors:
# Add Kubernetes metadata
k8sattributes: {}
# Detect OpenShift/K8s environment info
resourcedetection:
detectors:
- openshift
timeout: 2s
# Memory protection
memory_limiter:
check_interval: 1s
limit_percentage: 75
spike_limit_percentage: 15
# Batch for efficiency
batch:
send_batch_size: 10000
timeout: 10s
# Connectors - route traces to different pipelines
connectors:
routing/traces:
default_pipelines:
- traces/Default
error_mode: ignore
table:
# Route team-a namespace to tenantA
- statement: route() where attributes["k8s.namespace.name"] == "team-a"
pipelines:
- traces/tenantA
# Route team-b namespace to tenantB
- statement: route() where attributes["k8s.namespace.name"] == "team-b"
pipelines:
- traces/tenantB
# Exporters - send to TempoStack
exporters:
# Default tenant exporter
otlp/Default:
endpoint: tempo-simplest-gateway:8090
auth:
authenticator: bearertokenauth
headers:
X-Scope-OrgID: dev
tls:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
insecure_skip_verify: true
server_name_override: tempo-simplest-gateway.tempostack.svc.cluster.local
# Tenant A exporter
otlp/tenantA:
endpoint: tempo-simplest-gateway:8090
auth:
authenticator: bearertokenauth
headers:
X-Scope-OrgID: tenantA
tls:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
insecure_skip_verify: true
server_name_override: tempo-simplest-gateway.tempostack.svc.cluster.local
# Tenant B exporter
otlp/tenantB:
endpoint: tempo-simplest-gateway:8090
auth:
authenticator: bearertokenauth
headers:
X-Scope-OrgID: tenantB
tls:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
insecure_skip_verify: true
server_name_override: tempo-simplest-gateway.tempostack.svc.cluster.local
# Extensions - authentication
extensions:
bearertokenauth:
filename: /var/run/secrets/kubernetes.io/serviceaccount/token
# Service pipelines
service:
extensions:
- bearertokenauth
pipelines:
# Incoming traces pipeline
traces/in:
receivers:
- otlp
processors:
- resourcedetection
- k8sattributes
- memory_limiter
- batch
exporters:
- routing/traces
# Default tenant pipeline
traces/Default:
receivers:
- routing/traces
exporters:
- otlp/Default
# Tenant A pipeline
traces/tenantA:
receivers:
- routing/traces
exporters:
- otlp/tenantA
# Tenant B pipeline
traces/tenantB:
receivers:
- routing/traces
exporters:
- otlp/tenantB

GitOps Deployment

While the above is good for quick tests, it always makes sense to have a proper GitOps deployment. I have created a Chart and GitOps configuration that will:

Deploy the OTEL Operator
Configure the the Central Collector instance

The following sources will be used:

Helm Repository - to fetch the Helm Chart for the OTEL Operator including required Sub-Charts.
Setup OTEL Operator - To deploy and configure the OpenTelemetry Operator.

Feel free to clone or use whatever you need.

The following Sub-Charts are used:

helper-operator (version ~1.0.18) - Installs the OpenTelemetry Operator
helper-status-checker (version ~4.0.0) - Verifies the status of the OpenTelemetry Operator
opentelemetry-collector (version ~1.0.0) - Creates the Central OpenTelemetry Collector instance
tpl (version ~1.0.0) - Template Library

The following Argo CD Application will deploy the OTEL Operator:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: setup-otel-operator
namespace: openshift-gitops
spec:
destination:
name: in-cluster (1)
namespace: openshift-opentelemetry-operator (2)
info:
- name: Description
value: ApplicationSet that Deploys on Management Cluster Configuration (using Git Generator)
project: in-cluster (3)
source:
path: clusters/management-cluster/setup-otel-operator (4)
repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops' (5)
targetRevision: main
syncPolicy:
retry:
backoff:
duration: 5s
factor: 2
maxDuration: 3m
limit: 5

1	Target cluster, here the local cluster
2	Namespace of the target cluster, here the Operator will be installed.
3	Project of the target cluster
4	Path to the Git repository
5	URL to the Git repository

This will create the Argo CD Application that can be synchronized with the cluster:

Complete values file

To see the whole file expand the code:

---
otel: &channel-otel stable
otel-namespace: &otel-namespace openshift-opentelemetry-operator
######################################
# SUBCHART: helper-operator
# Operators that shall be installed.
######################################
helper-operator:
operators:
opentelemetry-product:
enabled: true
namespace:
name: *otel-namespace
create: true
subscription:
channel: *channel-otel
approval: Automatic
operatorName: opentelemetry-product
source: redhat-operators
sourceNamespace: openshift-marketplace
operatorgroup:
create: true
notownnamespace: true
########################################
# SUBCHART: helper-status-checker
# Verify the status of a given operator.
########################################
helper-status-checker:
enabled: true
approver: false
checks:
- operatorName: opentelemetry-product
namespace:
name: *otel-namespace
syncwave: 1
serviceAccount:
name: "status-checker-otel"
rh-build-of-opentelemetry:
#########################################################################################
# namespace ... disabled here, since we deployed it via Tempo already
#########################################################################################
namespace:
name: tempostack
create: false
#########################################################################################
# OPENTELEMETRY COLLECTOR - Production Configuration
#########################################################################################
collector:
enabled: true
name: otel
mode: deployment
replicas: 1
serviceAccount: otel-collector
managementState: managed
resources: {}
tolerations: []
config:
connectors:
routing/traces:
default_pipelines:
- traces/Default
error_mode: ignore
table:
- pipelines:
- traces/tenantA
statement: 'route() where attributes["k8s.namespace.name"] == "team-a"'
- pipelines:
- traces/tenantB
statement: 'route() where attributes["k8s.namespace.name"] == "team-b"'
- pipelines:
- traces/tenantC
statement: 'route() where attributes["k8s.namespace.name"] == "team-c"'
- pipelines:
- traces/tenantD
statement: 'route() where attributes["k8s.namespace.name"] == "team-d"'
- pipelines:
- traces/tenantX
statement: 'route() where attributes["k8s.namespace.name"] == "mockbin-1"'
receivers:
# OTLP receivers for traces, metrics, and logs
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
# Jaeger receiver (if migrating from Jaeger)
jaeger:
protocols:
grpc:
endpoint: 0.0.0.0:14250
thrift_http:
endpoint: 0.0.0.0:14268
processors:
batch:
send_batch_size: 10000
timeout: 10s
k8sattributes: {}
memory_limiter:
check_interval: 1s
limit_percentage: 75
spike_limit_percentage: 15
resourcedetection:
detectors:
- openshift
timeout: 2s
exporters:
otlp/tenantX:
auth:
authenticator: bearertokenauth
endpoint: 'tempo-simplest-gateway:8090'
headers:
X-Scope-OrgID: tenantX
tls:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
insecure: true
insecure_skip_verify: true
server_name_override: tempo-simplest-gateway.tempostack.svc.cluster.local
otlp:
auth:
authenticator: bearertokenauth
endpoint: 'tempo-simplest-gateway:8090'
headers:
X-Scope-OrgID: prod
tls:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
insecure: true
insecure_skip_verify: true
server_name_override: tempo-simplest-gateway.tempostack.svc.cluster.local
otlp/tenantA:
auth:
authenticator: bearertokenauth
endpoint: 'tempo-simplest-gateway:8090'
headers:
X-Scope-OrgID: tenantA
tls:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
insecure: true
insecure_skip_verify: true
server_name_override: tempo-simplest-gateway.tempostack.svc.cluster.local
otlp/tenantB:
auth:
authenticator: bearertokenauth
endpoint: 'tempo-simplest-gateway:8090'
headers:
X-Scope-OrgID: tenantB
tls:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
insecure: true
insecure_skip_verify: true
server_name_override: tempo-simplest-gateway.tempostack.svc.cluster.local
otlp/Default:
auth:
authenticator: bearertokenauth
endpoint: 'tempo-simplest-gateway:8090'
headers:
X-Scope-OrgID: dev
tls:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
insecure: true
insecure_skip_verify: true
server_name_override: tempo-simplest-gateway.tempostack.svc.cluster.local
extensions:
bearertokenauth:
filename: /var/run/secrets/kubernetes.io/serviceaccount/token
service:
extensions:
- bearertokenauth
pipelines:
traces/Default:
exporters:
- otlp/Default
receivers:
- routing/traces
traces/in:
exporters:
- routing/traces
processors:
- resourcedetection
- k8sattributes
- memory_limiter
- batch
receivers:
- otlp
traces/tenantA:
exporters:
- otlp/tenantA
receivers:
- routing/traces
traces/tenantB:
exporters:
- otlp/tenantB
receivers:
- routing/traces

The Central OpenTelemetry Collector

With this deployment a single Pod (because we have set the replica count to 1) is running in the namespace tempostack with the name otel-collector.

oc get pods -n tempostack | grep otel-col
otel-collector-75f8794dc6-rbq26 1/1 Running 0 52m

Stay Tuned

The next article will cover deploying an example application (Mockbin) and configuring Local OpenTelemetry Collectors in application namespaces to send traces to this Central Collector. We’ll also demonstrate how the namespace-based routing automatically directs traces to the correct TempoStack tenants.

The Hitchhiker's Guide to Observability - Example Applications - Part 4

Wed, 26 Nov 2025 00:00:00 +0000

With the architecture defined, TempoStack deployed, and the Central Collector configured, we’re now ready to complete the distributed tracing pipeline. It’s time to deploy real applications and see traces flowing through the entire system!

In this fourth installment, we’ll focus on the application layer - deploying Local OpenTelemetry Collectors in team namespaces and configuring example applications to generate traces. You’ll see how applications automatically get enriched with Kubernetes metadata, how namespace-based routing directs traces to the correct TempoStack tenants, and how the entire two-tier architecture comes together.

We’ll deploy an example application (Mockbin) into two different namespaces. Together with the application, we will deploy a local OpenTelemetry Collector. One with sidecar mode, one with deployment mode.

You’ll learn how to configure local collectors to add namespace attributes, forward traces to the central collector, and verify that traces appear in the UI with full Kubernetes context.

By the end of this article, you’ll have a fully functional distributed tracing system with applications generating traces, local collectors forwarding them, and the central collector routing everything to the appropriate TempoStack tenants. Let’s bring this architecture to life!

Application Mockbin - Step-by-Step Implementation

Prerequisites

Before starting, ensure you have the following Systems or Operators installed (used Operator versions in this article):

OpenShift or Kubernetes cluster (OpenShift v4.20)
Red Hat build of OpenTelemetry installed (v0.135.0-1)
Tempo Operator installed (v0.18.0-2)
S3-compatible storage (for TempoStack, based on OpenShift Data Foundation)
Cluster Observability Operator (v1.3.0) for now this Operator is only used to extend the OpenShift UI with the tracing UI)

Mockbin Application Introduction

The Mockbin Application was created and provided by Michaela Lang. She is doing a lot of testings with Service Mesh and Observability. I forked the source code for the Mockbin application to Mockbin and created an images to test everything at: Mockbin Image. Mockbin is a simple, OpenTelemetry-instrumented HTTP testing service built with Python’s aiohttp async framework. It serves as both a demonstration tool and a practical testing service for distributed tracing infrastructure.

Why Mockbin?

Unlike simple "hello world" examples, Mockbin demonstrates real-world OpenTelemetry implementation patterns that you can apply to your own Python applications. It shows how to properly instrument an async Python web service with full observability capabilities.

Key Features

Full OpenTelemetry Integration: Implements the complete observability stack - traces, metrics, and logs
Automatic Instrumentation: Uses OpenTelemetry’s auto-instrumentation for aiohttp
Manual Span Creation: Demonstrates how to create custom spans with attributes and events
Trace Context Propagation: Supports both W3C Trace Context (traceparent) and Zipkin B3 (x-b3-*) formats
OTLP Export: Sends all telemetry data via OpenTelemetry Protocol (OTLP) to collectors
Prometheus Metrics with Exemplars: Links high-cardinality traces to aggregated metrics
Structured Logging via OTLP: Logs are automatically correlated with traces
Nested Spans: Creates parent-child span relationships for complex operations
Exception Recording: Captures and records exceptions with full stack traces

Technical Architecture

The application consists of several Python modules:

proxy.py: Main application with HTTP endpoint handlers
tracing.py: OpenTelemetry configuration and trace context management
promstats.py: Prometheus metrics collection with trace exemplars
logfilter.py: OTLP logging with automatic trace correlation

HTTP Endpoints for Testing

Endpoint

Purpose

Use Case

Echo service - returns headers and environment variables

Basic trace generation

/logging/*

Generates multiple log entries linked to trace

Demonstrate trace-to-log correlation

/proxy/*

Chains multiple HTTP requests with nested spans

Test distributed trace propagation

/exception/{status}

Intentionally raises exception and returns custom status code

Test error trace capture

/webhook/alert-receiver

Converts AlertManager webhooks to traces

Alert-to-trace correlation

/outlier

Enables/disables failure mode (PUT/DELETE)

Test Service Mesh outlier detection

/metrics

Exposes Prometheus metrics

Metrics scraping

/health

Health check endpoint

Kubernetes liveness/readiness probes

OpenTelemetry Implementation Highlights

The application demonstrates several critical OpenTelemetry patterns:

Middleware-Based Instrumentation: Every HTTP request is automatically wrapped in a trace span via aiohttp middleware, capturing request metadata (method, URL, headers, etc.) as span attributes.
Context Propagation: Incoming requests have their trace context extracted from HTTP headers, and outgoing requests inject the context to maintain trace continuity across service boundaries.
Resource Attributes: Service metadata (name, namespace, version) is attached to all telemetry data for proper identification in the backend.
Manual Span Creation: Shows how to create nested spans for complex operations, add custom attributes, record events, and set span status (OK/ERROR).

Configuration via Environment Variables

The application is configured entirely through environment variables, making it easy to adapt to different environments, for example:

# OpenTelemetry Configuration
OTEL_EXPORTER_OTLP_ENDPOINT=http://otelcol-collector:4317
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=http://otelcol-collector:4317
OTEL_SPAN_SERVICE=mockbin

We will test OpenTelemetry Collector in deployment mode and in sidecar mode. When using sidecar mode, then the sidecar will try to automatically inject the OpenTelemetry Collector into the pod.

Deploy Mockbin Application & OpenTelemetry Collector - team-a

Let’s start with the deployment of the Mockbin Application and the OpenTelemetry Collector in the team-a namespace. Here we will use the OpenTelemetry Collector in deployment mode.

The deployment in sidecar mode will be done in the team-b namespace and is almost the same, just with the difference that we will use a different mode in the OTC resource and that we do not need to take care of the environment variables for the OTC.

Step 1a: Use Kustomize Manually

You can deploy the Mockbin Application and the OpenTelemetry Collector manually using Kustomize or by using a GitOps tool like ArgoCD (preferred). In the repository Mockbin your will find the sub-folder k8s with the Kustomize files for the Mockbin Application and the OpenTelemetry Collector.

You can manually deploy the application:

Step 1a.1: Create Namespace team-a

First create the namespace team-a.

apiVersion: v1
kind: Namespace
metadata:
name: team-a (1)

1	Namespace name

Step 1a.2: Deploy the Mockbin Application

Use the following command to deploy the application (assuming you cloned the repository to your local machine):

kustomize build . | kubectl apply -f -

This will deploy the required resources:

ServiceAccount
Service
Route
Deployment using the image: quay.io/tjungbau/mockbin:1.8.1

It is possible to kustomize the deployment by modifying the kustomization.yaml file. For example, you can change the image, or change from Route to Ingress object.

Step 1b: Use ArgoCD

Above can be achieved by using ArgoCD, which allows a more declarative approach. Create the following ArgoCD Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: mockbin-team-a (1)
namespace: openshift-gitops (2)
spec:
destination:
namespace: team-a (3)
server: 'https://kubernetes.default.svc' (4)
info:
- name: Description
value: Deploy Mockbin in team-a namespace
project: in-cluster
source:
path: k8s/ (5)
repoURL: 'https://github.com/tjungbauer/mockbin' (6)
targetRevision: main (7)
syncPolicy:
syncOptions:
- CreateNamespace=true (8)

1	Name of the Argo CD Application resource.
2	Namespace of the ArgoCD Application resource, here it is openshift-gitops. In your environment it might be different.
3	Namespace of the target application
4	Kubernetes API server URL. Here it is the local cluster where Argo CD is hosted.
5	Path to the Kustomize files
6	URL to the Git repository
7	Target revision
8	Create the namespace if it does not exist

Argo CD will leverage Kustomize to render the resources. It is the same as you would do manually, with the exception that the Namespace will be created automatically. The approach above will deploy the required resources:

Namespace
ServiceAccount
Service
Route
Deployment

In the Argo CD UI you can see the application and the resources that have been deployed.

Step 2: Verify the Deployment

Verify the Pods, Service and Route of the Mockbin Application:

kubectl get pods -n team-a
NAME READY STATUS RESTARTS AGE
pod/mockbin-c4587558b-ftvsd 2/2 Running 0 3d15h
pod/mockbin-c4587558b-zrxgc 2/2 Running 0 3d15h
kubectl get services -n team-a
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/mockbin ClusterIP 172.30.202.223 <none> 8080/TCP 3d16h
kubectl get routes -n team-a
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
route.route.openshift.io/mockbin mockbin-team-a.apps.ocp.aws.ispworld.at mockbin http edge/Redirect None

Access the route URL to see the Mockbin Application:

curl -k https://mockbin-team-a.apps.ocp.aws.ispworld.at

This will return a response looking like this:

{"headers": {"User-Agent": "curl/8.7.1", "Accept": "*/*", "Host": "mockbin-team-a.apps.ocp.aws.ispworld.at", .... lot's of header stuff"}

Step 3: Deploy the Local OpenTelemetry Collector - Mode Deployment

All steps below can be applied using GitOps again, using the Chart rh-build-of-opentelemetry. An example of a GitOps configuration (for the Central Collector) can be found at Setup OTEL Operator.

Create a ServiceAccount for the OpenTelemetry Collector

Let us first create a ServiceAccount for the OpenTelemetry Collector.

apiVersion: v1
kind: ServiceAccount
metadata:
name: otelcol-agent
namespace: team-a

Create the OpenTelemetry Collector Resource

Create the following OpenTelemetry Collector Resource. The mode shall be "deployment" The serviceAccount shall be the one we created in the previous step, and the namespace is "team-a".

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
name: otelcol-agent (1)
namespace: team-a (2)
spec:
mode: deployment (3)
replicas: 1 (4)
serviceAccount: otelcol-agent (5)
config:
# Receivers - accept traces from applications
receivers: (6)
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
# Processors
processors: (7)
# Add namespace identifier
attributes:
actions:
- action: insert
key: k8s.namespace.name
value: team-a
# Batch for efficiency
batch: {}
# Exporters - forward to central collector
exporters:
otlp/central: (8)
endpoint: otel-collector.tempostack.svc.cluster.local:4317
tls:
insecure: true
# Service pipeline
service:
pipelines: (9)
traces:
receivers:
- otlp
processors:
- batch
- attributes
exporters:
- otlp/central

1	Name of the OpenTelemetry Collector Resource.
2	Namespace team-aof the OpenTelemetry Collector Resource.
3	Mode deployment of the OpenTelemetry Collector Resource.
4	Number of replicas of the OpenTelemetry Collector Resource. For HA setup increase the number of replicas.
5	ServiceAccount of the OpenTelemetry Collector Resource, that was created in the previous step.
6	Receivers of the OpenTelemetry Collector Resource. We will receive traces using the OTLP protocol on gRPC and HTTP.
7	Processors of the OpenTelemetry Collector Resource. We will add the namespace identifier as an attribute and batch the traces for efficiency.
8	Exporters of the OpenTelemetry Collector Resource. We will forward the traces to the Central Collector.
9	Service pipeline of the OpenTelemetry Collector Resource.

Since we choose the deployment mode, the OpenTelemetry Collector will be deployed as a Deployment resource.

oc deployment all -n team-a
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/otelcol-agent-collector 1/1 1 1 3d2h

Step 4: Cool, and now what? - Environment Variables

If you came that far, then you have deployment TempoStack, the Central Collector, the Mockbin Application and the Local Collector. However, you will not receive any traces yet. We need to tell the Mockbin Application WHERE to send the traces.

To do this, we need to set the environment variables for the Mockbin Deployment resource. Edit the Deployment resource and add the following environment variables:

 env:
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: 'http://otelcol-agent-collector:4317' (1)
- name: OTEL_EXPORTER_OTLP_PROTOCOL
value: grpc (2)
- name: OTEL_SERVICE_NAME
value: mockbin (3)

1	The endpoint (service) of the Local Collector.
2	The protocol we would like to use.
3	The service name of the Mockbin Application.

This will trigger a restart of the Deployment. Once done, the environment variables should show up:

curl -k https://mockbin-team-a.apps.ocp.aws.ispworld.at
{"headers": {"...., "OTEL_EXPORTER_OTLP_ENDPOINT": "http://otelcol-agent-collector:4317", "OTEL_EXPORTER_OTLP_PROTOCOL": "grpc", ...."}}

This will now start sending traces to the Local Collector and from there to the Central Collector.

Traces, Traces, Traces

Now we have a working setup. We have a Central Collector, a Local Collector and a Mockbin Application. Our application is permanently sending requests to the Local Collector.

We can review the traces in the OpenShift UI: Observe > Traces.

Select the tempostack instance and select the tenant tenantA. If everything is working, you should see traces from the Mockbin Application.

If no traces show up, then most probably the RBAC rules are not configured correctly or the environment variables are not set correctly. You can check the logs of the OTC pods for errors.

Deploy Mockbin Application & OpenTelemetry Collector - team-b

Let us now deploy the Mockbin Application and the OpenTelemetry Collector in the team-b namespace. Here we will use the OpenTelemetry Collector in sidecar mode.

Step 1: Deploy the Application using Argo CD

Step 1b: Use ArgoCD

Above can be achieved by using ArgoCD, which allows a more declarative approach. Create the following ArgoCD Application (This is the same approach as above)

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: mockbin-team-b (1)
namespace: openshift-gitops (2)
spec:
destination:
namespace: team-b (3)
server: 'https://kubernetes.default.svc' (4)
info:
- name: Description
value: Deploy Mockbin in team-b namespace
project: in-cluster
source:
path: k8s/ (5)
repoURL: 'https://github.com/tjungbauer/mockbin' (6)
targetRevision: main (7)
syncPolicy:
syncOptions:
- CreateNamespace=true (8)

1	Name of the Argo CD Application resource.
2	Namespace of the ArgoCD Application resource, here it is openshift-gitops. In your environment it might be different.
3	Namespace of the target application
4	Kubernetes API server URL. Here it is the local cluster where Argo CD is hosted.
5	Path to the Kustomize files
6	URL to the Git repository
7	Target revision
8	Create the namespace if it does not exist

Step 2: Deploy the Local OpenTelemetry Collector - Sidecar Deployment

All steps below can be applied using GitOps again, using the Chart rh-build-of-opentelemetry. An example of a GitOps configuration (for the Central Collector) can be found at Setup OTEL Operator.

Create a ServiceAccount for the OpenTelemetry Collector

Let us first create a ServiceAccount for the OpenTelemetry Collector.

apiVersion: v1
kind: ServiceAccount
metadata:
name: otelcol-agent
namespace: team-b

Create the OpenTelemetry Collector Resource

Create the following OpenTelemetry Collector Resource. The mode shall be "sidecar" The serviceAccount shall be the one we created in the previous step, and the namespace is "team-b".

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
name: otelcol-agent (1)
namespace: team-b (2)
spec:
mode: sidecar (3)
replicas: 1 (4)
serviceAccount: otelcol-agent (5)
config:
# Receivers - accept traces from applications
receivers: (6)
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
# Processors
processors: (7)
# Add namespace identifier
attributes:
actions:
- action: insert
key: k8s.namespace.name
value: team-b
# Batch for efficiency
batch: {}
# Exporters - forward to central collector
exporters:
otlp/central: (8)
endpoint: otel-collector.tempostack.svc.cluster.local:4317
tls:
insecure: true
# Service pipeline
service:
pipelines: (9)
traces:
receivers:
- otlp
processors:
- batch
- attributes
exporters:
- otlp/central

1	Name of the OpenTelemetry Collector Resource.
2	Namespace team-aof the OpenTelemetry Collector Resource.
3	Mode sidecar of the OpenTelemetry Collector Resource.
4	Number of replicas of the OpenTelemetry Collector Resource. For HA setup increase the number of replicas.
5	ServiceAccount of the OpenTelemetry Collector Resource, that was created in the previous step.
6	Receivers of the OpenTelemetry Collector Resource. We will receive traces using the OTLP protocol on gRPC and HTTP.
7	Processors of the OpenTelemetry Collector Resource. We will add the namespace identifier as an attribute and batch the traces for efficiency.
8	Exporters of the OpenTelemetry Collector Resource. We will forward the traces to the Central Collector.
9	Service pipeline of the OpenTelemetry Collector Resource.

Step 3: Enable Sidecar Injection for Team-B

The big advantage of the sidecar mode is that the OpenTelemetry Collector is deployed as a sidecar container in the same pod as the application. This happens fully automatically, means you do not need to set any extra environment variables or anything like that.

To enable the sidecar injection, we need to add the following annotation to the namespace. This will automatically inject the OpenTelemetry Collector as a sidecar container into the ALL pods of the namespace:

Do not forget this to add it into GitOps process :)

First verify the current state of the deployment:

oc deployment all -n team-b
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/otelcol-agent-collector 1/1 1 1 3d2h

As you can see one container out of 1 is running (Read 1/1)

Now lets modify the namespace to enable the sidecar injection:

apiVersion: v1
kind: Namespace
metadata:
name: team-b
annotations:
sidecar.opentelemetry.io/inject: "true"

This can also be done by annotate the Deployment resource. In this case, the OpenTelemetry Collector will be injected into the specific pod.

Restart the deployment to apply the changes:

oc rollout restart deployment/mockbin -n team-b
deployment.apps/mockbin restarted

Now, when you check again, the sidecar container should be injected into the pods. The Deployment has now 2/2 containers ready:

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/mockbin 2/2 1 2 3m53s

Traces, Traces, Traces - Again

Again, we can review the traces in the OpenShift UI: Observe > Traces.

Select the tempostack instance and select the tenant tenantB this time.

Argo CD Stays in Progressing State

If you have deployed the Mockbin Application and the OpenTelemetry Collector as a sidecar deployment in the team-b namespace, you might notice that the Argo CD Application stays in the Progressing state.

This is because the OpenTelemetry Collector does not really return a state. The status is simply:

status:
version: 0.140.0

Nothing else … thats too less for Argo CD to work with.

In order to fix this, we need to add a custom health check to the Argo CD instance.

The Argo CD instance knows the following part:

apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
name: openshift-gitops
namespace: openshift-gitops
spec:
[...]
resourceHealthChecks: (1)
- check: |
hs = {}
if obj.status ~= nil then
if obj.status.version ~= nil and obj.status.version ~= "" then
hs.status = "Healthy"
hs.message = "Running version " .. obj.status.version
else
hs.status = "Progressing"
hs.message = "Waiting for version report"
end
else
hs.status = "Progressing"
hs.message = "Status not available"
end
return hs
group: opentelemetry.io
kind: OpenTelemetryCollector
[...]

1	Custom health check for the OpenTelemetry Collector.

Above will end up in a ConfigMap that is managed by the OpenShift GitOps operator.

This will make Argo CD happy and the Application will be in the Progressing state.

The Hitchhiker's Guide to Observability - Understanding Traces - Part 5

Thu, 27 Nov 2025 00:00:00 +0000

With the architecture established, TempoStack deployed, the Central Collector configured, and applications generating traces, it’s time to take a step back and understand what we’re actually building. Before you deploy more applications and start troubleshooting performance issues, you need to understand how to read and interpret distributed traces.

Let’s decode the matrix of distributed tracing!

Understanding Distributed Traces

This article is not a comprehensive guide to distributed tracing. It is a quick overview to understand the building blocks of a trace.

We need to understand the building blocks of a trace to be able to interpret them in the UI. As the UI, we will use the integrated tracing interface inside OpenShift.

Also compare with: Trace Structure in Tempo Documentation.

What You Can Do With Traces

Performance Optimization
- Identify slow operations (database queries, API calls)
- Find bottlenecks in the critical path
- Compare performance across versions/deployments
Root Cause Analysis
- Trace errors back to their origin
- See the complete context of a failure
- Understand cascading failures
Service Dependencies
- Visualize your service architecture
- Identify tightly coupled services
- Plan capacity and scaling
User Experience Monitoring
- Track end-to-end latency for user actions
- Identify outliers and edge cases
- Correlate user complaints with actual traces
Capacity Planning
- Understand resource usage patterns
- Identify underutilized or overloaded services
- Plan infrastructure scaling
A/B Testing and Rollouts
- Compare performance between feature flags
- Verify canary deployments
- Measure impact of code changes

What is a Trace?

A trace represents the complete journey of a request as it flows through your system. Every service, database call, and external API interaction along the way will be tracked.

Key Characteristics:

Unique Trace ID: Every trace has a globally unique identifier (128-bit or 64-bit)
Timeline: Traces capture the temporal relationship between operations
Distributed Context: Maintains continuity across service boundaries
Hierarchical Structure: Organized as a tree of spans

What is a Span?

A span is the fundamental unit of work in distributed tracing. It represents a single operation within a trace - such as handling an HTTP request, executing a database query, or calling an external service.

Span Components

The following components of a span can be considered (among others):

Component

Description

Name

Human-readable description of the operation (e.g., "GET /api/users", "SELECT FROM orders")

Trace ID

Links this span to its parent trace

Span ID

Unique identifier for this specific span

Start Time

When the operation began (nanosecond precision)

Duration

How long the operation took

Span Kind

Type of span: SERVER, CLIENT, PRODUCER, CONSUMER, INTERNAL

Status

Operation outcome: OK, ERROR, UNSET

Trace Tree Structure

A trace forms a directed acyclic graph (DAG) - typically a tree structure where each span can have multiple children but only one parent.

Visual Example: Basic Trace Structure

---
config:
theme: 'neutral'
---
graph TD
%% Grouping everything as a Trace
subgraph Trace
direction TB
SpanA[Span A]
SpanB[Span B]
SpanC[Span C]
SpanD[Span D]
SpanE[Span E]
end
%% Quotes added to handle the curly braces
SpanA -->|"{Span context}"| SpanB
SpanA -->|"{Span context}"| SpanC
SpanC -->|"{Span context}"| SpanD
SpanC -->|"{Span context}"| SpanE

What This Trace Structure Shows:

Span A is the root span (parent of all other spans)
Span B and Span C are direct children of Span A (parallel operations)
Span D and Span E are children of Span C (sequential or parallel sub-operations)
The span context is propagated from parent to child, maintaining trace continuity
This hierarchical structure allows you to understand the complete request flow

Let’s have a look at a real trace in the OpenShift UI under Observe > Traces:

Span Attributes: Adding Context

Attributes are key-value pairs that add semantic meaning to spans. OpenTelemetry defines semantic conventions - standardized attribute names for common scenarios.

HTTP Attributes:

http.method: "POST"
http.url: "https://api.example.com/checkout"
http.status_code: 200
http.user_agent: "Mozilla/5.0..."

Database Attributes:

db.system: "postgresql"
db.name: "orders"
db.statement: "SELECT * FROM orders WHERE user_id = $1"
db.connection_string: "postgresql://db.example.com:5432"

Kubernetes Attributes (added by k8sattributes processor):

k8s.namespace.name: "team-a"
k8s.pod.name: "checkout-service-7d8f9c-xyz12"
k8s.deployment.name: "checkout-service"
k8s.node.name: "worker-node-2"

Custom Business Attributes:

user.id: "12345"
order.id: "ORD-98765"
order.total: 149.99
payment.method: "credit_card"
inventory.items_count: 3

Span Events: Timestamped Logs

Events are timestamped messages within a span that mark significant moments:

Span: Process Payment
├─ Event @ 10ms: "Payment request validated"
├─ Event @ 50ms: "Calling payment gateway"
├─ Event @ 750ms: "Payment gateway responded"
└─ Event @ 760ms: "Payment confirmed"

Use Cases for Span Events:

Debug checkpoints
Exception details
State transitions
External API interactions

Span Status and Error Handling

Spans have three status codes:

UNSET: Default, operation completed (not necessarily successful)
OK: Explicitly marked as successful
ERROR: Operation failed

Let’s call one of your application endpoints on the path /exception/500. This will return a 500 status code and the span will be marked as ERROR.

curl -X GET https://<YOUR-APPLICATION-URL>/exception/500

Now we can see the span in the trace with the error status. Note how the span is highlighted in red, indicating an error occurred:

The Hitchhiker's Guide to Observability - Adding A New Tenant - Part 6

Fri, 28 Nov 2025 00:00:00 +0000

While we have created our distributed tracing infrastructure, we created two tenants as an example. In this article, I will show you how to add a new tenant and which changes must be made in the TempoStack and the OpenTelemetry Collector.

This article was mainly created as a quick reference guide to see which changes must be made when adding new tenants.

Quick Checklist for Adding a New Tenant

Use this checklist to ensure you haven’t missed any steps:

TempoStack: Add tenant to spec.tenants.authentication[]
ClusterRole (write): Add tenant name to tempostack-traces-write resources
ClusterRole (read): Add tenant name to tempostack-traces-reader resources
Central OTC Routing: Add routing rule in connectors.routing/traces.table[]
Central OTC Exporter: Add otlp/[tenant-name] exporter with correct X-Scope-OrgID
Central OTC Pipeline: Add traces/[tenant-name] pipeline
Namespace: Create namespace for the team
ServiceAccount: Create ServiceAccount for local collector
Local OTC: Deploy OpenTelemetryCollector in team namespace
Application: Deploy application with OTEL configuration
Verify: Check logs, generate traces, view in Jaeger UI

You do NOT need to create new ClusterRoles for each tenant. The k8sattributes-otel-collector ClusterRole already grants the central collector cluster-wide read access to pods, namespaces, and replicasets across all namespaces. This single ClusterRole serves all tenants.

Common Mistakes to Avoid

Mismatched Tenant Names: Ensure the tenant name in TempoStack matches the X-Scope-OrgID header in the exporter
Wrong Namespace Attribute: The routing rule matches on k8s.namespace.name - ensure the local collector sets this correctly
Forgot RBAC Update: Without updating ClusterRoles, traces will be rejected
Typo in Pipeline Names: Pipeline names in the routing connector must match the actual pipeline definitions
Missing Exporter in Pipeline: Each new pipeline must reference the correct exporter (e.g., otlp/team-d)

Adding a New Tenant - team-d

Let’s add a new tenant called "team-d" with their own namespace, local collector, and route to TempoStack tenant "tenantD".

Step 1: Update TempoStack Configuration

If the tenant "tenantD" does not already exist in TempoStack, we need to add it to the tenants list.

Edit the TempoStack resource in the tempostack namespace and add the new tenant to the authentication list:

spec:
tenants:
mode: openshift
authentication:
- tenantId: 1610b0c3-c509-4592-a256-a1871353dbfc
tenantName: tenantA
- tenantId: 1610b0c3-c509-4592-a256-a1871353dbfd
tenantName: tenantB
- tenantId: 1610b0c3-c509-4592-a256-a1871353dbfe
tenantName: tenantC
# Add new tenant
- tenantId: 1610b0c3-c509-4592-a256-a1871353dbff (1)
tenantName: tenantD

1	Our new tenantID with the tenantName "tenantD"

Step 2: Update RBAC Permissions - write

Update the ClusterRoles to grant write permissions for the new tenant. Edit the ClusterRole tempostack-traces-write:

oc edit clusterrole tempostack-traces-write

rules:
- verbs:
- create
apiGroups:
- tempo.grafana.com
resources:
- tenantA
- tenantB
- tenantC
- tenantD (1)
resourceNames:
- traces

1	Our new tenant with the tenantName "tenantD"

Step 3: Update RBAC Permissions - read

Update the ClusterRoles to grant read permissions for the new tenant. Edit the ClusterRole tempostack-traces-reader:

oc edit clusterrole tempostack-traces-reader

rules:
- verbs:
- get
apiGroups:
- tempo.grafana.com
resources:
- dev
- tenantA
- tenantB
- tenantC
- tenantD (1)
resourceNames:
- traces

1	Our new tenant with the tenantName "tenantD"

Step 4: Update Central OpenTelemetry Collector

The central collector needs configuration changes to route traces from the new namespace to the appropriate TempoStack tenant. Edit the OpenTelemetry Collector in the tempostack namespace and add the new routing rule:

oc edit otelcol otel -n tempostack

Add the New Routing Rule

Add the new routing rule to the connectors section:

spec:
config:
connectors:
routing/traces:
default_pipelines:
- traces/Default
error_mode: ignore
table:
- statement: route() where attributes["k8s.namespace.name"] == "team-a"
pipelines:
- traces/tenantA
- statement: route() where attributes["k8s.namespace.name"] == "team-b"
pipelines:
- traces/tenantB
- statement: route() where attributes["k8s.namespace.name"] == "team-c"
pipelines:
- traces/tenantC
# Add new routing rule
- statement: route() where attributes["k8s.namespace.name"] == "team-d" (1)
pipelines:
- traces/tenantD

1	Our new tenant with the tenantName "tenantD"

Add Exporter for the New Tenant

Add the new exporter to the exporters section:

spec:
config:
exporters:
# ... existing exporters ...
# New tenant exporter
otlp/tenantD: (1)
endpoint: tempo-simplest-gateway:8090
auth:
authenticator: bearertokenauth
headers:
X-Scope-OrgID: tenantD (2)
tls:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
insecure_skip_verify: true
server_name_override: tempo-simplest-gateway.tempostack.svc.cluster.local

1	Our new exporter with the name "otlp/tenantD"
2	The `X-Scope-OrgID` header value that identifies the tenant

Add Pipeline for the New Tenant

Add the new pipeline to the pipelines section:

spec:
config:
service:
pipelines:
# ... existing pipelines ...
# New tenant pipeline
traces/tenantD: (1)
receivers:
- routing/traces
exporters:
- otlp/tenantD

1	Our new tenant with the tenantName "tenantD"

Step 5: Install the Application and Local OpenTelemetry Collector

Follow the steps from the previous article to install the application and local OpenTelemetry Collector in the new namespace.