The Hitchhiker's Guide to Observability - Grafana Tempo - Part 2

- Thomas Jungbauer Thomas Jungbauer ( Lastmod: 2025-11-28 ) - 13 min read

image from The Hitchhiker's Guide to Observability - Grafana Tempo - Part 2

After covering the fundamentals and architecture in Part 1, it’s time to get our hands dirty! This article walks through the complete implementation of a distributed tracing infrastructure on OpenShift.

We’ll deploy and configure the Tempo Operator and a multi-tenant TempoStack instance. For S3 storage we will use the integrated OpenShift Data Foundation. However, you can use whatever S3-compatible storage you have available.

Grafana Tempo - Step-by-Step Implementation

Prerequisites

Before starting, ensure you have the following Systems or Operators installed (used Operator versions in this article):

  • OpenShift or Kubernetes cluster (OpenShift v4.20)

  • Red Hat build of OpenTelemetry installed (v0.135.0-1)

  • Tempo Operator installed (v0.18.0-2)

  • S3-compatible storage (for TempoStack, based on OpenShift Data Foundation)

  • Cluster Observability Operator (v1.3.0) - for now this Operator is only used to extend the OpenShift UI with the tracing UI.

For all configurations I also created a proper GitOps implementation (of course :)). However, first I would like to show the actual configuration. The GitOps implementation can be found at the section GitOps Deployment.

Step 1: Verify/Deploy Tempo Operator

Let’s first verify if the Tempo Operator is installed and ready to use. If everything is fine, then the Operator will be deployed in the namespace openshift-tempo-operator:

Tempo Operator

Step 2: Deploy TempoStack Resource

TempoStack is the central trace storage backend. We are using the Tempo Operator here, which provides the TempoStack resource and multi-tenancy capability. During my tests I deployed the TempoStack resource and also created a Helm Chart that is able to render this resource. However, the Operator itself also provides a TempoMonolith resource. This will put everything into one Pod, while the TempoStack rolls out the stack in separate containers (like ingester, gateway, etc.).

My Helm Chart currently does not support the TempoMonolith resource yet. If you require this, please ping me and I will try to add it.
Be sure that the S3 bucket is available and a Secret (here called tempo-s3) with the following keys exists (valid for OpenShift Data Foundation): access_key_id, access_key_secret, bucket, endpoint. The layout of the Secret will look slightly different depending on the S3 storage backend you are using.

Create the TempoStack instance:

The following TempoStack resource has been used

apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: simplest (1)
  namespace: tempostack (2)
spec:
  managementState: Managed
  replicationFactor: 1 (3)

  # Resource limits
  resources: (4)
    total:
      limits:
        cpu: "2"
        memory: 2Gi

  # Trace retention
  retention: (5)
    global:
      traces: 48h0m0s

  # S3 storage configuration
  storage:
    secret:
      credentialMode: static (6)
      name: tempo-s3 (7)
      type: s3
    tls:
      enabled: false
  storageSize: 500Gi

  # Multi-tenancy configuration
  tenants:
    mode: openshift
    authentication: (8)
      - tenantId: 1610b0c3-c509-4592-a256-a1871353dbfa
        tenantName: tenantA
      - tenantId: 1610b0c3-c509-4592-a256-a1871353dbfb
        tenantName: tenantB
      - tenantId: 1610b0c3-c509-4592-a256-a1871353dbfc
        tenantName: tenantC

  # Gateway and UI
  template: (9)
    gateway:
      enabled: true
      component:
        replicas: 1
      ingress:
        type: route
        route:
          termination: reencrypt
    queryFrontend:
      component:
        replicas: 1
      jaegerQuery:
        enabled: true
        servicesQueryDuration: 72h0m0s
1Name of the TempoStack instance
2Namespace of the TempoStack instance
3Integer value for the number of ingesters that must acknowledge the data from the distributors before accepting a span.
4Defines resources for the TempoStack instance. Default is (limit only) 2 CPU and 2Gi memory.
5Configuration options for retention of traces. The default value is 48h.
6Credential mode for the S3 storage. Depends how the storage will be integrated. Default is static.
7Secret name for the S3 storage.
8Configuration options for the tenants. In this example: tenantA, tenantB, tenantC. Consists of tenantName and tenantId, both can be defined by the user.
9Configuration options for the different Tempo components. In this example: gateway, query-frontend. Other components could be: distributor, ingester, compactor or querier.

Key Configuration Points:

  • Multi-tenancy: Supports 3 tenants currently (tenantA, tenantB, tenantC). This part must be modified when a new tenant enters the realm.

  • Retention: Traces stored for 48 hours.

  • Storage: Uses S3-compatible backend (requires separate secret).

  • Gateway: Exposes OTLP endpoint with TLS.

Step 3: Configure RBAC for TempoStack Trace Access read/write

Set up ClusterRoles to control who can read and write traces.

The ClusterRoles must be updated whenever a new tenant is configured in TempoStack. The name of the tenant must be added in the resources array.

Traces Reader Role:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: tempostack-traces-reader (1)
rules:
  - verbs: (2)
      - get
    apiGroups:
      - tempo.grafana.com
    resources:
      - tenantA (3)
      - tenantB
      - tenantC
    resourceNames:
      - traces
1Name of the ClusterRole
2Verbs for the ClusterRole.
  • apiGroups: tempo.grafana.com

  • resources:

    • List of Tenants

  • resourceNames:

    • traces

  • verbs:

    • get

3List of tenants that are allowed to read the traces.

Bind Reader Role to Authenticated Users:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tempostack-traces-reader
subjects:
  - kind: Group
    apiGroup: rbac.authorization.k8s.io
    name: 'system:authenticated' (1)
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: tempostack-traces-reader
1The Group that is allowed to read the traces. In this example: system:authenticated. This means ALL authenticated users will be able to read all the traces (see warning below).
With this ClusterRoleBinding, anybody who is authenticated (system:authenticated) will be able to see the traces for the defined tenants (A, B, C). This is for an easy showcase in this article. For production environments, you should implement more granular RBAC controls per tenant.

Traces Writer Role:

This ClusterRole is used to write traces into TempoStack. Typically you will use this ClusterRole for the OpenTelemetry Collector, so it can write into TempoStack.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: tempostack-traces-write (1)
rules:
  - verbs:
      - create (2)
    apiGroups:
      - tempo.grafana.com
    resources: (3)
      - tenantB
      - tenantA
      - tenantC
    resourceNames:
      - traces
1Name of the ClusterRoleBinding
2This time the verb is create. This means the user will be able to write new traces into TempoStack.
3List of tenants.

Bind Writer Role to Central Collector:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tempostack-traces
subjects:
  - kind: ServiceAccount
    name: otel-collector (1)
    namespace: tempostack (2)
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: tempostack-traces-write
1The ServiceAccount that is allowed to write the traces. In this example: otel-collector. We will create this Service Account in the next article.
2The namespace of the ServiceAccount. In this example: tempostack.

GitOps Deployment

While the above is good for quick tests, it always makes sense to have a proper GitOps deployment. I have created a Chart and GitOps configuration that will:

  • Deploy the Tempo Operator

  • Configure the TempoStack instance

  • Configure the RBAC configurations for the TempoStack instance

The following sources will be used:

  • Helm Repository - to fetch the Helm Chart for the Tempo Operator including required Sub-Charts.

  • Setup Tempo Operator - To deploy and configure the Tempo Operator, configure object storage, magically create a Secret with the required keys, etc.

Feel free to clone or use whatever you need.

The following Sub-Charts are used:

The this Argo CD Application we can deploy the Tempo Operator:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: setup-tempo-operator
  namespace: openshift-gitops
spec:
  destination:
    name: in-cluster (1)
    namespace: openshift-tempo-operator (2)
  info:
    - name: Description
      value: ApplicationSet that Deploys on Management Cluster Configuration (using Git Generator)
  project: in-cluster (3)
  source:
    path: clusters/management-cluster/setup-tempo-operator (4)
    repoURL: 'https://github.com/tjungbauer/openshift-clusterconfig-gitops' (5)
    targetRevision: main
  syncPolicy:
    retry:
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
      limit: 5
1Target cluster, here the local cluster
2Namespace of the target cluster, here the Operator will be installed.
3Project of the target cluster
4Path to the Git repository
5URL to the Git repository
Since many things are moving in the background, it will take a while until the Argo CD Application is synced (i.e. Creation of S3 Bucket)

This will create the Argo CD Application that can be synchronized with the cluster:

Tempo Deployment via Argo CD

As seen above, many resources are created using a single Helm Chart (ok, with some Sub-Charts). The actual configuration is done in the values.yaml file. The full values file is quite long, so I will break it down to the different Sub-Charts and add the complete file at the end that latest file I am using for testing can be found at values.yaml.

Values File Snippets for GitOps

TempStack Settings

The following will configure the TempoStack resource. Verify the upstream Helm Chart tempo-tracing for detailed and additional information about the settings.

tempo-tracing:
  tempostack: (1)
    enabled: true
    name: simplest

    managementState: Managed

    namespace: (2)
      create: true
      name: tempostack
      descr: "Namespace for the TempoStack"
      display: "TempoStack"
      additionalAnnotations: {}
      additionalLabels: {}

    storage: (3)
      secret:
        name: tempo-s3
        type: s3
        credentialMode: static

    storageSize: 500Gi
    replicationFactor: 1
    serviceAccount: tempo-simplest (4)

    tenants: (5)
      mode: openshift
      enabled: true

      authentication: (6)
        - tenantName: 'tenantA'
          tenantId: '1610b0c3-c509-4592-a256-a1871353dbfc'
          permissions: (7)
            - write
            - read

        - tenantName: 'tenantB'
          tenantId: '1610b0c3-c509-4592-a256-a1871353dbfd'
          permissions:
            - write
            - read

    observability:
      enabled: true

      tracing:
        jaeger_agent_endpoint: 'localhost:6831'
        otlp_http_endpoint: 'http://localhost:4320'

    template:
      gateway:
        enabled: true
        rbac: false

        ingress:
          type: 'route'
          termination: 'reencrypt'
        component:
          replicas: 1

      queryFrontend:
        jaegerQuery:
          enabled: true
        component:
          replicas: 1
1Basic settings, like name of the instance
2Namespace for the TempoStack instance.
3Storage configuration for the TempoStack instance. Here we use type s3 and the Secret called "tempo-s3" which will be generated.
4ServiceAccount for the TempoStack instance.
5Tenant configuration for the TempoStack instance.
6List of tenants. This list must be extended when new tenants shall be configured.
7RBAC permissions for this tenant. This will add the tenant as resource to the READ and/or WRITE ClusterRole

Tempo Operator Deployment

The following settings will deploy the Operator and verify the status of the Operator installation. Only when the Operator has been installed successfully will Argo CD continue with the synchronization.

helper-operator:
  operators:
    tempo-operator: (1)
      enabled: true
      namespace: (2)
        name: openshift-tempo-operator
        create: true
      subscription: (3)
        channel: stable
        approval: Automatic
        operatorName: tempo-product
        source: redhat-operators
        sourceNamespace: openshift-marketplace
      operatorgroup:
        create: true
        notownnamespace: true

########################################
# SUBCHART: helper-status-checker
# Verify the status of a given operator.
########################################
helper-status-checker:
  enabled: true

  approver: false (4)

  checks:
    - operatorName: tempo-operator (5)
      namespace:
        name: openshift-tempo-operator
      syncwave: 1

      serviceAccount:
        name: "status-checker-tempo"
1Install the Operator
2Namespace settings of the Operator
3Settings of the Operator itself, like name, channel and approval strategy.
4Approver settings for the status checker. Here disabled, because we are using automatic approval.
5Operator name to be verified.

S3 Bucket Deployment

The following Sub-Chart can be used to automatically create a Bucket in OpenShift Data Foundation.

########################################
# SUBCHART: helper-objectstore

# A helper chart that simply creates another backingstore for logging.
# This is a chart in a very early state, and not everything can be customized for now.
# It will create the objects:
#  - BackingStore
#  - BackingClass
#  - StorageClass

# NOTE: Currently only PV type is supported
########################################
helper-objectstore:

  # -- Enable objectstore configuration
  # @default -- false
  enabled: true

  # -- Syncwave for Argo CD
  # @default - 1
  syncwave: 1

  # -- Name of the BackingStore
  backingstore_name: tempo-backingstore

  # -- Size of the BackingStore that each volume shall have.
  backingstore_size: 400Gi (1)

  # -- CPU Limit for the Noobaa Pod
  # @default -- 500m
  limits_cpu: 500m

  # -- Memory Limit for the Noobaa Pod.
  # @default -- 2Gi
  limits_memory: 2Gi

  pvPool:
    # -- Number of volumes that shall be used
    # @default -- 1
    numOfVolumes: 1

    # Type of BackingStore. Currently pv-pool is the only one supported by this Helm Chart.
    # @default -- pv-pool
    type: pv-pool

  # -- The StorageClass the BackingStore is based on
  baseStorageClass: gp3-csi (2)

  # -- Name of the StorageClass that shall be created for the bucket.
  storageclass_name: tempo-bucket-storage-class (3)

  # Bucket that shall be created
  bucket:
    # -- Shall a new bucket be enabled?
    # @default -- false
    enabled: true

    # -- Name of the bucket that shall be created
    name: tempo-bucket (4)

    # -- Target Namespace for that bucket.
    namespace: tempostack (5)

    # -- Syncwave for bucketclaim creation. This should be done very early, but it depends on ODF.
    # @default -- 2
    syncwave: 2

    # -- Name of the storageclass for our bucket
    # @default -- openshift-storage.noobaa.io
    storageclass: tempo-bucket-storage-class (6)
1Size of the bucket.
2StorageClass the bucket is based on.
3Name of the StorageClass that shall be created for the bucket.
4Name of the bucket that shall be created.
5Namespace for the bucket.
6StorageClass for the bucket.

Automatic Secret Creation for TempoStack

TempoStack is expecting a Secret with the following keys:

  • access_key_id

  • access_key_secret

  • bucket

  • endpoint

  • region (only for specific settings)

OpenShift Data Foundation creates a secret with access_key_id and access_key_secret and a ConfigMap with endpoint, region and the name of the bucket. Unfortunately, this does not work for TempoStack. Therefore, we are using a helper-odf-bucket-secret chart that will create a new Secret with the required keys. This chart creates a Job that reads the required information and creates a new Secret with the required keys.

##############################################
# SUBCHART: helper-odf-bucket-secret
# Creates a Secret that Tempo requires
#
# A Kubernetes Job is created, that reads the
# data from the Secret and ConfigMap and
# creates a new secret for Tempo.
##############################################
helper-odf-bucket-secret:
  # -- Enable Job to create a Secret for TempoStack.
  # @default -- false
  enabled: true

  # -- Syncwave for Argo CD.
  # @default -- 3
  syncwave: 3

  # -- Namespace where TempoStack is deployed and where the Secret shall be created.
  namespace: tempostack

  # -- Name of Secret that shall be created.
  secretname: tempo-s3 (1)

  # Bucket Configuration
  bucket:
    # -- Name of the Bucket shall has been created.
    name: tempo-bucket (2)

  # -- Keys that shall be used to create the Secret.
  keys: (3)
    # -- Overwrite access_key_id key.
    # @default -- access_key_id
    access_key_id: access_key_id
    # -- Overwrite access_key_secret key.
    # @default -- access_key_secret
    access_key_secret: access_key_secret
    # -- Overwrite bucket key.
    # @default -- bucket
    bucket: bucket
    # -- Overwrite endpoint key.
    # @default -- endpoint
    endpoint: endpoint
    # -- Overwrite region key. Region is only set if set_region is true.
    # @default -- region
    region: region

  # -- Set region key.
  # @default -- false
  set_region: false (4)
1Name of the Secret that shall be created.
2Name of the bucket that shall be used.
3Keys that shall be used to create the Secret.
4Set region key. Here disabled, because we are not using a specific region.
Using OpenShift Data Foundation with TempoStack requires you to NOT set a region. Therefore, it is disabled above.

Complete values file

To see the whole file expand the code:

The TempoStack

Let’s imagine all of the above works and we have a TempoStack instance running in the namespace tempostack with the name simplest. Several Pods are running, like the distributor, ingester, gateway, query-frontend, compactor and querier. I admit it is not highly available, but this can be easily changed in the values file above (replica count).

oc get pods -n tempostack | grep simplest
tempo-simplest-compactor-584689c78f-t7pxb           1/1     Running     0              3d2h
tempo-simplest-distributor-6fb5d7dc9d-wrzt4         1/1     Running     0              3d2h
tempo-simplest-gateway-bbcb774b9-p44lq              2/2     Running     0              11h
tempo-simplest-ingester-0                           1/1     Running     0              3d2h
tempo-simplest-querier-6cf9d7b6d8-mvc9d             1/1     Running     0              3d2h
tempo-simplest-query-frontend-7d859f9f9f-xzj97      3/3     Running     0              3d2h

Now we can continue with the OpenTelemetry Collector deployment. But before we do this, let’s first discuss how to:

  • Add Tracing UI to OpenShift


Extend the OpenShift UI

While we have Tempo installed, it will not be visible in the OpenShift UI by default. Here a separate Operator has been created by Red Hat, that will take care of this extension: Cluster Observability Operator.

This Operator be installed:

Cluster Observability Operator
The Operator can be deployed with the Chart helper-operator as well. However, I did not merge this together with the TempoStack deployment, because this Operator also serves different purposes.

We only require a very small bit of this Operator. Amongst other resources, it also provides a resource called UIPlugin.

The resource must be configured as follows:

apiVersion: observability.openshift.io/v1alpha1
kind: UIPlugincd
metadata:
  name: distributed-tracing (1)
spec:
  type: DistributedTracing (2)
1The name MUST be distributed-tracing.
2The type MUST be DistributedTracing.

This will extend the OpenShift UI with a new navigation link "Observe" > "Traces".

Tempo UI

On the next Episode

The next article will cover the deployment of the Central OpenTelemetry Collector. We will configure the RBAC permissions required for the Central Collector to enrich traces with Kubernetes metadata and deploy the Central OpenTelemetry Collector with its complete configuration.