The Hitchhiker's Guide to Observability - Adding A New Tenant - Part 6

OpenShift Observability Distributed Tracing OpenTelemetry Tempo Grafana

- Thomas Jungbauer Thomas Jungbauer ( Lastmod: 2025-11-28 ) - 3 min read

image from The Hitchhiker's Guide to Observability - Adding A New Tenant - Part 6

While we have created our distributed tracing infrastructure, we created two tenants as an example. In this article, I will show you how to add a new tenant and which changes must be made in the TempoStack and the OpenTelemetry Collector.

This article was mainly created as a quick reference guide to see which changes must be made when adding new tenants.

Quick Checklist for Adding a New Tenant

Use this checklist to ensure you haven’t missed any steps:

  • TempoStack: Add tenant to spec.tenants.authentication[]

  • ClusterRole (write): Add tenant name to tempostack-traces-write resources

  • ClusterRole (read): Add tenant name to tempostack-traces-reader resources

  • Central OTC Routing: Add routing rule in connectors.routing/traces.table[]

  • Central OTC Exporter: Add otlp/[tenant-name] exporter with correct X-Scope-OrgID

  • Central OTC Pipeline: Add traces/[tenant-name] pipeline

  • Namespace: Create namespace for the team

  • ServiceAccount: Create ServiceAccount for local collector

  • Local OTC: Deploy OpenTelemetryCollector in team namespace

  • Application: Deploy application with OTEL configuration

  • Verify: Check logs, generate traces, view in Jaeger UI

You do NOT need to create new ClusterRoles for each tenant. The k8sattributes-otel-collector ClusterRole already grants the central collector cluster-wide read access to pods, namespaces, and replicasets across all namespaces. This single ClusterRole serves all tenants.

Common Mistakes to Avoid

  1. Mismatched Tenant Names: Ensure the tenant name in TempoStack matches the X-Scope-OrgID header in the exporter

  2. Wrong Namespace Attribute: The routing rule matches on k8s.namespace.name - ensure the local collector sets this correctly

  3. Forgot RBAC Update: Without updating ClusterRoles, traces will be rejected

  4. Typo in Pipeline Names: Pipeline names in the routing connector must match the actual pipeline definitions

  5. Missing Exporter in Pipeline: Each new pipeline must reference the correct exporter (e.g., otlp/team-d)

Adding a New Tenant - team-d

Let’s add a new tenant called "team-d" with their own namespace, local collector, and route to TempoStack tenant "tenantD".

Step 1: Update TempoStack Configuration

If the tenant "tenantD" does not already exist in TempoStack, we need to add it to the tenants list.

Edit the TempoStack resource in the tempostack namespace and add the new tenant to the authentication list:

spec:
  tenants:
    mode: openshift
    authentication:
      - tenantId: 1610b0c3-c509-4592-a256-a1871353dbfc
        tenantName: tenantA
      - tenantId: 1610b0c3-c509-4592-a256-a1871353dbfd
        tenantName: tenantB
      - tenantId: 1610b0c3-c509-4592-a256-a1871353dbfe
        tenantName: tenantC
      # Add new tenant
      - tenantId: 1610b0c3-c509-4592-a256-a1871353dbff (1)
        tenantName: tenantD
1Our new tenantID with the tenantName "tenantD"

Step 2: Update RBAC Permissions - write

Update the ClusterRoles to grant write permissions for the new tenant. Edit the ClusterRole tempostack-traces-write:

oc edit clusterrole tempostack-traces-write
rules:
  - verbs:
      - create
    apiGroups:
      - tempo.grafana.com
    resources:
      - tenantA
      - tenantB
      - tenantC
      - tenantD (1)
    resourceNames:
      - traces
1Our new tenant with the tenantName "tenantD"

Step 3: Update RBAC Permissions - read

Update the ClusterRoles to grant read permissions for the new tenant. Edit the ClusterRole tempostack-traces-reader:

oc edit clusterrole tempostack-traces-reader
rules:
  - verbs:
      - get
    apiGroups:
      - tempo.grafana.com
    resources:
      - dev
      - tenantA
      - tenantB
      - tenantC
      - tenantD (1)
    resourceNames:
      - traces
1Our new tenant with the tenantName "tenantD"

Step 4: Update Central OpenTelemetry Collector

The central collector needs configuration changes to route traces from the new namespace to the appropriate TempoStack tenant. Edit the OpenTelemetry Collector in the tempostack namespace and add the new routing rule:

oc edit otelcol otel -n tempostack

Add the New Routing Rule

Add the new routing rule to the connectors section:

spec:
  config:
    connectors:
      routing/traces:
        default_pipelines:
          - traces/Default
        error_mode: ignore
        table:
          - statement: route() where attributes["k8s.namespace.name"] == "team-a"
            pipelines:
              - traces/tenantA
          - statement: route() where attributes["k8s.namespace.name"] == "team-b"
            pipelines:
              - traces/tenantB
          - statement: route() where attributes["k8s.namespace.name"] == "team-c"
            pipelines:
              - traces/tenantC
          # Add new routing rule
          - statement: route() where attributes["k8s.namespace.name"] == "team-d" (1)
            pipelines:
              - traces/tenantD
1Our new tenant with the tenantName "tenantD"

Add Exporter for the New Tenant

Add the new exporter to the exporters section:

spec:
  config:
    exporters:
      # ... existing exporters ...

      # New tenant exporter
      otlp/tenantD: (1)
        endpoint: tempo-simplest-gateway:8090
        auth:
          authenticator: bearertokenauth
        headers:
          X-Scope-OrgID: tenantD (2)
        tls:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
          insecure_skip_verify: true
          server_name_override: tempo-simplest-gateway.tempostack.svc.cluster.local
1Our new exporter with the name "otlp/tenantD"
2The X-Scope-OrgID header value that identifies the tenant

Add Pipeline for the New Tenant

Add the new pipeline to the pipelines section:

spec:
  config:
    service:
      pipelines:
        # ... existing pipelines ...

        # New tenant pipeline
        traces/tenantD: (1)
          receivers:
            - routing/traces
          exporters:
            - otlp/tenantD
1Our new tenant with the tenantName "tenantD"

Step 5: Install the Application and Local OpenTelemetry Collector

Follow the steps from the previous article to install the application and local OpenTelemetry Collector in the new namespace.

See Also:

  • The Hitchhiker's Guide to Observability Introduction - Part 1 –

    With this article I would like to summarize and, especially, remember my setup. This is Part 1 of a series of articles that I split up so it is easier to read and understand and not too long. Initially, there will be 6 parts, but I will add more as needed.

  • The Hitchhiker's Guide to Observability - Grafana Tempo - Part 2 –

    After covering the fundamentals and architecture in Part 1, it’s time to get our hands dirty! This article walks through the complete implementation of a distributed tracing infrastructure on OpenShift. We’ll deploy and configure the Tempo Operator and a multi-tenant TempoStack instance. For S3 storage we will use the integrated OpenShift Data Foundation. However, you can use whatever S3-compatible storage you have available.

  • The Hitchhiker's Guide to Observability - Central Collector - Part 3 –

    With the architecture defined in Part 1 and TempoStack deployed in Part 2, it’s time to tackle the heart of our distributed tracing system: the Central OpenTelemetry Collector. This is the critical component that sits between your application namespaces and TempoStack, orchestrating trace flow, metadata enrichment, and tenant routing.

  • The Hitchhiker's Guide to Observability - Example Applications - Part 4 –

    With the architecture defined, TempoStack deployed, and the Central Collector configured, we’re now ready to complete the distributed tracing pipeline. It’s time to deploy real applications and see traces flowing through the entire system! In this fourth installment, we’ll focus on the application layer - deploying Local OpenTelemetry Collectors in team namespaces and configuring example applications to generate traces. You’ll see how applications automatically get enriched with Kubernetes metadata, how namespace-based routing directs traces to the correct TempoStack tenants, and how the entire two-tier architecture comes together.

  • The Hitchhiker's Guide to Observability - Understanding Traces - Part 5 –

    With the architecture established, TempoStack deployed, the Central Collector configured, and applications generating traces, it’s time to take a step back and understand what we’re actually building. Before you deploy more applications and start troubleshooting performance issues, you need to understand how to read and interpret distributed traces. Let’s decode the matrix of distributed tracing!

Copyright © 2020 - 2025 Toni Schmidbauer & Thomas Jungbauer