Multi-Tenant Observability with Alloy

Route metrics, logs, and traces to isolated tenants using Grafana Alloy.

Introduction

30 Second Summary

Think about how a single apartment building serves multiple families. Each family gets their own locked unit with separate mailboxes and no way to peek into the neighbor's space. The same principle applies when multiple engineering teams share one observability platform.

In this project, you will build a multi-tenant observability platform that collects metrics, logs, and traces from two simulated teams and proves that each team's data stays completely isolated at the storage level.

What You'll Build

A complete local observability stack using Docker Compose with Grafana Alloy as the telemetry router, Mimir for metrics, Loki for logs, Tempo for traces, and Grafana for visualization.

By the end of this project, you'll have:

  • A running Docker Compose stack with 7 services (Alloy, Mimir, Loki, Tempo, Grafana, plus 2 Python apps) all configured for multi-tenancy.
  • Two Python microservices instrumented with OpenTelemetry, emitting metrics, logs, and traces to separate tenants via the X-Scope-OrgID header.
  • Confirmed tenant isolation by querying each tenant's data independently in Grafana and verifying cross-tenant queries return empty results.
  • Secret Mission: Enable cross-tenant querying for a platform admin unified view using pipe-separated tenant headers.

Are there any prerequisites?

You'll need Docker Desktop installed on your machine along with Python 3.10 or higher. You should be comfortable with basic terminal commands and Docker Compose fundamentals (defining services in a YAML file).

Set Up the Project Structure and Docker Compose

You have a project folder with the directory structure ready. Now you need to define how all seven services in your observability platform connect and communicate.

The Docker Compose file is the blueprint for your entire stack. It tells Docker which container images to pull, which ports to expose, and how services depend on each other.

In this step, get ready to:
  • Write a Docker Compose file defining all 7 services.
  • Pull all container images from Docker Hub.
  • Validate the configuration with docker compose config.
Confirm your project folder structure

Your project folder from the previous step already contains the directories each service needs for its configuration and application code.

  • Confirm the folders exist by running:
ls

You should see config and apps listed in the output.

Write the Docker Compose file

The compose file defines three architectural layers: storage backends that hold telemetry data, a collection pipeline that routes data, and application services that generate telemetry.

  • Create a new file called docker-compose.yaml in your project root and add the storage backend services by pasting the following code:
services:
  mimir:
    image: grafana/mimir:2.15.0
    command: ["--config.file=/etc/mimir/mimir.yaml"]
    volumes:
      - ./config/mimir.yaml:/etc/mimir/mimir.yaml
      - mimir-data:/tmp/mimir
    ports:
      - "9009:9009"

  loki:
    image: grafana/loki:3.7.3
    command: ["-config.file=/etc/loki/loki.yaml"]
    volumes:
      - ./config/loki.yaml:/etc/loki/loki.yaml
      - loki-data:/loki
    ports:
      - "3100:3100"

  tempo:
    image: grafana/tempo:2.7.2
    command: ["-target=all", "-config.file=/etc/tempo/tempo.yaml"]
    volumes:
      - ./config/tempo.yaml:/etc/tempo/tempo.yaml
      - tempo-data:/var/tempo
    ports:
      - "3200:3200"
      - "4320:4317"
      - "4321:4318"

What does this code do?

These three services are your telemetry storage backends. Each one handles a different signal type:

  • Mimir stores metrics (time-series data like request counts and durations). It runs on port 9009.
  • Loki stores logs (structured text entries from your applications). It runs on port 3100.
  • Tempo stores traces (end-to-end request journeys). It runs on port 3200 for HTTP and maps ports 4320 and 4321 to internal OTLP ports.

Each service mounts a config file from the config/ folder (you will write these configs in the next step) and a named volume for persistent data storage.

Tempo uses grafana/tempo:2.7.2 (the 2.x line) because Tempo 3.0 requires Kafka as a dependency. The 2.x line supports the same multi-tenancy feature without the extra infrastructure.

  • Below the tempo service in docker-compose.yaml, add the Alloy pipeline and Grafana visualization services:
  alloy:
    image: grafana/alloy:v1.17.1
    command:
      - run
      - /etc/alloy/config.alloy
      - --storage.path=/var/lib/alloy/data
      - --server.http.listen-addr=0.0.0.0:12345
    volumes:
      - ./config/config.alloy:/etc/alloy/config.alloy
    ports:
      - "12345:12345"
      - "4317:4317"
    depends_on:
      - mimir
      - loki
      - tempo

  grafana:
    image: grafana/grafana:13.1.0
    volumes:
      - ./config/grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - GF_AUTH_DISABLE_LOGIN_FORM=true
    ports:
      - "3000:3000"
    depends_on:
      - mimir
      - loki
      - tempo

What does this code do?

Grafana Alloy is the telemetry collector. It receives OpenTelemetry data from your Python apps on port 4317 and routes it to the correct backend with the right tenant header. Port 12345 exposes Alloy's web UI for pipeline monitoring.

Grafana is the visualization layer on port 3000. The environment variables disable login so you can access dashboards immediately. The depends_on fields ensure backends start before the services that need them.

  • Below the grafana service, add the two Python application services and the named volumes section:
  team-frontend:
    build:
      context: ./apps/frontend
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://alloy:4317
      - OTEL_SERVICE_NAME=frontend-api
      - OTEL_RESOURCE_ATTRIBUTES=service.namespace=team-frontend
    depends_on:
      - alloy

  team-backend:
    build:
      context: ./apps/backend
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://alloy:4317
      - OTEL_SERVICE_NAME=batch-processor
      - OTEL_RESOURCE_ATTRIBUTES=service.namespace=team-backend
    depends_on:
      - alloy

volumes:
  mimir-data:
  loki-data:
  tempo-data:

What does this code do?

These two services are your instrumented applications. Each one builds from a local Dockerfile in the apps/ directory (you will write these in Step 4).

The critical environment variable is OTEL_RESOURCE_ATTRIBUTES=service.namespace=team-frontend (or team-backend). This resource attribute is what Alloy uses to determine which tenant each application's telemetry belongs to.

The volumes section at the bottom declares named volumes so telemetry data persists across container restarts.

  • Save docker-compose.yaml.

Why service.namespace for tenant routing?

In production, teams often tag their telemetry with a service.namespace resource attribute to group related services. Alloy can read this attribute and use it to set the X-Scope-OrgID header, which is how Mimir, Loki, and Tempo identify which tenant owns the data.

✔️ Awesome, I've got everything!

Great. Double check you have saved docker-compose.yaml before moving on.

ⓧ I'd like to double check the full code

Here is the complete docker-compose.yaml file for comparison:

services:
  mimir:
    image: grafana/mimir:2.15.0
    command: ["--config.file=/etc/mimir/mimir.yaml"]
    volumes:
      - ./config/mimir.yaml:/etc/mimir/mimir.yaml
      - mimir-data:/tmp/mimir
    ports:
      - "9009:9009"

  loki:
    image: grafana/loki:3.7.3
    command: ["-config.file=/etc/loki/loki.yaml"]
    volumes:
      - ./config/loki.yaml:/etc/loki/loki.yaml
      - loki-data:/loki
    ports:
      - "3100:3100"

  tempo:
    image: grafana/tempo:2.7.2
    command: ["-target=all", "-config.file=/etc/tempo/tempo.yaml"]
    volumes:
      - ./config/tempo.yaml:/etc/tempo/tempo.yaml
      - tempo-data:/var/tempo
    ports:
      - "3200:3200"
      - "4320:4317"
      - "4321:4318"

  alloy:
    image: grafana/alloy:v1.17.1
    command:
      - run
      - /etc/alloy/config.alloy
      - --storage.path=/var/lib/alloy/data
      - --server.http.listen-addr=0.0.0.0:12345
    volumes:
      - ./config/config.alloy:/etc/alloy/config.alloy
    ports:
      - "12345:12345"
      - "4317:4317"
    depends_on:
      - mimir
      - loki
      - tempo

  grafana:
    image: grafana/grafana:13.1.0
    volumes:
      - ./config/grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - GF_AUTH_DISABLE_LOGIN_FORM=true
    ports:
      - "3000:3000"
    depends_on:
      - mimir
      - loki
      - tempo

  team-frontend:
    build:
      context: ./apps/frontend
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://alloy:4317
      - OTEL_SERVICE_NAME=frontend-api
      - OTEL_RESOURCE_ATTRIBUTES=service.namespace=team-frontend
    depends_on:
      - alloy

  team-backend:
    build:
      context: ./apps/backend
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://alloy:4317
      - OTEL_SERVICE_NAME=batch-processor
      - OTEL_RESOURCE_ATTRIBUTES=service.namespace=team-backend
    depends_on:
      - alloy

volumes:
  mimir-data:
  loki-data:
  tempo-data:
Validate and pull the container images

Before starting any containers, you need to confirm the compose file is syntactically valid and that Docker can access all the required images.

  • Validate your Docker Compose configuration by running:
docker compose config -q

What does this command do?

The docker compose config command parses, resolves, and validates your compose file. The -q (quiet) flag tells it to only validate without printing the resolved output. If nothing prints and the command exits cleanly, your file is valid.

If the command completes without any output, your configuration is valid.

Seeing a YAML syntax error?

  • Check that indentation uses spaces, not tabs. YAML requires consistent spacing (2 spaces per level in this file).
  • Verify that every ports value is wrapped in quotes (e.g., "9009:9009" not 9009:9009).
  • Make sure the volumes: section at the bottom is at the same indent level as services: (no indent).

Help me fix a YAML syntax error in my docker-compose.yaml

  • Pull all pre-built images (skipping the two services that build from local Dockerfiles) by running:
docker compose pull --ignore-buildable

What does this command do?

This downloads the five pre-built container images (Mimir, Loki, Tempo, Alloy, Grafana) from Docker Hub to your local machine. The --ignore-buildable flag skips team-frontend and team-backend since those use build: instead of a remote image.

You should see download progress for each image. This may take a few minutes depending on your internet connection.

Pull failing or timing out?

  • Make sure Docker Desktop is running. You can check by looking for the Docker whale icon in your system tray.
  • Verify you have internet connectivity. Docker pulls images from Docker Hub by default.
  • If you see a rate-limit error, wait a few minutes and try again. Docker Hub limits anonymous pulls.

Help me troubleshoot Docker image pull failures

  • Confirm all images are available locally by running:
docker compose config --images

You should see the five image names listed: grafana/mimir:2.15.0, grafana/loki:3.7.3, grafana/tempo:2.7.2, grafana/alloy:v1.17.1, and grafana/grafana:13.1.0.

Your Docker Compose blueprint is complete and validated. Next up, you will write the backend configuration files that enable multi-tenancy in Mimir, Loki, and Tempo.

Configure Multi-Tenant Backends

Your Docker Compose stack defines seven services, but each backend needs to know how to handle multiple tenants.

In this step, you will write the configuration files for Mimir, Loki, and Tempo. Each one has a single flag that activates multi-tenancy, forcing every incoming request to carry a tenant identifier in the X-Scope-OrgID header.

In this step, get ready to:
  • Write the Mimir configuration with multi-tenancy enabled and filesystem storage.
  • Write the Loki configuration with authentication enabled and local storage.
  • Write the Tempo configuration with multi-tenancy enabled and OTLP receivers.
Write the Mimir configuration

Mimir is the metrics backend. Setting multitenancy_enabled: true tells Mimir to require a tenant ID on every write and query request.

  • Create a new file called config/mimir.yaml inside your multi-tenant-observability project folder, and paste the following configuration:
multitenancy_enabled: true

blocks_storage:
  backend: filesystem
  bucket_store:
    sync_dir: /tmp/mimir/tsdb-sync
  filesystem:
    dir: /tmp/mimir/data/tsdb
  tsdb:
    dir: /tmp/mimir/tsdb

compactor:
  data_dir: /tmp/mimir/compactor
  sharding_ring:
    kvstore:
      store: memberlist

distributor:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: memberlist

ingester:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: memberlist
    replication_factor: 1

ruler_storage:
  backend: filesystem
  filesystem:
    dir: /tmp/mimir/rules

server:
  http_listen_port: 9009
  log_level: warn

store_gateway:
  sharding_ring:
    replication_factor: 1

What does this configuration do?

The top-level multitenancy_enabled: true flag enforces that every request to Mimir must include the X-Scope-OrgID HTTP header. Without it, Mimir rejects the request.

The blocks_storage section configures Mimir to store time-series data on the local filesystem. In production you would use object storage like S3, but filesystem is perfect for local development.

The replication_factor: 1 setting is appropriate because you are running a single Mimir instance. The memberlist kvstore handles internal coordination within the process.

  • Save config/mimir.yaml.
  • Verify the file was created by running:
cat config/mimir.yaml | head -1

You should see multitenancy_enabled: true printed in your terminal.

Don't see the expected output?

  • Make sure you saved the file inside the config/ folder, not the project root.
  • Check that your terminal is in the project root directory where docker-compose.yaml lives.
  • help me troubleshoot my mimir.yaml file location
Write the Loki configuration

Loki is the logs backend. Its equivalent of Mimir's multi-tenancy flag is auth_enabled: true. When set, Loki requires the X-Scope-OrgID header on every API request.

  • Create a new file called config/loki.yaml and paste the following configuration:
auth_enabled: true

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: "2020-10-24"
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

What does this configuration do?

The auth_enabled: true flag enables multi-tenancy. Without the X-Scope-OrgID header, Loki rejects the request entirely.

The common block sets shared defaults for all Loki components. It uses the local filesystem for chunk and rule storage. The schema_config section defines how Loki indexes log data, using the tsdb store with schema v13.

  • Save config/loki.yaml.
  • Verify by running:
cat config/loki.yaml | head -1

You should see auth_enabled: true printed.

Seeing a "No such file" error?

  • Double check the filename is loki.yaml (not loki.yml). The Docker Compose volume mount expects .yaml.
  • Confirm you are in the project root and the config/ folder already exists from Step 1.
  • help me find my loki.yaml file
Write the Tempo configuration

Tempo is the traces backend. Like Mimir, it uses multitenancy_enabled: true. Tempo also needs to define its own OTLP receiver endpoints so it can accept trace data directly.

  • Create a new file called config/tempo.yaml and paste the following configuration:
multitenancy_enabled: true

server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: "0.0.0.0:4317"
        http:
          endpoint: "0.0.0.0:4318"

storage:
  trace:
    backend: local
    local:
      path: /var/tempo/traces
    wal:
      path: /var/tempo/wal

ingester:
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1

compactor:
  ring:
    kvstore:
      store: inmemory

What does this configuration do?

The multitenancy_enabled: true flag works identically to Mimir. Every trace push must include the X-Scope-OrgID header.

The distributor.receivers.otlp section opens gRPC on port 4317 and HTTP on port 4318 inside the Tempo container. These are the standard OTLP ports that Alloy will forward traces to.

The storage.trace section stores traces and write-ahead logs on the local filesystem.

  • Save config/tempo.yaml.
  • Verify all three config files exist by running:
ls config/

You should see mimir.yaml, loki.yaml, and tempo.yaml listed in the output.

Missing one of the files?

  • Make sure you created all three files inside the config/ directory, not a subdirectory like config/mimir/.
  • Check that you used the .yaml extension for all three files. The Docker Compose mounts reference .yaml specifically.
  • help me verify my config files are in the right place

Why does each backend have a different flag name?

Mimir and Tempo use multitenancy_enabled: true while Loki uses auth_enabled: true. The behavior is identical. All three require the X-Scope-OrgID header on every request. The naming difference is historical since each tool was originally developed as a separate project.

✔️ Awesome, I've got everything!

Great. Double check you have saved all three files before moving on.

ⓧ I'd like to double check the full code

Here is the full content of each config file for comparison.

config/mimir.yaml:

multitenancy_enabled: true

blocks_storage:
  backend: filesystem
  bucket_store:
    sync_dir: /tmp/mimir/tsdb-sync
  filesystem:
    dir: /tmp/mimir/data/tsdb
  tsdb:
    dir: /tmp/mimir/tsdb

compactor:
  data_dir: /tmp/mimir/compactor
  sharding_ring:
    kvstore:
      store: memberlist

distributor:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: memberlist

ingester:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: memberlist
    replication_factor: 1

ruler_storage:
  backend: filesystem
  filesystem:
    dir: /tmp/mimir/rules

server:
  http_listen_port: 9009
  log_level: warn

store_gateway:
  sharding_ring:
    replication_factor: 1

config/loki.yaml:

auth_enabled: true

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: "2020-10-24"
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

config/tempo.yaml:

multitenancy_enabled: true

server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: "0.0.0.0:4317"
        http:
          endpoint: "0.0.0.0:4318"

storage:
  trace:
    backend: local
    local:
      path: /var/tempo/traces
    wal:
      path: /var/tempo/wal

ingester:
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1

compactor:
  ring:
    kvstore:
      store: inmemory

All three backends are now configured to enforce tenant isolation. Next up, you will configure Grafana Alloy to route telemetry from each Python app to the correct tenant.

Configure Alloy Per-Tenant Pipelines

Your backends (Mimir, Loki, Tempo) are now configured to enforce multi-tenancy. They will reject any request that does not include an X-Scope-OrgID header identifying the tenant.

Now you need the routing layer that sits between your apps and the backends. Grafana Alloy receives all telemetry via OTLP, batches it, and forwards it to the correct backend with the right tenant header attached.

In this step, get ready to:
  • Write the Alloy pipeline config with an OTLP receiver, batch processor, and per-tenant exporters.
  • Create per-tenant Grafana datasources that pass the correct X-Scope-OrgID header for querying.
  • Verify all config files are in place before launching the stack.
Configure the OTLP receiver and batch processor

Alloy uses a declarative pipeline config where components wire together. You need a receiver to accept incoming telemetry and a batch processor to group data before forwarding it to the backends.

  • Create a new file called config.alloy inside your config/ folder and paste the following code:
otelcol.receiver.otlp "default" {
  grpc {
    endpoint = "0.0.0.0:4317"
  }
  http {
    endpoint = "0.0.0.0:4318"
  }

  output {
    metrics = [otelcol.processor.batch.default.input]
    logs    = [otelcol.processor.batch.default.input]
    traces  = [otelcol.processor.batch.default.input]
  }
}

otelcol.processor.batch "default" {
  output {
    metrics = [otelcol.exporter.otlphttp.mimir_frontend.input, otelcol.exporter.otlphttp.mimir_backend.input]
    logs    = [otelcol.exporter.otlphttp.loki_frontend.input, otelcol.exporter.otlphttp.loki_backend.input]
    traces  = [otelcol.exporter.otlp.tempo_frontend.input, otelcol.exporter.otlp.tempo_backend.input]
  }
}

What does this code do?

The otelcol.receiver.otlp block opens gRPC (port 4317) and HTTP (port 4318) endpoints that both Python apps will send their telemetry to.

The output section routes all incoming metrics, logs, and traces into the batch processor. The batch processor then fans out each signal type to two exporters: one for team-frontend and one for team-backend. You will define those exporters next.

Add per-tenant exporters with X-Scope-OrgID headers

Each exporter targets one backend service and injects the X-Scope-OrgID header that tells the backend which tenant owns this data. Mimir and Loki use HTTP endpoints, while Tempo uses gRPC.

  • Add the following exporter blocks below the batch processor in config/config.alloy:
otelcol.exporter.otlphttp "mimir_frontend" {
  client {
    endpoint = "http://mimir:9009/otlp"
    headers  = {"X-Scope-OrgID" = "team-frontend"}
  }
}

otelcol.exporter.otlphttp "mimir_backend" {
  client {
    endpoint = "http://mimir:9009/otlp"
    headers  = {"X-Scope-OrgID" = "team-backend"}
  }
}

otelcol.exporter.otlphttp "loki_frontend" {
  client {
    endpoint = "http://loki:3100/otlp"
    headers  = {"X-Scope-OrgID" = "team-frontend"}
  }
}

otelcol.exporter.otlphttp "loki_backend" {
  client {
    endpoint = "http://loki:3100/otlp"
    headers  = {"X-Scope-OrgID" = "team-backend"}
  }
}

otelcol.exporter.otlp "tempo_frontend" {
  client {
    endpoint = "tempo:4317"
    headers  = {"X-Scope-OrgID" = "team-frontend"}
    tls {
      insecure = true
    }
  }
}

otelcol.exporter.otlp "tempo_backend" {
  client {
    endpoint = "tempo:4317"
    headers  = {"X-Scope-OrgID" = "team-backend"}
    tls {
      insecure = true
    }
  }
}

What does this code do?

  • mimir_frontend and mimir_backend send metrics to Mimir's OTLP endpoint on port 9009, each with a different X-Scope-OrgID value.
  • loki_frontend and loki_backend send logs to Loki's OTLP endpoint on port 3100.
  • tempo_frontend and tempo_backend send traces to Tempo via gRPC on port 4317. The tls { insecure = true } block disables TLS since this is container-to-container traffic on a Docker network.

Why otlphttp for Mimir and Loki but otlp for Tempo?

Mimir and Loki expose HTTP-based OTLP ingestion endpoints (/otlp path). Tempo accepts traces over gRPC natively, so it uses the otelcol.exporter.otlp component which speaks gRPC by default.

  • Save config/config.alloy.

Seeing syntax errors later when Alloy starts?

Alloy uses // for comments, not #. If you added hash-style comments, Alloy will reject the file.

Double-check that every block label (like "default" or "mimir_frontend") is wrapped in double quotes, not single quotes.

Help me debug my Alloy config

✔️ Awesome, I've got everything!

Great. Make sure you have saved config/config.alloy before moving on.

ⓧ I'd like to double check the full code

otelcol.receiver.otlp "default" {
  grpc {
    endpoint = "0.0.0.0:4317"
  }
  http {
    endpoint = "0.0.0.0:4318"
  }

  output {
    metrics = [otelcol.processor.batch.default.input]
    logs    = [otelcol.processor.batch.default.input]
    traces  = [otelcol.processor.batch.default.input]
  }
}

otelcol.processor.batch "default" {
  output {
    metrics = [otelcol.exporter.otlphttp.mimir_frontend.input, otelcol.exporter.otlphttp.mimir_backend.input]
    logs    = [otelcol.exporter.otlphttp.loki_frontend.input, otelcol.exporter.otlphttp.loki_backend.input]
    traces  = [otelcol.exporter.otlp.tempo_frontend.input, otelcol.exporter.otlp.tempo_backend.input]
  }
}

otelcol.exporter.otlphttp "mimir_frontend" {
  client {
    endpoint = "http://mimir:9009/otlp"
    headers  = {"X-Scope-OrgID" = "team-frontend"}
  }
}

otelcol.exporter.otlphttp "mimir_backend" {
  client {
    endpoint = "http://mimir:9009/otlp"
    headers  = {"X-Scope-OrgID" = "team-backend"}
  }
}

otelcol.exporter.otlphttp "loki_frontend" {
  client {
    endpoint = "http://loki:3100/otlp"
    headers  = {"X-Scope-OrgID" = "team-frontend"}
  }
}

otelcol.exporter.otlphttp "loki_backend" {
  client {
    endpoint = "http://loki:3100/otlp"
    headers  = {"X-Scope-OrgID" = "team-backend"}
  }
}

otelcol.exporter.otlp "tempo_frontend" {
  client {
    endpoint = "tempo:4317"
    headers  = {"X-Scope-OrgID" = "team-frontend"}
    tls {
      insecure = true
    }
  }
}

otelcol.exporter.otlp "tempo_backend" {
  client {
    endpoint = "tempo:4317"
    headers  = {"X-Scope-OrgID" = "team-backend"}
    tls {
      insecure = true
    }
  }
}
Create the Grafana datasource provisioning file

Grafana needs to know how to query each backend on behalf of a specific tenant. You will create a provisioning file that pre-configures six datasources, each passing the correct X-Scope-OrgID header so Grafana can query tenant-specific data.

  • Create a new file called grafana-datasources.yaml inside your config/ folder and paste the following:
apiVersion: 1

datasources:
  - name: Mimir - team-frontend
    type: prometheus
    access: proxy
    url: http://mimir:9009/prometheus
    jsonData:
      httpHeaderName1: "X-Scope-OrgID"
    secureJsonData:
      httpHeaderValue1: "team-frontend"

  - name: Mimir - team-backend
    type: prometheus
    access: proxy
    url: http://mimir:9009/prometheus
    jsonData:
      httpHeaderName1: "X-Scope-OrgID"
    secureJsonData:
      httpHeaderValue1: "team-backend"

  - name: Loki - team-frontend
    type: loki
    access: proxy
    url: http://loki:3100
    jsonData:
      httpHeaderName1: "X-Scope-OrgID"
    secureJsonData:
      httpHeaderValue1: "team-frontend"

  - name: Loki - team-backend
    type: loki
    access: proxy
    url: http://loki:3100
    jsonData:
      httpHeaderName1: "X-Scope-OrgID"
    secureJsonData:
      httpHeaderValue1: "team-backend"

  - name: Tempo - team-frontend
    type: tempo
    access: proxy
    url: http://tempo:3200
    jsonData:
      httpHeaderName1: "X-Scope-OrgID"
    secureJsonData:
      httpHeaderValue1: "team-frontend"

  - name: Tempo - team-backend
    type: tempo
    access: proxy
    url: http://tempo:3200
    jsonData:
      httpHeaderName1: "X-Scope-OrgID"
    secureJsonData:
      httpHeaderValue1: "team-backend"

What does this file do?

Grafana's provisioning system reads this YAML file on startup and automatically creates the listed datasources. Each datasource points to the same backend URL but injects a different X-Scope-OrgID header value.

The jsonData.httpHeaderName1 field names the header, and secureJsonData.httpHeaderValue1 provides the tenant value. Grafana stores the value securely so it does not appear in the API.

  • Save config/grafana-datasources.yaml.
  • Confirm all your config files are in place by running:
ls config/

You should see config.alloy, grafana-datasources.yaml, loki.yaml, mimir.yaml, and tempo.yaml in the output.

Missing a file?

Make sure you created both new files inside the config/ folder, not in the project root. The Docker Compose volumes expect them at ./config/config.alloy and ./config/grafana-datasources.yaml.

Help me find my missing config file

✔️ Awesome, I've got everything!

Great. Make sure both new files are saved before moving on.

ⓧ I'd like to double check the full code

apiVersion: 1

datasources:
  - name: Mimir - team-frontend
    type: prometheus
    access: proxy
    url: http://mimir:9009/prometheus
    jsonData:
      httpHeaderName1: "X-Scope-OrgID"
    secureJsonData:
      httpHeaderValue1: "team-frontend"

  - name: Mimir - team-backend
    type: prometheus
    access: proxy
    url: http://mimir:9009/prometheus
    jsonData:
      httpHeaderName1: "X-Scope-OrgID"
    secureJsonData:
      httpHeaderValue1: "team-backend"

  - name: Loki - team-frontend
    type: loki
    access: proxy
    url: http://loki:3100
    jsonData:
      httpHeaderName1: "X-Scope-OrgID"
    secureJsonData:
      httpHeaderValue1: "team-frontend"

  - name: Loki - team-backend
    type: loki
    access: proxy
    url: http://loki:3100
    jsonData:
      httpHeaderName1: "X-Scope-OrgID"
    secureJsonData:
      httpHeaderValue1: "team-backend"

  - name: Tempo - team-frontend
    type: tempo
    access: proxy
    url: http://tempo:3200
    jsonData:
      httpHeaderName1: "X-Scope-OrgID"
    secureJsonData:
      httpHeaderValue1: "team-frontend"

  - name: Tempo - team-backend
    type: tempo
    access: proxy
    url: http://tempo:3200
    jsonData:
      httpHeaderName1: "X-Scope-OrgID"
    secureJsonData:
      httpHeaderValue1: "team-backend"

Your entire pipeline is now configured. Alloy knows where to receive telemetry, how to batch it, and where to send it with the correct tenant headers. Next up, you will build the two Python apps that actually generate the metrics, logs, and traces.

Build the Instrumented Python Apps

Your observability backends are configured and Alloy knows how to route telemetry per tenant. Now you need applications that actually produce telemetry data.

In this step, you will build two Python microservices instrumented with the OpenTelemetry SDK. Each app emits metrics, logs, and traces to Alloy over the OTLP protocol. The critical detail is the service.namespace resource attribute, which identifies which tenant the telemetry belongs to.

In this step, get ready to:
  • Write a team-frontend app that simulates HTTP request telemetry.
  • Write a team-backend app that simulates batch job telemetry.
  • Create Docker images for both apps and verify they build.
Write the team-frontend app

The frontend app simulates an HTTP API server that handles requests. It emits a request counter, a duration histogram, structured log lines, and request traces with HTTP attributes.

  • Create the apps/frontend directory (if it does not already exist) by running this command:
mkdir -p apps/frontend
  • Create a new file called app.py inside apps/frontend and paste in the following code:
import time
import logging
import random
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry._logs import set_logger_provider
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
import os

# Create an OpenTelemetry resource identifying this service and its tenant
resource = Resource.create({
    "service.name": os.environ.get("OTEL_SERVICE_NAME", "frontend-api"),
    "service.namespace": os.environ.get("OTEL_RESOURCE_ATTRIBUTES", "service.namespace=team-frontend").split("=")[-1],
})

# OTLP endpoint where Alloy listens for telemetry
endpoint = os.environ.get("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317")

# Set up trace provider with OTLP gRPC exporter
trace_provider = TracerProvider(resource=resource)
trace_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint=endpoint, insecure=True)))
trace.set_tracer_provider(trace_provider)

# Set up metric provider with periodic export every 5 seconds
metric_reader = PeriodicExportingMetricReader(OTLPMetricExporter(endpoint=endpoint, insecure=True), export_interval_millis=5000)
meter_provider = MeterProvider(resource=resource, metric_readers=[metric_reader])
metrics.set_meter_provider(meter_provider)

# Set up log provider with OTLP gRPC exporter
logger_provider = LoggerProvider(resource=resource)
logger_provider.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter(endpoint=endpoint, insecure=True)))
set_logger_provider(logger_provider)
handler = LoggingHandler(logger_provider=logger_provider)
logging.getLogger().addHandler(handler)
logging.getLogger().setLevel(logging.INFO)

# Create instruments for tracking HTTP requests
tracer = trace.get_tracer("frontend-api")
meter = metrics.get_meter("frontend-api")
request_counter = meter.create_counter("http_requests_total", description="Total HTTP requests")
request_duration = meter.create_histogram("http_request_duration_seconds", description="HTTP request duration")

logger = logging.getLogger("frontend-api")

# Simulate HTTP traffic in an infinite loop
while True:
    with tracer.start_as_current_span("handle_request") as span:
        method = random.choice(["GET", "POST", "PUT"])
        path = random.choice(["/api/users", "/api/products", "/api/orders"])
        duration = random.uniform(0.01, 0.5)
        status = random.choice([200, 200, 200, 201, 400, 500])

        span.set_attribute("http.method", method)
        span.set_attribute("http.url", path)
        span.set_attribute("http.status_code", status)

        request_counter.add(1, {"method": method, "path": path, "status": str(status)})
        request_duration.record(duration, {"method": method, "path": path})

        logger.info(f"{method} {path} - {status} ({duration:.3f}s)")
        if status == 500:
            logger.error(f"Internal server error on {method} {path}")

        time.sleep(random.uniform(1, 3))

What does this code do?

This file sets up all three telemetry signals and runs a simulation loop:

  • Resource creation attaches service.name and service.namespace to every piece of telemetry. The namespace is what Alloy will use to determine this app's tenant.
  • Three providers (trace, metric, log) each export to the same OTLP endpoint where Alloy listens.
  • The while True loop simulates random HTTP requests, recording a counter, histogram, trace span, and log line for each.
  • Status 500 responses trigger an additional error log, giving you something to filter on in Grafana later.
  • Save apps/frontend/app.py.

Seeing import errors in your editor?

Your local Python environment does not have the OpenTelemetry packages installed yet. That is expected. The dependencies will be installed inside the Docker container when you build the image in Substep 3.

If you want local editor autocomplete, you can install them locally too, but it is not required for this project.

Help me understand why my editor shows import errors for OpenTelemetry

Write the team-backend app

The backend app follows the same structure as the frontend but simulates batch processing jobs instead of HTTP requests. It uses different metric names and span attributes to represent its workload.

  • Create the apps/backend directory by running this command:
mkdir -p apps/backend
  • Create a new file called app.py inside apps/backend and paste in the following code:
import time
import logging
import random
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry._logs import set_logger_provider
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
import os

# Create an OpenTelemetry resource identifying this service and its tenant
resource = Resource.create({
    "service.name": os.environ.get("OTEL_SERVICE_NAME", "batch-processor"),
    "service.namespace": os.environ.get("OTEL_RESOURCE_ATTRIBUTES", "service.namespace=team-backend").split("=")[-1],
})

# OTLP endpoint where Alloy listens for telemetry
endpoint = os.environ.get("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317")

# Set up trace provider with OTLP gRPC exporter
trace_provider = TracerProvider(resource=resource)
trace_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint=endpoint, insecure=True)))
trace.set_tracer_provider(trace_provider)

# Set up metric provider with periodic export every 5 seconds
metric_reader = PeriodicExportingMetricReader(OTLPMetricExporter(endpoint=endpoint, insecure=True), export_interval_millis=5000)
meter_provider = MeterProvider(resource=resource, metric_readers=[metric_reader])
metrics.set_meter_provider(meter_provider)

# Set up log provider with OTLP gRPC exporter
logger_provider = LoggerProvider(resource=resource)
logger_provider.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter(endpoint=endpoint, insecure=True)))
set_logger_provider(logger_provider)
handler = LoggingHandler(logger_provider=logger_provider)
logging.getLogger().addHandler(handler)
logging.getLogger().setLevel(logging.INFO)

# Create instruments for tracking batch jobs
tracer = trace.get_tracer("batch-processor")
meter = metrics.get_meter("batch-processor")
jobs_processed = meter.create_counter("batch_jobs_processed_total", description="Total batch jobs processed")
job_duration = meter.create_histogram("batch_job_duration_seconds", description="Batch job processing duration")

logger = logging.getLogger("batch-processor")

# Simulate batch processing in an infinite loop
while True:
    with tracer.start_as_current_span("process_batch") as span:
        batch_size = random.randint(10, 100)
        job_type = random.choice(["data-sync", "report-gen", "cleanup", "notification"])
        duration = random.uniform(0.5, 5.0)
        success = random.random() > 0.1

        span.set_attribute("batch.size", batch_size)
        span.set_attribute("batch.job_type", job_type)
        span.set_attribute("batch.success", success)

        jobs_processed.add(1, {"job_type": job_type, "success": str(success)})
        job_duration.record(duration, {"job_type": job_type})

        logger.info(f"Processed batch: type={job_type}, size={batch_size}, duration={duration:.2f}s")
        if not success:
            logger.error(f"Batch job failed: type={job_type}, size={batch_size}")

        time.sleep(random.uniform(2, 5))

How does this differ from the frontend app?

The structure is identical, but the business logic is different:

  • service.namespace is set to team-backend instead of team-frontend. This is the value Alloy uses to route data to the correct tenant's storage.
  • Metrics track batch jobs (batch_jobs_processed_total, batch_job_duration_seconds) instead of HTTP requests.
  • Span attributes use batch.size and batch.job_type instead of http.method and http.url.
  • The loop runs slower (2-5 seconds between iterations) to simulate heavier batch workloads.
  • Save apps/backend/app.py.
Write the shared Dockerfile and requirements.txt

Both apps use the same two Python dependencies and an identical Dockerfile. You will create these files in both the apps/frontend and apps/backend directories.

  • Create requirements.txt inside apps/frontend with the following content:
opentelemetry-sdk==1.42.1
opentelemetry-exporter-otlp-proto-grpc==1.42.1
  • Create requirements.txt inside apps/backend with exactly the same content:
opentelemetry-sdk==1.42.1
opentelemetry-exporter-otlp-proto-grpc==1.42.1

Why pin exact versions?

The OpenTelemetry Python SDK requires all packages to share the same version. Mixing 1.42.1 with a different version causes import errors at runtime. Pinning both to 1.42.1 guarantees compatibility.

  • Create a Dockerfile inside apps/frontend with this content:
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py .
CMD ["python", "app.py"]
  • Create the same Dockerfile inside apps/backend with identical content:
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py .
CMD ["python", "app.py"]

What does this Dockerfile do?

  • FROM python:3.12-slim uses a minimal Python base image to keep the container small.
  • COPY requirements.txt and RUN pip install are separate layers, so Docker caches installed packages between rebuilds.
  • CMD ["python", "app.py"] starts the telemetry simulation when the container launches.
  • Save all four files (requirements.txt and Dockerfile in both apps/frontend and apps/backend).
  • Verify both images build successfully by running these commands:
docker compose build team-frontend
docker compose build team-backend
  • You should see output ending with lines like Successfully built or Successfully tagged for each service. Both builds download the python:3.12-slim base image and install the two OpenTelemetry packages.

Build failing with network errors?

If pip install fails inside the container, make sure Docker Desktop has internet access. Check that no VPN or firewall is blocking outbound connections from Docker.

If you see a COPY failed error, double-check that requirements.txt and app.py are both saved in the correct directory (apps/frontend or apps/backend).

Help me debug my Docker build failure for the Python apps

✔️ Awesome, I've got everything!

Great. Double check you have saved all six new files before moving on.

ⓧ I'd like to double check the full code

Here is the complete content of each new file for comparison.

apps/frontend/app.py:

import time
import logging
import random
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry._logs import set_logger_provider
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
import os

resource = Resource.create({
    "service.name": os.environ.get("OTEL_SERVICE_NAME", "frontend-api"),
    "service.namespace": os.environ.get("OTEL_RESOURCE_ATTRIBUTES", "service.namespace=team-frontend").split("=")[-1],
})

endpoint = os.environ.get("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317")

trace_provider = TracerProvider(resource=resource)
trace_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint=endpoint, insecure=True)))
trace.set_tracer_provider(trace_provider)

metric_reader = PeriodicExportingMetricReader(OTLPMetricExporter(endpoint=endpoint, insecure=True), export_interval_millis=5000)
meter_provider = MeterProvider(resource=resource, metric_readers=[metric_reader])
metrics.set_meter_provider(meter_provider)

logger_provider = LoggerProvider(resource=resource)
logger_provider.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter(endpoint=endpoint, insecure=True)))
set_logger_provider(logger_provider)
handler = LoggingHandler(logger_provider=logger_provider)
logging.getLogger().addHandler(handler)
logging.getLogger().setLevel(logging.INFO)

tracer = trace.get_tracer("frontend-api")
meter = metrics.get_meter("frontend-api")
request_counter = meter.create_counter("http_requests_total", description="Total HTTP requests")
request_duration = meter.create_histogram("http_request_duration_seconds", description="HTTP request duration")

logger = logging.getLogger("frontend-api")

while True:
    with tracer.start_as_current_span("handle_request") as span:
        method = random.choice(["GET", "POST", "PUT"])
        path = random.choice(["/api/users", "/api/products", "/api/orders"])
        duration = random.uniform(0.01, 0.5)
        status = random.choice([200, 200, 200, 201, 400, 500])

        span.set_attribute("http.method", method)
        span.set_attribute("http.url", path)
        span.set_attribute("http.status_code", status)

        request_counter.add(1, {"method": method, "path": path, "status": str(status)})
        request_duration.record(duration, {"method": method, "path": path})

        logger.info(f"{method} {path} - {status} ({duration:.3f}s)")
        if status == 500:
            logger.error(f"Internal server error on {method} {path}")

        time.sleep(random.uniform(1, 3))

apps/frontend/requirements.txt:

opentelemetry-sdk==1.42.1
opentelemetry-exporter-otlp-proto-grpc==1.42.1

apps/frontend/Dockerfile:

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py .
CMD ["python", "app.py"]

apps/backend/app.py:

import time
import logging
import random
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry._logs import set_logger_provider
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
import os

resource = Resource.create({
    "service.name": os.environ.get("OTEL_SERVICE_NAME", "batch-processor"),
    "service.namespace": os.environ.get("OTEL_RESOURCE_ATTRIBUTES", "service.namespace=team-backend").split("=")[-1],
})

endpoint = os.environ.get("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317")

trace_provider = TracerProvider(resource=resource)
trace_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint=endpoint, insecure=True)))
trace.set_tracer_provider(trace_provider)

metric_reader = PeriodicExportingMetricReader(OTLPMetricExporter(endpoint=endpoint, insecure=True), export_interval_millis=5000)
meter_provider = MeterProvider(resource=resource, metric_readers=[metric_reader])
metrics.set_meter_provider(meter_provider)

logger_provider = LoggerProvider(resource=resource)
logger_provider.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter(endpoint=endpoint, insecure=True)))
set_logger_provider(logger_provider)
handler = LoggingHandler(logger_provider=logger_provider)
logging.getLogger().addHandler(handler)
logging.getLogger().setLevel(logging.INFO)

tracer = trace.get_tracer("batch-processor")
meter = metrics.get_meter("batch-processor")
jobs_processed = meter.create_counter("batch_jobs_processed_total", description="Total batch jobs processed")
job_duration = meter.create_histogram("batch_job_duration_seconds", description="Batch job processing duration")

logger = logging.getLogger("batch-processor")

while True:
    with tracer.start_as_current_span("process_batch") as span:
        batch_size = random.randint(10, 100)
        job_type = random.choice(["data-sync", "report-gen", "cleanup", "notification"])
        duration = random.uniform(0.5, 5.0)
        success = random.random() > 0.1

        span.set_attribute("batch.size", batch_size)
        span.set_attribute("batch.job_type", job_type)
        span.set_attribute("batch.success", success)

        jobs_processed.add(1, {"job_type": job_type, "success": str(success)})
        job_duration.record(duration, {"job_type": job_type})

        logger.info(f"Processed batch: type={job_type}, size={batch_size}, duration={duration:.2f}s")
        if not success:
            logger.error(f"Batch job failed: type={job_type}, size={batch_size}")

        time.sleep(random.uniform(2, 5))

apps/backend/requirements.txt:

opentelemetry-sdk==1.42.1
opentelemetry-exporter-otlp-proto-grpc==1.42.1

apps/backend/Dockerfile:

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py .
CMD ["python", "app.py"]

Both apps are packaged and ready to run. In the next step, you will launch the entire stack and prove that telemetry from each app lands in its correct tenant's storage.

Launch the Stack and Prove Tenant Isolation

Your configuration is written and your apps are instrumented. Now it is time to bring the entire platform online and confirm that the X-Scope-OrgID header pattern actually isolates telemetry at the storage level.

You will start all 7 services with a single command, verify the pipelines are healthy in the Alloy web UI, query each tenant's data in Grafana, and prove that one tenant cannot see another tenant's telemetry.

In this step, get ready to:
  • Launch all 7 Docker Compose services and confirm they are healthy.
  • Explore each tenant's metrics, logs, and traces in Grafana.
  • Prove tenant isolation by confirming cross-tenant queries return empty results.
Start the stack and verify healthy services

Docker Compose will build your Python app images, create the internal network, and start all 7 services in the correct dependency order.

  • Launch the full stack in detached mode by running this command:
docker compose up -d --build

What does this command do?

The up -d flag starts all services in the background. The --build flag forces Docker to rebuild the Python app images from your Dockerfiles, picking up any changes.

Docker Compose will start mimir, loki, and tempo first (they have no dependencies), then alloy and grafana (which depend on the backends), and finally team-frontend and team-backend (which depend on alloy).

  • Wait about 30 seconds for all services to initialize, then check their status by running:
docker compose ps

You should see all 7 services listed with a status of "Up" or "Running".

Services not all running?

  • If a service shows "Exited", check its logs with docker compose logs <service-name> to see the error. A common cause is a YAML syntax error in the config file for that service.
  • If you see port conflicts (e.g. port 3000 already in use), stop the conflicting process or change the host port mapping in docker-compose.yaml.
  • help me debug my Docker Compose services not starting
  • Verify Alloy's pipelines are active by opening your browser and navigating to http://localhost:12345.
  • Click the Graph tab at the top of the Alloy UI.

You should see a visual graph of your OTLP receiver, batch processor, and all 6 exporter components connected together. All components should show a healthy status.

What is the Alloy web UI?

Alloy ships with a built-in web interface for monitoring pipeline health. The Graph page shows a visual representation of how your components are wired together. Clicking any component opens its detail page where you can inspect its arguments, health, and exports.

Explore each tenant's data in Grafana

Your Python apps have been running for at least 30 seconds now, which means they have already emitted metrics, logs, and traces. Grafana's pre-provisioned datasources let you query each tenant's data immediately.

  • Open Grafana by navigating to http://localhost:3000 in your browser.
  • Click Explore in the left side menu.
  • Select Mimir - team-frontend from the data source dropdown in the upper left.
  • In the query editor, type http_requests_total in the metric field and click Run query.

You should see a time series graph showing the frontend app's HTTP request counter, broken down by method, path, and status labels.

  • Switch the data source to Loki - team-frontend.
  • Enter {service_name="frontend-api"} in the query editor and click Run query.

You should see log lines like GET /api/users - 200 (0.045s) appearing from the frontend app.

  • Switch the data source to Tempo - team-frontend.
  • Select the Search tab in the query editor and click Run query to find recent traces.

You should see handle_request spans from the frontend-api service.

  • Now switch to Mimir - team-backend and query batch_jobs_processed_total.
  • Switch to Loki - team-backend and query {service_name="batch-processor"}.
  • Switch to Tempo - team-backend and search for recent traces.

You should see batch processing metrics, log lines like Processed batch: type=data-sync, and process_batch spans. Each tenant's datasource shows only that tenant's data.

Why do I see six datasources?

You provisioned 6 datasources in grafana-datasources.yaml: one per signal type (metrics, logs, traces) times two tenants (team-frontend, team-backend). Each datasource sends its own X-Scope-OrgID header value, so Grafana can only see data belonging to that specific tenant.

Give the apps a minute

If queries return empty results, wait 60 seconds and retry. The Python apps emit telemetry every 1-5 seconds, but the metric export interval is 5 seconds and the batch processor needs time to flush.

Prove tenant isolation with cross-tenant queries

Seeing data in each tenant's datasource is a good start. The real proof of multi-tenancy is confirming that one tenant's datasource cannot access another tenant's data.

  • In Grafana Explore, select Mimir - team-frontend from the data source dropdown.
  • Query batch_jobs_processed_total (this is a backend-only metric) and click Run query.

The query should return no data. This metric exists in Mimir's storage, but it was written with the team-backend tenant ID. The team-frontend datasource can only read data tagged to its own tenant.

  • Switch to Loki - team-frontend and query {service_name="batch-processor"}.
  • Click Run query.

Again, the query returns empty. The backend's logs are stored under team-backend and are invisible to the team-frontend datasource.

How does isolation work at the storage level?

When Alloy sends data to Mimir, Loki, or Tempo, it includes the X-Scope-OrgID header. Each backend uses this header to store the data in a tenant-specific namespace. On read, the same header is required to access that namespace. If the header value does not match, the data is simply not visible.

This is the exact same pattern used in production Grafana Cloud deployments to isolate thousands of tenants on shared infrastructure.

Seeing data when you expected empty results?

  • Double-check that you selected the correct datasource in the dropdown. The name should say Mimir - team-frontend (not team-backend).
  • Verify your grafana-datasources.yaml has the correct httpHeaderValue1 for each datasource entry. Each tenant's datasource must use its own tenant name.
  • help me debug why cross-tenant queries are returning data

Your multi-tenant observability platform is fully operational. You have proven that two teams can share the same infrastructure without seeing each other's data. Next up, you will tackle the Secret Mission and enable a platform admin view that can query across all tenants.

Secret mission

Enable Cross-Tenant Platform Admin View

You've proven that tenant isolation works. But what about the platform team that needs to see everything? In production, SREs and platform engineers need a unified view across all tenants to debug cross-team issues and monitor overall system health.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally using Docker containers, so there are no ongoing cloud costs. However, the running containers do consume system memory and CPU while active.

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • 7 running Docker containers (mimir, loki, tempo, alloy, grafana, team-frontend, team-backend).
  • 3 Docker volumes (mimir-data, loki-data, tempo-data) storing telemetry data.
  • Docker images pulled and built for the stack (Grafana, Mimir, Loki, Tempo, Alloy, Python base, plus 2 custom app images).
  • Project files on your Desktop (config files, Python apps, Docker Compose file).

Keep everything running

No action needed. Choose this if you are still actively experimenting with queries in Grafana or want to extend the stack further.

  • Your 7 containers will continue running and consuming system resources (approximately 1-2 GB of RAM total).
  • Grafana remains accessible at localhost:3000 and Alloy at localhost:12345.
  • Docker volumes will continue storing telemetry data on disk.

Pause - I'll come back to this later

Stop all running containers to free up memory and CPU, but keep your files, volumes, and images so you can restart quickly later.

  • Open your terminal and navigate to your project folder on the Desktop (the folder containing docker-compose.yaml).
  • Stop and remove all containers by running:
docker compose down
  • Your Docker volumes and images are preserved. To restart later, run docker compose up -d from the same folder.

Delete - I don't want to use this again

Remove all containers, volumes, images, and project files completely. This frees up all disk space and system resources.

  • Open your terminal and navigate to your project folder on the Desktop (the folder containing docker-compose.yaml).
  • Stop all containers and remove volumes and images by running:
docker compose down -v --rmi all
  • This removes all 7 containers, the 3 named volumes (mimir-data, loki-data, tempo-data), and all Docker images used by the stack.
  • Delete the project folder from your Desktop. On macOS, run:
cd ~/Desktop
rm -rf $(basename $OLDPWD)
  • On Windows, run:
cd ~\Desktop
Remove-Item -Recurse -Force $env:OLDPWD
  • Alternatively, right-click the project folder on your Desktop and select Move to Trash (macOS) or Delete (Windows).

Nice Work!

Nice Work!

Nice work! You just built a production-grade multi-tenant observability platform using Grafana Alloy, Mimir, Loki, Tempo, and Grafana.

You've learned how to:

  • Built a complete multi-tenant observability stack with 7 Docker Compose services routing metrics, logs, and traces through Grafana Alloy to isolated storage backends.
  • Configured X-Scope-OrgID header-based tenant isolation across all three signal types, ensuring each team's telemetry stays separated at the storage level.
  • Created two instrumented Python microservices emitting OpenTelemetry data and proved tenant isolation by confirming cross-tenant queries return empty results.
  • Secret Mission: Enabled cross-tenant querying for a platform admin unified view using pipe-separated tenant headers.

Ready to quiz yourself?