Multi-Tenant LGTM Stack on Fargate

Deploy Mimir, Loki, Tempo, Alloy, and Grafana on ECS Fargate with tenant isolation.

Introduction

30 Second Summary

When multiple teams share a single monitoring platform, you need guarantees that one team's data never leaks into another's view. This is multi-tenancy in observability.

In this project, you will deploy a complete multi-tenant telemetry platform on AWS Fargate using Grafana's LGTM stack. Five containers will run as a single task, collecting metrics, logs, and traces from multiple tenants while keeping their data strictly isolated.

What You'll Build

A production-grade observability platform running Grafana Mimir, Grafana Loki, Grafana Tempo, Grafana Alloy, and Grafana as a multi-container ECS Fargate task with S3-backed storage and tenant isolation via the X-Scope-OrgID header.

By the end of this project, you'll have:

  • A running multi-tenant telemetry platform with Mimir (metrics), Loki (logs), and Tempo (traces) deployed on ECS Fargate.
  • S3-backed persistent storage for all three backends with automatic per-tenant prefix isolation.
  • Grafana Alloy configured as the unified OTLP collector that injects tenant headers to route data through the pipeline.
  • Verified tenant isolation by sending test telemetry as two different tenants and confirming data separation in Grafana dashboards.
  • Secret Mission: Add per-tenant rate limits and retention overrides to each backend.

Are there any prerequisites?

You need an AWS account with the AWS CLI configured and Docker installed locally. This project also assumes you have an existing ECS cluster with a VPC and subnets already provisioned (typically set up by your infrastructure team or a prior project).

Create AWS Resources

Your telemetry platform needs three categories of AWS resources before any containers can run. Storage buckets hold the metric, log, and trace data that each backend writes. Container registries store the custom Docker images you will build in a later step. IAM roles grant the running containers permission to access those buckets.

In this step, you will create all of these foundational resources using the AWS CLI. Each resource follows a consistent naming convention prefixed with shiproom-telemetry- so everything stays organized and easy to identify.

In this step, get ready to:
  • Create three S3 buckets for Mimir, Loki, and Tempo backend storage.
  • Create five ECR repositories for your custom container images.
  • Create IAM roles with least-privilege S3 permissions for your Fargate task.
Create the S3 storage buckets

Each backend (Mimir, Loki, Tempo) writes its data to a dedicated S3 bucket. Separating storage by backend keeps costs transparent and makes cleanup straightforward.

  • Create the three S3 buckets by running these commands:
aws s3 mb s3://shiproom-telemetry-mimir-blocks --region us-east-1
aws s3 mb s3://shiproom-telemetry-loki-chunks --region us-east-1
aws s3 mb s3://shiproom-telemetry-tempo-traces --region us-east-1

What do these commands do?

Each aws s3 mb command creates ("makes") a new S3 bucket in the us-east-1 region.

  • shiproom-telemetry-mimir-blocks stores Mimir's time-series metric blocks.
  • shiproom-telemetry-loki-chunks stores Loki's compressed log chunks and index data.
  • shiproom-telemetry-tempo-traces stores Tempo's trace data.
  • Confirm the buckets exist by running:
aws s3 ls | grep shiproom-telemetry

You should see all three bucket names listed with their creation dates.

Seeing a BucketAlreadyExists error?

S3 bucket names are globally unique across all AWS accounts. If someone else already owns one of these names, add a short suffix to each bucket name (e.g. shiproom-telemetry-mimir-blocks-abc123). Remember to use the same names when you write your backend config files in the next step.

help me fix this S3 bucket naming conflict

Create the ECR repositories

You need a container registry for each component's custom Docker image. Amazon ECR stores these images privately in your account, and your Fargate task pulls from ECR at launch time.

  • Create the five ECR repositories by running these commands:
aws ecr create-repository --repository-name shiproom-telemetry-mimir --region us-east-1
aws ecr create-repository --repository-name shiproom-telemetry-loki --region us-east-1
aws ecr create-repository --repository-name shiproom-telemetry-tempo --region us-east-1
aws ecr create-repository --repository-name shiproom-telemetry-alloy --region us-east-1
aws ecr create-repository --repository-name shiproom-telemetry-grafana --region us-east-1

What do these commands do?

Each aws ecr create-repository command creates a private container image repository. You will push custom Docker images (official base image + your config file) to these repos in Step 4.

  • shiproom-telemetry-mimir for the Mimir metrics backend.
  • shiproom-telemetry-loki for the Loki logs backend.
  • shiproom-telemetry-tempo for the Tempo traces backend.
  • shiproom-telemetry-alloy for the Alloy telemetry collector.
  • shiproom-telemetry-grafana for the Grafana dashboard with provisioned data sources.
  • Verify the repositories exist by running:
aws ecr describe-repositories --region us-east-1 --query "repositories[?starts_with(repositoryName, 'shiproom-telemetry')].repositoryName" --output table

You should see a table listing all five repository names.

Seeing an AccessDeniedException?

Your AWS CLI user or role needs the ecr:CreateRepository permission. Check that your configured AWS CLI profile has admin access or at minimum the ECR full-access managed policy attached.

help me fix ECR permissions

Create IAM roles with S3 permissions

Your Fargate task needs two IAM roles. The execution role lets ECS pull container images and write logs. The task role lets the running containers read and write to your S3 buckets.

  • Create a trust policy file that allows ECS tasks to assume these roles by running:
cat > trust-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ecs-tasks.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

What is a trust policy?

A trust policy defines WHO can assume a role. This policy says the ECS tasks service (ecs-tasks.amazonaws.com) is allowed to use these roles. Without this, your Fargate containers would not be able to take on the role's permissions.

  • Create the task execution role by running:
aws iam create-role \
  --role-name shiproom-telemetry-execution-role \
  --assume-role-policy-document file://trust-policy.json
  • Attach the AWS managed ECS execution policy to this role by running:
aws iam attach-role-policy \
  --role-name shiproom-telemetry-execution-role \
  --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy

What does AmazonECSTaskExecutionRolePolicy grant?

This AWS managed policy gives ECS permission to pull images from ECR and write container logs to CloudWatch. Every Fargate task needs these baseline permissions to start successfully.

  • Create the task role (the one your containers use at runtime) by running:
aws iam create-role \
  --role-name shiproom-telemetry-task-role \
  --assume-role-policy-document file://trust-policy.json

Now you need to grant this role access to your three S3 buckets. The policy follows least-privilege principles by only granting the specific actions the backends need.

  • Create the S3 access policy file by running:
cat > s3-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::shiproom-telemetry-mimir-blocks/*",
        "arn:aws:s3:::shiproom-telemetry-loki-chunks/*",
        "arn:aws:s3:::shiproom-telemetry-tempo-traces/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": [
        "arn:aws:s3:::shiproom-telemetry-mimir-blocks",
        "arn:aws:s3:::shiproom-telemetry-loki-chunks",
        "arn:aws:s3:::shiproom-telemetry-tempo-traces"
      ]
    }
  ]
}
EOF

Why two separate statements in this policy?

Object-level actions (s3:PutObject, s3:GetObject, s3:DeleteObject) apply to objects inside the bucket, so the resource ARN ends with /*.

The s3:ListBucket action applies to the bucket itself, so its ARN has no trailing path. Mimir, Loki, and Tempo all need to list bucket contents to find existing data on startup.

  • Attach this inline policy to the task role by running:
aws iam put-role-policy \
  --role-name shiproom-telemetry-task-role \
  --policy-name telemetry-s3-access \
  --policy-document file://s3-policy.json
  • Verify the task role's trust relationship by running:
aws iam get-role --role-name shiproom-telemetry-task-role --query "Role.AssumeRolePolicyDocument"

You should see the trust policy output showing ecs-tasks.amazonaws.com as the trusted principal.

  • Verify the inline policy is attached by running:
aws iam get-role-policy \
  --role-name shiproom-telemetry-task-role \
  --policy-name telemetry-s3-access

You should see the full S3 policy document with both statements listed.

Seeing a MalformedPolicyDocument error?

Check that your trust-policy.json and s3-policy.json files have valid JSON. Common causes include a trailing comma after the last array element or mismatched braces.

Also confirm you are running the commands from the same directory where you created the JSON files. The file:// prefix references a local file path relative to your current working directory.

help me debug my IAM policy error

Clean up the helper files

You can safely delete trust-policy.json and s3-policy.json now. They were only needed to pass the policy documents to the AWS CLI. The policies themselves are stored in IAM.

Your storage, registries, and permissions are all in place. Next up, you will write the YAML configuration files that tell Mimir, Loki, and Tempo how to use these buckets and enforce multi-tenancy.

Configure the Telemetry Backends

Your AWS infrastructure is in place. S3 buckets are waiting for data, ECR repos are ready for images, and IAM roles define the permissions boundary.

Now you need to tell each backend how to behave. Each backend in the LGTM stack needs a YAML configuration file that enables multi-tenancy and points storage at the correct S3 bucket.

In this step, get ready to:
  • Write a Mimir configuration that enables multi-tenancy and stores metric blocks in S3.
  • Write a Loki configuration that enables tenant authentication and stores log chunks in S3.
  • Write a Tempo configuration that enables multi-tenancy and stores trace data in S3.
Write the Mimir configuration

Grafana Mimir is the metrics backend. It stores time-series data (like CPU usage or request counts) and responds to PromQL queries. The config file tells Mimir to require a tenant ID on every request and to store its data blocks in your S3 bucket.

  • Create the project directory and the Mimir subdirectory by running these commands:
cd ~/Desktop
mkdir -p shiproom-telemetry/mimir
cd shiproom-telemetry
  • Create a file called mimir.yaml inside the mimir/ directory and paste the following configuration:
multitenancy_enabled: true

server:
  http_listen_port: 9009
  log_level: warn

common:
  storage:
    backend: s3
    s3:
      endpoint: s3.us-east-1.amazonaws.com
      bucket_name: shiproom-telemetry-mimir-blocks
      region: us-east-1

memberlist:
  join_members:
    - localhost:7946

ingester:
  ring:
    replication_factor: 1
    kvstore:
      store: memberlist

blocks_storage:
  backend: s3
  s3:
    endpoint: s3.us-east-1.amazonaws.com
    bucket_name: shiproom-telemetry-mimir-blocks
    region: us-east-1
  tsdb:
    dir: /data/tsdb
  bucket_store:
    sync_dir: /data/bucket-store-sync

compactor:
  compaction_interval: 30m
  data_dir: /data/compactor
  sharding_ring:
    kvstore:
      store: memberlist

limits:
  ingestion_rate: 10000
  ingestion_burst_size: 200000
  max_global_series_per_user: 150000
  max_global_series_per_metric: 20000

What does this configuration do?

multitenancy_enabled: true forces Mimir to require an X-Scope-OrgID header on every incoming request. Without this header, requests are rejected. This is how Mimir isolates each tenant's metrics.

The blocks_storage section points to your shiproom-telemetry-mimir-blocks S3 bucket. Mimir will store compressed metric blocks there, organized by tenant prefix automatically.

The limits section sets default per-tenant guardrails. ingestion_rate: 10000 caps samples per second, and max_global_series_per_user: 150000 caps total active series. These prevent a single tenant from overwhelming the system.

Why no AWS access keys in the config?

When this container runs on Fargate, it uses the task role you created in Step 1. The AWS SDK credential chain automatically picks up the role's temporary credentials. No static keys needed.

  • Save mimir/mimir.yaml.
  • Confirm the file exists by running:
ls mimir/

You should see mimir.yaml in the output.

Don't see mimir.yaml?

  • Make sure you saved the file inside the mimir/ subdirectory, not in the project root.
  • Check that you're running ls from the shiproom-telemetry directory on your Desktop.

Help me troubleshoot my mimir.yaml file location.

Write the Loki configuration

Grafana Loki is the logs backend. It indexes log metadata (labels) and stores the actual log content in S3, making it far cheaper than full-text indexing solutions.

  • Create the Loki subdirectory by running:
mkdir loki
  • Create a file called loki.yaml inside the loki/ directory and paste the following configuration:
auth_enabled: true

server:
  http_listen_port: 3100
  log_level: warn

common:
  ring:
    kvstore:
      store: memberlist
    replication_factor: 1
  path_prefix: /loki

schema_config:
  configs:
    - from: "2024-01-01"
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

storage_config:
  aws:
    s3: s3://us-east-1/shiproom-telemetry-loki-chunks
    region: us-east-1
  tsdb_shipper:
    active_index_directory: /loki/tsdb-index
    cache_location: /loki/tsdb-cache

limits_config:
  retention_period: 720h
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20

compactor:
  working_directory: /loki/compactor
  retention_enabled: true

What does this configuration do?

auth_enabled: true is Loki's equivalent of Mimir's multi-tenancy flag. It requires the X-Scope-OrgID header on every request. Without it, Loki assigns all logs to a default tenant called fake.

The schema_config section uses tsdb as the index store and v13 schema. This is the latest recommended schema for new Loki deployments, offering better query performance than older formats.

The retention_period: 720h (30 days) tells the compactor to delete log chunks older than 30 days. The compactor runs inside Loki and handles this automatically.

  • Save loki/loki.yaml.
  • Confirm the file exists by running:
ls loki/

You should see loki.yaml in the output.

Don't see loki.yaml?

  • Confirm you created the loki/ directory before saving the file. If the directory doesn't exist, your editor may have saved the file elsewhere.
  • Run pwd to confirm you're inside ~/Desktop/shiproom-telemetry.

Help me find my loki.yaml file.

Write the Tempo configuration

Grafana Tempo is the traces backend. It stores distributed traces (the journey of a request across services) and supports querying by trace ID. Tempo accepts trace data via the OTLP protocol on standard ports.

  • Create the Tempo subdirectory by running:
mkdir tempo
  • Create a file called tempo.yaml inside the tempo/ directory and paste the following configuration:
multitenancy_enabled: true

server:
  http_listen_port: 3200
  log_level: warn

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

storage:
  trace:
    backend: s3
    s3:
      endpoint: s3.us-east-1.amazonaws.com
      bucket: shiproom-telemetry-tempo-traces
      region: us-east-1
    wal:
      path: /var/tempo/wal
    local:
      path: /var/tempo/blocks

overrides:
  defaults:
    ingestion:
      burst_size_bytes: 20000000
      rate_limit_bytes: 4000000

What does this configuration do?

multitenancy_enabled: true works the same way as in Mimir. Every trace ingestion request must include the X-Scope-OrgID header to identify the tenant.

The distributor.receivers.otlp section opens two ports for receiving traces: gRPC on 4317 and HTTP on 4318. These are the standard OTLP ports that any OpenTelemetry-instrumented app can send traces to.

The overrides.defaults.ingestion section sets baseline rate limits. rate_limit_bytes: 4000000 caps ingestion at roughly 4 MB/s per tenant, preventing any single tenant from saturating the pipeline.

  • Save tempo/tempo.yaml.
  • Verify all three config files exist by running:
ls mimir/ loki/ tempo/

You should see mimir.yaml, loki.yaml, and tempo.yaml listed under their respective directories.

Missing one of the config files?

  • Run find . -name "*.yaml" to search for YAML files anywhere in your project tree. If a file ended up in the wrong location, move it with mv.
  • Check for typos in directory names. The directories must be mimir, loki, and tempo (all lowercase).

Help me fix my project directory structure.

✔️ Awesome, I've got everything!

Great. Double check you've saved all three files before moving on.

ⓧ I'd like to double check the full code

Here are the complete configurations for all three files.

mimir/mimir.yaml:

multitenancy_enabled: true

server:
  http_listen_port: 9009
  log_level: warn

common:
  storage:
    backend: s3
    s3:
      endpoint: s3.us-east-1.amazonaws.com
      bucket_name: shiproom-telemetry-mimir-blocks
      region: us-east-1

memberlist:
  join_members:
    - localhost:7946

ingester:
  ring:
    replication_factor: 1
    kvstore:
      store: memberlist

blocks_storage:
  backend: s3
  s3:
    endpoint: s3.us-east-1.amazonaws.com
    bucket_name: shiproom-telemetry-mimir-blocks
    region: us-east-1
  tsdb:
    dir: /data/tsdb
  bucket_store:
    sync_dir: /data/bucket-store-sync

compactor:
  compaction_interval: 30m
  data_dir: /data/compactor
  sharding_ring:
    kvstore:
      store: memberlist

limits:
  ingestion_rate: 10000
  ingestion_burst_size: 200000
  max_global_series_per_user: 150000
  max_global_series_per_metric: 20000

loki/loki.yaml:

auth_enabled: true

server:
  http_listen_port: 3100
  log_level: warn

common:
  ring:
    kvstore:
      store: memberlist
    replication_factor: 1
  path_prefix: /loki

schema_config:
  configs:
    - from: "2024-01-01"
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

storage_config:
  aws:
    s3: s3://us-east-1/shiproom-telemetry-loki-chunks
    region: us-east-1
  tsdb_shipper:
    active_index_directory: /loki/tsdb-index
    cache_location: /loki/tsdb-cache

limits_config:
  retention_period: 720h
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20

compactor:
  working_directory: /loki/compactor
  retention_enabled: true

tempo/tempo.yaml:

multitenancy_enabled: true

server:
  http_listen_port: 3200
  log_level: warn

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

storage:
  trace:
    backend: s3
    s3:
      endpoint: s3.us-east-1.amazonaws.com
      bucket: shiproom-telemetry-tempo-traces
      region: us-east-1
    wal:
      path: /var/tempo/wal
    local:
      path: /var/tempo/blocks

overrides:
  defaults:
    ingestion:
      burst_size_bytes: 20000000
      rate_limit_bytes: 4000000

All three backends are configured. Next up, you'll configure Grafana Alloy as the telemetry gateway that routes incoming data to these backends with the correct tenant header.

Configure Alloy and Grafana

Your three telemetry backends are configured and ready to accept multi-tenant data. They can store metrics, logs, and traces in S3. But they still need an entry point that collects telemetry from applications and routes each signal to the correct backend.

That entry point is Grafana Alloy. Alloy receives telemetry in OTLP format and forwards metrics to Mimir, logs to Loki, and traces to Tempo. It also injects the X-Scope-OrgID header on every request so each backend knows which tenant the data belongs to. On the visualization side, Grafana provides dashboards where you can query each tenant's data independently.

In this step, get ready to:
  • Write the Alloy pipeline configuration that receives OTLP and forwards to all three backends.
  • Write the Grafana data source provisioning file with per-tenant header injection.
  • Write a Dockerfile for Grafana that bakes in the provisioning config.
Write the Alloy pipeline configuration

Alloy uses a declarative pipeline syntax where each component receives data, processes it, and passes it downstream. You will define a receiver, a batch processor, and three exporters that each target a different backend.

  • Create a folder called alloy in your project directory, then create a file called config.alloy inside it.
  • Paste the following pipeline configuration into alloy/config.alloy:
otelcol.receiver.otlp "default" {
  grpc {
    endpoint = "0.0.0.0:4317"
    include_metadata = true
  }

  http {
    endpoint = "0.0.0.0:4318"
    include_metadata = true
  }

  output {
    metrics = [otelcol.processor.batch.default.input]
    logs    = [otelcol.processor.batch.default.input]
    traces  = [otelcol.processor.batch.default.input]
  }
}

otelcol.processor.batch "default" {
  timeout = "2s"
  send_batch_size = 1024

  output {
    metrics = [otelcol.exporter.prometheus.mimir.input]
    logs    = [otelcol.exporter.loki.loki.input]
    traces  = [otelcol.exporter.otlphttp.tempo.input]
  }
}

otelcol.exporter.prometheus "mimir" {
  forward_to = [prometheus.remote_write.mimir.receiver]
}

prometheus.remote_write "mimir" {
  endpoint {
    url = "http://localhost:9009/api/v1/push"

    headers = {
      "X-Scope-OrgID" = "tenant-a",
    }
  }
}

otelcol.exporter.loki "loki" {
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://localhost:3100/loki/api/v1/push"

    headers = {
      "X-Scope-OrgID" = "tenant-a",
    }
  }
}

otelcol.exporter.otlphttp "tempo" {
  client {
    endpoint = "http://localhost:4318"

    headers = {
      "X-Scope-OrgID" = "tenant-a",
    }
  }
}

What does this pipeline do?

  • otelcol.receiver.otlp listens on ports 4317 (gRPC) and 4318 (HTTP) for incoming OTLP telemetry. The include_metadata = true setting propagates incoming connection metadata (like tenant headers) to downstream components.
  • otelcol.processor.batch buffers signals for 2 seconds or until 1024 items accumulate, then flushes them in a single batch. This reduces the number of API calls to each backend.
  • Each exporter targets a different backend over localhost. The X-Scope-OrgID header is set to tenant-a as a static default. In the verification step, you will send data as different tenants by overriding this header in curl requests.

Why a static tenant header in Alloy?

In production, the tenant header comes from the incoming request metadata (which include_metadata = true enables). For this project, you set a static default so the pipeline works immediately for verification. The final deployment step demonstrates sending data as different tenants by including X-Scope-OrgID in curl requests.

  • Save alloy/config.alloy.
  • Confirm the file exists by running:
ls alloy/config.alloy

You should see alloy/config.alloy printed in the output.

File not found?

  • Make sure you created the alloy directory before creating config.alloy inside it.
  • Check that your terminal is in the project root directory (the same level as the mimir, loki, and tempo folders).
  • Alloy config uses // for comments, not #. If you added hash comments, remove them.

help me troubleshoot why my Alloy config file isn't in the right place

Write the Grafana data source provisioning

Grafana needs to know how to reach each backend and which tenant header to send. Provisioning lets you define data sources as YAML so Grafana configures itself on startup without manual clicks.

  • Create a folder called grafana in your project directory, then create a file called grafana-datasources.yaml inside it.
  • Paste the following provisioning configuration into grafana/grafana-datasources.yaml:
apiVersion: 1

datasources:
  - name: Mimir (tenant-a)
    type: prometheus
    access: proxy
    url: http://localhost:9009/prometheus
    jsonData:
      httpHeaderName1: X-Scope-OrgID
    secureJsonData:
      httpHeaderValue1: tenant-a

  - name: Loki (tenant-a)
    type: loki
    access: proxy
    url: http://localhost:3100
    jsonData:
      httpHeaderName1: X-Scope-OrgID
    secureJsonData:
      httpHeaderValue1: tenant-a

  - name: Tempo (tenant-a)
    type: tempo
    access: proxy
    url: http://localhost:3200
    jsonData:
      httpHeaderName1: X-Scope-OrgID
      tracesToLogsV2:
        datasourceUid: loki-tenant-a
        filterByTraceID: true
      tracesToMetrics:
        datasourceUid: mimir-tenant-a
    secureJsonData:
      httpHeaderValue1: tenant-a

  - name: Mimir (tenant-b)
    type: prometheus
    access: proxy
    url: http://localhost:9009/prometheus
    jsonData:
      httpHeaderName1: X-Scope-OrgID
    secureJsonData:
      httpHeaderValue1: tenant-b

  - name: Loki (tenant-b)
    type: loki
    access: proxy
    url: http://localhost:3100
    jsonData:
      httpHeaderName1: X-Scope-OrgID
    secureJsonData:
      httpHeaderValue1: tenant-b

  - name: Tempo (tenant-b)
    type: tempo
    access: proxy
    url: http://localhost:3200
    jsonData:
      httpHeaderName1: X-Scope-OrgID
      tracesToLogsV2:
        datasourceUid: loki-tenant-b
        filterByTraceID: true
      tracesToMetrics:
        datasourceUid: mimir-tenant-b
    secureJsonData:
      httpHeaderValue1: tenant-b

What does this provisioning file do?

  • Six data sources are defined: three for tenant-a and three for tenant-b. Each set includes a Prometheus source (Mimir), a Loki source, and a Tempo source.
  • The httpHeaderName1 and httpHeaderValue1 fields inject the X-Scope-OrgID header on every query. This tells each backend which tenant's data to return.
  • The tracesToLogsV2 and tracesToMetrics fields on the Tempo sources link traces to their corresponding Loki and Mimir sources. This enables one-click navigation from a trace span to its associated logs or metrics.
  • Save grafana/grafana-datasources.yaml.
  • Confirm the file exists by running:
ls grafana/grafana-datasources.yaml

You should see grafana/grafana-datasources.yaml printed in the output.

YAML indentation errors later?

  • YAML is whitespace-sensitive. Make sure you use spaces (not tabs) for indentation.
  • Each datasourceUid value (like loki-tenant-a) must exactly match what Grafana assigns. If cross-signal links don't work at runtime, you can update these UIDs from the Grafana UI later.

help me fix YAML formatting in my Grafana datasources provisioning file

Write the Grafana Dockerfile

Grafana's official Docker image supports provisioning by placing YAML files in a specific directory. You will create a simple Dockerfile that copies your data source config into the image at build time.

  • Create a file called Dockerfile inside the grafana folder.
  • Paste the following into grafana/Dockerfile:
FROM grafana/grafana:13.1.0
COPY grafana-datasources.yaml /etc/grafana/provisioning/datasources/datasources.yaml
ENV GF_SECURITY_ADMIN_PASSWORD=changeme
ENV GF_AUTH_ANONYMOUS_ENABLED=false

What does this Dockerfile do?

  • FROM grafana/grafana:13.1.0 uses the official Grafana 13.1.0 image as the base.
  • COPY places the provisioning YAML into Grafana's auto-load directory. Grafana reads this on startup and creates all six data sources automatically.
  • The two ENV lines set a default admin password and disable anonymous access. Change changeme to something stronger for any real deployment.
  • Save grafana/Dockerfile.
  • Confirm both Grafana files exist by running:
ls grafana/

You should see Dockerfile and grafana-datasources.yaml listed.

Missing Dockerfile in the listing?

  • Make sure the file is named exactly Dockerfile with a capital D and no file extension.
  • Confirm you saved it inside the grafana folder, not in the project root.

help me figure out why my Grafana Dockerfile isn't showing up

✔️ Awesome, I've got everything!

Great. Double check that you have saved all three new files: alloy/config.alloy, grafana/grafana-datasources.yaml, and grafana/Dockerfile.

ⓧ I'd like to double check the full code

Here is the complete content for each file created in this step.

otelcol.receiver.otlp "default" {
  grpc {
    endpoint = "0.0.0.0:4317"
    include_metadata = true
  }

  http {
    endpoint = "0.0.0.0:4318"
    include_metadata = true
  }

  output {
    metrics = [otelcol.processor.batch.default.input]
    logs    = [otelcol.processor.batch.default.input]
    traces  = [otelcol.processor.batch.default.input]
  }
}

otelcol.processor.batch "default" {
  timeout = "2s"
  send_batch_size = 1024

  output {
    metrics = [otelcol.exporter.prometheus.mimir.input]
    logs    = [otelcol.exporter.loki.loki.input]
    traces  = [otelcol.exporter.otlphttp.tempo.input]
  }
}

otelcol.exporter.prometheus "mimir" {
  forward_to = [prometheus.remote_write.mimir.receiver]
}

prometheus.remote_write "mimir" {
  endpoint {
    url = "http://localhost:9009/api/v1/push"

    headers = {
      "X-Scope-OrgID" = "tenant-a",
    }
  }
}

otelcol.exporter.loki "loki" {
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://localhost:3100/loki/api/v1/push"

    headers = {
      "X-Scope-OrgID" = "tenant-a",
    }
  }
}

otelcol.exporter.otlphttp "tempo" {
  client {
    endpoint = "http://localhost:4318"

    headers = {
      "X-Scope-OrgID" = "tenant-a",
    }
  }
}
apiVersion: 1

datasources:
  - name: Mimir (tenant-a)
    type: prometheus
    access: proxy
    url: http://localhost:9009/prometheus
    jsonData:
      httpHeaderName1: X-Scope-OrgID
    secureJsonData:
      httpHeaderValue1: tenant-a

  - name: Loki (tenant-a)
    type: loki
    access: proxy
    url: http://localhost:3100
    jsonData:
      httpHeaderName1: X-Scope-OrgID
    secureJsonData:
      httpHeaderValue1: tenant-a

  - name: Tempo (tenant-a)
    type: tempo
    access: proxy
    url: http://localhost:3200
    jsonData:
      httpHeaderName1: X-Scope-OrgID
      tracesToLogsV2:
        datasourceUid: loki-tenant-a
        filterByTraceID: true
      tracesToMetrics:
        datasourceUid: mimir-tenant-a
    secureJsonData:
      httpHeaderValue1: tenant-a

  - name: Mimir (tenant-b)
    type: prometheus
    access: proxy
    url: http://localhost:9009/prometheus
    jsonData:
      httpHeaderName1: X-Scope-OrgID
    secureJsonData:
      httpHeaderValue1: tenant-b

  - name: Loki (tenant-b)
    type: loki
    access: proxy
    url: http://localhost:3100
    jsonData:
      httpHeaderName1: X-Scope-OrgID
    secureJsonData:
      httpHeaderValue1: tenant-b

  - name: Tempo (tenant-b)
    type: tempo
    access: proxy
    url: http://localhost:3200
    jsonData:
      httpHeaderName1: X-Scope-OrgID
      tracesToLogsV2:
        datasourceUid: loki-tenant-b
        filterByTraceID: true
      tracesToMetrics:
        datasourceUid: mimir-tenant-b
    secureJsonData:
      httpHeaderValue1: tenant-b
FROM grafana/grafana:13.1.0
COPY grafana-datasources.yaml /etc/grafana/provisioning/datasources/datasources.yaml
ENV GF_SECURITY_ADMIN_PASSWORD=changeme
ENV GF_AUTH_ANONYMOUS_ENABLED=false

Your telemetry gateway and visualization layer are fully configured. Next up, you will build Docker images for all five components and push them to ECR.

Build and Push Docker Images

Your configuration files are ready, but ECS Fargate can only pull container images from a registry. It cannot mount local config files at deploy time.

The solution is to bake each config into its own Docker image. You will write a simple Dockerfile for each component that starts from the official base image and copies your config file inside. Then you will push all five images to the ECR repositories you created earlier.

In this step, get ready to:
  • Write Dockerfiles for Mimir, Loki, Tempo, and Alloy that bake in configs.
  • Authenticate Docker to ECR and push all five images.
  • Verify every image is present in its ECR repository.
Write Dockerfiles for Mimir, Loki, Tempo, and Alloy

Each Dockerfile follows the same three-line pattern: pull the official image at the pinned version, copy the config file into the image, and set the entrypoint to run the binary in monolithic mode.

  • Create the file mimir/Dockerfile with the following content:
FROM grafana/mimir:3.1.0
COPY mimir.yaml /etc/mimir/mimir.yaml
CMD ["-config.file=/etc/mimir/mimir.yaml", "-target=all"]

What does this Dockerfile do?

The FROM line pulls Grafana Mimir 3.1.0 as the base image.

The COPY bakes your local mimir.yaml into the image at the path the binary expects. The CMD runs Mimir in monolithic mode (-target=all), which means all components (distributor, ingester, querier, compactor) run in a single process.

  • Create the file loki/Dockerfile with the following content:
FROM grafana/loki:3.7.3
COPY loki.yaml /etc/loki/loki.yaml
CMD ["-config.file=/etc/loki/loki.yaml", "-target=all"]
  • Create the file tempo/Dockerfile with the following content:
FROM grafana/tempo:3.0.2
COPY tempo.yaml /etc/tempo/tempo.yaml
CMD ["-config.file=/etc/tempo/tempo.yaml", "-target=all"]
  • Create the file alloy/Dockerfile with the following content:
FROM grafana/alloy:v1.17.1
COPY config.alloy /etc/alloy/config.alloy
CMD ["run", "/etc/alloy/config.alloy", "--server.http.listen-addr=0.0.0.0:12345"]

Why does Alloy's CMD look different?

Mimir, Loki, and Tempo are Go binaries that accept flags directly. Alloy uses a run subcommand followed by the config path. The --server.http.listen-addr flag exposes Alloy's internal UI and health endpoint on port 12345.

  • Verify you now have five Dockerfiles across your project directories: mimir/Dockerfile, loki/Dockerfile, tempo/Dockerfile, alloy/Dockerfile, and grafana/Dockerfile (written in the previous step).

Dockerfile not saving in the right place?

Each Dockerfile must live in the same directory as the config it copies. If COPY mimir.yaml fails later during build, it means the Dockerfile is not in the mimir/ folder alongside mimir.yaml.

Help me figure out why my Dockerfile COPY is failing.

Authenticate Docker to ECR and build all five images

Before you can push images, Docker needs a temporary authentication token for your ECR registry. The token lasts 12 hours.

  • Authenticate Docker to your ECR registry by running this command:
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin [[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com

What does this command do?

The aws ecr get-login-password subcommand retrieves a temporary password from AWS. The pipe sends that password to docker login so Docker can push to your private registry.

You should see Login Succeeded in the terminal output.

Seeing an authorization error?

Make sure your AWS CLI credentials have the ecr:GetAuthorizationToken permission. Double check that the account ID in the command matches your actual AWS account.

Help me fix my ECR authentication error.

  • Build and push all five images by running these commands:
# Build and push Mimir
docker build -t [[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/shiproom-telemetry-mimir:latest mimir/
docker push [[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/shiproom-telemetry-mimir:latest

# Build and push Loki
docker build -t [[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/shiproom-telemetry-loki:latest loki/
docker push [[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/shiproom-telemetry-loki:latest

# Build and push Tempo
docker build -t [[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/shiproom-telemetry-tempo:latest tempo/
docker push [[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/shiproom-telemetry-tempo:latest

# Build and push Alloy
docker build -t [[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/shiproom-telemetry-alloy:latest alloy/
docker push [[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/shiproom-telemetry-alloy:latest

# Build and push Grafana
docker build -t [[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/shiproom-telemetry-grafana:latest grafana/
docker push [[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/shiproom-telemetry-grafana:latest

What is happening here?

Each docker build command builds the image from the specified directory and tags it with the full ECR repository URI. The :latest tag marks this as the most recent version.

Each docker push uploads the built image to the matching ECR repository. This is what makes the images available to Fargate at deploy time.

  • Wait for all pushes to complete. Each image will show layer upload progress in the terminal.

Push failing with a denied or retrying error?

If you see a denied error, your Docker login token may have expired. Re-run the aws ecr get-login-password command from above to re-authenticate.

If you see retrying errors, check your internet connection and ensure Docker Desktop is running.

Help me debug my ECR push failure.

Verify all images are in ECR
  • Confirm all five images exist in ECR by running:
aws ecr list-images --repository-name shiproom-telemetry-mimir
aws ecr list-images --repository-name shiproom-telemetry-loki
aws ecr list-images --repository-name shiproom-telemetry-tempo
aws ecr list-images --repository-name shiproom-telemetry-alloy
aws ecr list-images --repository-name shiproom-telemetry-grafana

What should I see?

Each command returns a JSON object with an imageIds array. You should see an entry with imageTag set to latest for every repository. If any repository shows an empty list, re-run the build and push commands for that specific image.

Seeing an empty imageIds list?

Scroll up in your terminal to check for errors during the docker push for that repository. The most common cause is a typo in the repository name or a failed push that you missed.

Help me figure out why my ECR repository is empty.

All five images are now sitting in ECR, ready for Fargate to pull. Next up, you will wire them together into a single multi-container task definition and deploy your telemetry platform.

Deploy to Fargate and Verify Multi-Tenancy

Your container images are built and sitting in ECR, ready to run.

Now you need to bring all five containers together as a single Fargate task. In this step, you will register a multi-container task definition, launch it as an ECS service, and prove that tenant isolation works by sending test data as two separate tenants.

In this step, get ready to:
  • Register the multi-container ECS task definition.
  • Create an ECS service with a public IP for external access.
  • Verify multi-tenancy by sending test telemetry as two tenants and confirming data isolation in Grafana.
Write and register the ECS task definition

The task definition tells ECS exactly how to run your five containers together. In awsvpc network mode, all containers in the same task share a single network namespace. They communicate over localhost without needing service discovery.

  • Create the CloudWatch log group that all containers will stream logs to:
aws logs create-log-group --log-group-name /ecs/shiproom-telemetry --region us-east-1

Why a log group first?

Every container in the task definition references /ecs/shiproom-telemetry as its log destination. If the log group does not exist when the task starts, the container fails to launch.

  • Create the task definition file by running:
cat > task-definition.json << 'EOF'
{
  "family": "shiproom-telemetry",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "2048",
  "memory": "4096",
  "executionRoleArn": "arn:aws:iam::[[ACCOUNT_ID="your AWS account ID"]]:role/shiproom-telemetry-execution-role",
  "taskRoleArn": "arn:aws:iam::[[ACCOUNT_ID="your AWS account ID"]]:role/shiproom-telemetry-task-role",
  "containerDefinitions": [
    {
      "name": "mimir",
      "image": "[[ACCOUNT_ID="your AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/shiproom-telemetry-mimir:latest",
      "essential": true,
      "portMappings": [
        { "containerPort": 9009, "protocol": "tcp" }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/shiproom-telemetry",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "mimir"
        }
      }
    },
    {
      "name": "loki",
      "image": "[[ACCOUNT_ID="your AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/shiproom-telemetry-loki:latest",
      "essential": true,
      "portMappings": [
        { "containerPort": 3100, "protocol": "tcp" }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/shiproom-telemetry",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "loki"
        }
      }
    },
    {
      "name": "tempo",
      "image": "[[ACCOUNT_ID="your AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/shiproom-telemetry-tempo:latest",
      "essential": true,
      "portMappings": [
        { "containerPort": 3200, "protocol": "tcp" },
        { "containerPort": 4317, "protocol": "tcp" },
        { "containerPort": 4318, "protocol": "tcp" }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/shiproom-telemetry",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "tempo"
        }
      }
    },
    {
      "name": "alloy",
      "image": "[[ACCOUNT_ID="your AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/shiproom-telemetry-alloy:latest",
      "essential": true,
      "portMappings": [
        { "containerPort": 4317, "protocol": "tcp" },
        { "containerPort": 4318, "protocol": "tcp" },
        { "containerPort": 12345, "protocol": "tcp" }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/shiproom-telemetry",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "alloy"
        }
      }
    },
    {
      "name": "grafana",
      "image": "[[ACCOUNT_ID="your AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/shiproom-telemetry-grafana:latest",
      "essential": true,
      "portMappings": [
        { "containerPort": 3000, "protocol": "tcp" }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/shiproom-telemetry",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "grafana"
        }
      }
    }
  ]
}
EOF

What does this task definition do?

  • family names this definition shiproom-telemetry so ECS can track revisions.
  • awsvpc network mode puts all five containers on the same ENI, sharing localhost.
  • 2048 CPU units (2 vCPU) and 4096 MB memory are split across all containers.
  • The executionRoleArn lets ECS pull images from ECR. The taskRoleArn lets containers access S3.
  • Each container streams logs to CloudWatch via the awslogs driver.
  • Register the task definition with ECS by running:
aws ecs register-task-definition --cli-input-json file://task-definition.json
  • Confirm the registration succeeded. You should see a JSON response containing "family": "shiproom-telemetry" and "revision": 1 in the output.

Seeing an access denied error?

Make sure your AWS CLI credentials have the ecs:RegisterTaskDefinition permission. Also verify the IAM role ARNs in the JSON match the exact roles you created in Step 1.

help me debug an access denied error when registering my ECS task definition

Create the ECS service

The task definition describes what to run. An ECS service keeps it running and places it in your VPC.

  • Create the service by running the following command, replacing the cluster name, subnet, and security group with your existing values:
aws ecs create-service \
  --cluster [[ECS_CLUSTER="your existing ECS cluster name"]] \
  --service-name telemetry \
  --task-definition shiproom-telemetry \
  --desired-count 1 \
  --launch-type FARGATE \
  --network-configuration 'awsvpcConfiguration={subnets=[[[SUBNET_ID="your public subnet ID"]]],securityGroups=[[[SG_ID="your security group ID"]]],assignPublicIp=ENABLED}'

What does this command do?

  • --service-name telemetry creates a service named telemetry in your cluster.
  • --desired-count 1 tells ECS to keep exactly one copy of the task running.
  • assignPublicIp=ENABLED gives the task a public IP so you can reach Grafana and Alloy from outside the VPC.
  • Wait about 60 seconds for the task to reach the RUNNING state. Check the task status by running:
aws ecs list-tasks --cluster [[ECS_CLUSTER="your existing ECS cluster name"]] --service-name telemetry
  • You should see a task ARN in the output. Copy that ARN and describe the task to get its public IP:
aws ecs describe-tasks \
  --cluster [[ECS_CLUSTER="your existing ECS cluster name"]] \
  --tasks [[TASK_ARN="the task ARN from the previous command"]] \
  --query 'tasks[0].attachments[0].details[?name==`networkInterfaceId`].value' \
  --output text
  • Use the network interface ID to look up the public IP:
aws ec2 describe-network-interfaces \
  --network-interface-ids [[ENI_ID="the ENI ID from the previous command"]] \
  --query 'NetworkInterfaces[0].Association.PublicIp' \
  --output text
  • Note down the public IP. You will use it to access Grafana and send test telemetry.

Task stuck in PROVISIONING or STOPPED?

Check the task's stopped reason by describing the task without the --query flag. Common causes are incorrect image URIs in the task definition, missing log group, or security group blocking outbound traffic to ECR and S3.

help me debug why my ECS Fargate task is not reaching RUNNING state

Verify multi-tenancy with test data

The whole point of this stack is tenant isolation. You will push test logs directly to Loki as two different tenants using the X-Scope-OrgID header, then confirm in Grafana that each tenant only sees their own data.

  • Send a test log as tenant-a by running:
curl -H "Content-Type: application/json" \
  -H "X-Scope-OrgID: tenant-a" \
  -s -X POST "http://[[TASK_IP="your task public IP"]]:3100/loki/api/v1/push" \
  --data-raw '{"streams":[{"stream":{"service":"test","env":"dev"},"values":[["'$(date +%s)000000000'","Hello from tenant-a"]]}]}'
  • Send a test log as tenant-b by running:
curl -H "Content-Type: application/json" \
  -H "X-Scope-OrgID: tenant-b" \
  -s -X POST "http://[[TASK_IP="your task public IP"]]:3100/loki/api/v1/push" \
  --data-raw '{"streams":[{"stream":{"service":"test","env":"dev"},"values":[["'$(date +%s)000000000'","Hello from tenant-b"]]}]}'

What do these curl commands do?

Each command pushes a single log line to Loki's /loki/api/v1/push endpoint. The X-Scope-OrgID header tells Loki which tenant owns the data. Loki stores each tenant's logs in separate index paths within the S3 bucket.

The $(date +%s)000000000 generates a current timestamp in nanoseconds, which is the format Loki expects.

  • Open Grafana in your browser by navigating to http://your-task-public-ip:3000 in the address bar.
  • Log in with username admin and password changeme.
  • Click **Explore** in the left sidebar.
  • Select the **Loki (tenant-a)** data source from the dropdown at the top.
  • Enter {service="test"} in the query field and click **Run query**.
  • You should see the log line "Hello from tenant-a" in the results.
  • Switch the data source to **Loki (tenant-b)** and run the same query.
  • You should see only "Hello from tenant-b" with no cross-tenant leakage.

Why is this proof important?

Both tenants share the exact same Loki instance, port, and S3 bucket. The X-Scope-OrgID header is the only thing separating their data. If tenant-a's query returned tenant-b's logs, the multi-tenancy boundary would be broken.

Not seeing any logs in Grafana?

Make sure your security group allows inbound traffic on port 3100 (Loki) and port 3000 (Grafana). Also check that the curl commands returned an HTTP 204 (no content), which means Loki accepted the push. If you see a 401 or 403, the X-Scope-OrgID header may be missing.

help me troubleshoot why my Grafana Loki data source shows no log results

✔️ Awesome, I've got everything!

Great. Double check you have saved task-definition.json and that your ECS service is running with all containers healthy.

ⓧ I'd like to double check the full code

Here is the complete task-definition.json for reference:

{
  "family": "shiproom-telemetry",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "2048",
  "memory": "4096",
  "executionRoleArn": "arn:aws:iam::ACCOUNT_ID:role/shiproom-telemetry-execution-role",
  "taskRoleArn": "arn:aws:iam::ACCOUNT_ID:role/shiproom-telemetry-task-role",
  "containerDefinitions": [
    {
      "name": "mimir",
      "image": "ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/shiproom-telemetry-mimir:latest",
      "essential": true,
      "portMappings": [
        { "containerPort": 9009, "protocol": "tcp" }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/shiproom-telemetry",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "mimir"
        }
      }
    },
    {
      "name": "loki",
      "image": "ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/shiproom-telemetry-loki:latest",
      "essential": true,
      "portMappings": [
        { "containerPort": 3100, "protocol": "tcp" }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/shiproom-telemetry",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "loki"
        }
      }
    },
    {
      "name": "tempo",
      "image": "ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/shiproom-telemetry-tempo:latest",
      "essential": true,
      "portMappings": [
        { "containerPort": 3200, "protocol": "tcp" },
        { "containerPort": 4317, "protocol": "tcp" },
        { "containerPort": 4318, "protocol": "tcp" }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/shiproom-telemetry",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "tempo"
        }
      }
    },
    {
      "name": "alloy",
      "image": "ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/shiproom-telemetry-alloy:latest",
      "essential": true,
      "portMappings": [
        { "containerPort": 4317, "protocol": "tcp" },
        { "containerPort": 4318, "protocol": "tcp" },
        { "containerPort": 12345, "protocol": "tcp" }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/shiproom-telemetry",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "alloy"
        }
      }
    },
    {
      "name": "grafana",
      "image": "ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/shiproom-telemetry-grafana:latest",
      "essential": true,
      "portMappings": [
        { "containerPort": 3000, "protocol": "tcp" }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/shiproom-telemetry",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "grafana"
        }
      }
    }
  ]
}

Secret mission

Add Per-Tenant Rate Limits and Retention Overrides

Your multi-tenant platform treats all tenants equally right now. In production, different tenants have different needs. A high-volume production tenant needs generous ingestion limits and long retention, while a dev-environment tenant should be tightly capped. Add per-tenant runtime overrides to Mimir and Tempo so each backend enforces differentiated service tiers.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. The Fargate task incurs compute charges while running, and S3 stores data at a per-GB rate.

Cost warning

Your ECS Fargate task costs approximately $2-3 per day while running (2 vCPU, 4 GB memory). S3 storage costs ~$0.023/GB/month for Standard. If you are not actively using the telemetry platform, pause or delete to avoid charges.

Resources you used:

  • 3 S3 buckets (shiproom-telemetry-mimir-blocks, shiproom-telemetry-loki-chunks, shiproom-telemetry-tempo-traces).
  • 5 ECR repositories (shiproom-telemetry-mimir, shiproom-telemetry-loki, shiproom-telemetry-tempo, shiproom-telemetry-alloy, shiproom-telemetry-grafana).
  • 1 ECS Fargate service (telemetry) with task definition shiproom-telemetry.
  • 2 IAM roles (shiproom-telemetry-task-role, shiproom-telemetry-execution-role).
  • 1 CloudWatch log group (/ecs/shiproom-telemetry).
  • Local project files on your Desktop.

Keep everything running

No action needed. Choose this if you are still actively building or plan to connect your Flask app to the telemetry platform soon.

  • Your Fargate task stays active and ready to receive telemetry data.
  • Expect approximately $2-3/day in Fargate charges while the task runs.
  • S3 storage costs remain minimal until significant telemetry data accumulates.

Pause - I'll come back to this later

Shut down running Fargate tasks to stop compute charges while preserving your data and configuration.

  • Scale the ECS service down to zero by running:
  • S3 data persists at minimal cost. No Fargate charges accrue while the desired count is zero.
  • To resume later, set the desired count back to 1.
aws ecs update-service --cluster [[CLUSTER_NAME="your-cluster-name"]] --service telemetry --desired-count 0

When you are ready to resume, run the same command with --desired-count 1.

Delete - I don't want to use this again

Remove all project resources completely. This cannot be undone.

  • Delete the ECS service by running:
aws ecs delete-service --cluster [[CLUSTER_NAME="your-cluster-name"]] --service telemetry --force
  • Deregister the task definition (replace the revision number with your current revision) by running:
aws ecs deregister-task-definition --task-definition shiproom-telemetry:[[REVISION="your revision number"]]
  • Delete all 5 ECR repositories by running:
aws ecr delete-repository --repository-name shiproom-telemetry-mimir --force
aws ecr delete-repository --repository-name shiproom-telemetry-loki --force
aws ecr delete-repository --repository-name shiproom-telemetry-tempo --force
aws ecr delete-repository --repository-name shiproom-telemetry-alloy --force
aws ecr delete-repository --repository-name shiproom-telemetry-grafana --force
  • Empty and delete all 3 S3 buckets by running:
aws s3 rb s3://shiproom-telemetry-mimir-blocks --force
aws s3 rb s3://shiproom-telemetry-loki-chunks --force
aws s3 rb s3://shiproom-telemetry-tempo-traces --force
  • Delete the IAM task role. First remove its inline policy, then delete the role:
aws iam delete-role-policy --role-name shiproom-telemetry-task-role --policy-name [[POLICY_NAME="your inline policy name"]]
aws iam delete-role --role-name shiproom-telemetry-task-role
  • Delete the IAM execution role. First detach the managed policy, then delete the role:
aws iam detach-role-policy --role-name shiproom-telemetry-execution-role --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
aws iam delete-role --role-name shiproom-telemetry-execution-role
  • Delete the CloudWatch log group by running:
aws logs delete-log-group --log-group-name /ecs/shiproom-telemetry
  • Remove local project files by running:
cd ~/Desktop
rm -rf mimir loki tempo alloy grafana task-definition.json

Nice Work!

Nice Work!

You just deployed a production-grade, multi-tenant telemetry platform on AWS Fargate. Five containers working together to collect, store, and visualize metrics, logs, and traces with full tenant isolation.

Here's what you accomplished:

  • Built a multi-tenant telemetry platform with Grafana Mimir (metrics), Loki (logs), and Tempo (traces) running in monolithic mode on ECS Fargate with S3-backed persistent storage.
  • Configured Grafana Alloy as the unified OTLP collector with X-Scope-OrgID header injection for tenant routing across all three backends.
  • Verified tenant isolation by sending test signals as two different tenants and confirming data separation in Grafana with per-tenant data sources linked for cross-signal correlation.
  • (Secret Mission) Added per-tenant runtime overrides for rate limiting and retention policies across all three backends.

Ready to quiz yourself?