Build an LLM Observability Pipeline

Stream LLM traces through Kafka into ClickHouse and visualize them in Grafana.

Introduction

30 Second Summary

Every time an AI-powered app responds to a user, it generates data. Which model answered? How long did it take? How much did it cost? Teams building with AI need a way to track all of this in real time.

In this project, you will build a local observability pipeline that streams mock LLM trace events through Kafka, stores them in ClickHouse, and visualizes everything on a live Grafana dashboard.

What You'll Build

A real-time data pipeline that simulates tracing API calls across multiple LLM providers and displays cost, latency, and quality metrics on a live dashboard.

By the end of this project, you'll have:

  • A Docker Compose stack that spins up Kafka, ClickHouse, and Grafana with a single command.
  • A Python producer script that generates realistic mock LLM trace events and streams them through Kafka into ClickHouse automatically.
  • A live Grafana dashboard with panels for cost breakdown, latency per provider, and quality scores that update in real time.
  • Secret Mission: Add a cost-efficiency ranking panel that compares models by quality per dollar spent.

Are there any prerequisites?

You need Python, Docker, and a code editor (like VS Code or Cursor) installed on your machine. No prior Docker Compose or Kafka experience is required.

Set Up the Infrastructure with Docker Compose

An observability pipeline needs multiple services working together: a message broker to receive events, an analytics database to store them, and a dashboard to visualize them.

Docker Compose lets you define all three services in a single file and launch them with one command. In this step, you will write a docker-compose.yml that brings up Kafka, ClickHouse, and Grafana on a shared network.

In this step, get ready to:
  • Write a Docker Compose file that defines Kafka, ClickHouse, and Grafana as services.
  • Start all three containers and verify they are running.
  • Log into Grafana and confirm the ClickHouse data source plugin is available.
Create the project directory and Docker Compose file

Docker Compose reads a YAML file that declares each service, its image, environment variables, ports, and network. All three services will share a bridge network called llm-pipeline so they can reach each other by container name.

  • Open your terminal by pressing Cmd+Space (macOS) or the Windows key (Windows), typing Terminal, and pressing Enter.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir llm-pipeline
cd llm-pipeline
  • Confirm the folder exists by running:
ls

You should see the llm-pipeline folder listed in the output (if you ran ls from ~/Desktop). Since you already moved into it, the folder will appear empty for now.

  • Open your code editor. Press Cmd+Space (macOS) or the Windows key (Windows), type the name of your editor (e.g. VS Code or Cursor), and press Enter.
  • Open the llm-pipeline folder in your editor. Click File in the top menu bar, then Open Folder, and select the llm-pipeline folder on your Desktop.
  • Create a new file called docker-compose.yml in the root of your llm-pipeline folder.
  • Paste the following configuration into docker-compose.yml:
services:
  kafka:
    image: apache/kafka:latest
    container_name: kafka
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_NUM_PARTITIONS: 3
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
    ports:
      - "9092:9092"
    networks:
      - llm-pipeline

  clickhouse:
    image: clickhouse/clickhouse-server:latest
    container_name: clickhouse
    environment:
      CLICKHOUSE_DB: default
      CLICKHOUSE_USER: default
      CLICKHOUSE_PASSWORD: ""
      CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT: 1
    ports:
      - "8123:8123"
      - "9000:9000"
    ulimits:
      nofile:
        soft: 262144
        hard: 262144
    depends_on:
      - kafka
    networks:
      - llm-pipeline

  grafana:
    image: grafana/grafana-oss:13.0.2
    container_name: grafana
    environment:
      GF_SECURITY_ADMIN_USER: admin
      GF_SECURITY_ADMIN_PASSWORD: admin
      GF_PLUGINS_PREINSTALL: grafana-clickhouse-datasource
    ports:
      - "3000:3000"
    depends_on:
      - clickhouse
    networks:
      - llm-pipeline

networks:
  llm-pipeline:
    driver: bridge

What does this file do?

This file defines three services that Docker Compose will start together:

  • kafka runs Apache Kafka in KRaft mode (no Zookeeper needed). It acts as both broker and controller on a single node. Port 9092 is exposed for your Python producer to connect to.
  • clickhouse runs ClickHouse with the HTTP interface on port 8123 and native protocol on port 9000. The ulimits setting ensures stable operation in Docker.
  • grafana runs Grafana OSS 13.0.2 with the ClickHouse data source plugin pre-installed via GF_PLUGINS_PREINSTALL. Port 3000 gives you access to the dashboard UI.

The llm-pipeline bridge network lets each container address the others by service name (e.g. Grafana reaches ClickHouse at clickhouse:9000).

Why KRaft instead of Zookeeper?

KRaft is Kafka's built-in consensus layer that replaced Zookeeper. It simplifies single-node setups because you don't need to run a separate Zookeeper container. The apache/kafka:latest image runs in KRaft combined mode by default.

  • Save docker-compose.yml.

File not saving?

Make sure the file is named exactly docker-compose.yml (not docker-compose.yaml or dockercompose.yml). Check that the file is saved inside the llm-pipeline folder, not one level above it.

help me fix a file saving issue in my code editor

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before moving on.

ⓧ I'd like to double check the full code

services:
  kafka:
    image: apache/kafka:latest
    container_name: kafka
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_NUM_PARTITIONS: 3
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
    ports:
      - "9092:9092"
    networks:
      - llm-pipeline

  clickhouse:
    image: clickhouse/clickhouse-server:latest
    container_name: clickhouse
    environment:
      CLICKHOUSE_DB: default
      CLICKHOUSE_USER: default
      CLICKHOUSE_PASSWORD: ""
      CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT: 1
    ports:
      - "8123:8123"
      - "9000:9000"
    ulimits:
      nofile:
        soft: 262144
        hard: 262144
    depends_on:
      - kafka
    networks:
      - llm-pipeline

  grafana:
    image: grafana/grafana-oss:13.0.2
    container_name: grafana
    environment:
      GF_SECURITY_ADMIN_USER: admin
      GF_SECURITY_ADMIN_PASSWORD: admin
      GF_PLUGINS_PREINSTALL: grafana-clickhouse-datasource
    ports:
      - "3000:3000"
    depends_on:
      - clickhouse
    networks:
      - llm-pipeline

networks:
  llm-pipeline:
    driver: bridge
Start the containers

With the Compose file saved, a single command pulls all the images and starts the three containers in the background.

  • In your terminal, make sure you are inside the llm-pipeline folder (where docker-compose.yml lives).
  • Start all services in detached mode by running:
docker compose up -d

What does this command do?

docker compose up reads your docker-compose.yml, pulls any images not already cached, creates the llm-pipeline network, and starts all three containers. The -d flag runs them in the background so your terminal stays free.

The first run downloads several hundred megabytes of images. This may take 1-3 minutes depending on your internet connection.

  • Verify all three containers are running by running:
docker compose ps

You should see three rows with kafka, clickhouse, and grafana listed. The STATUS column should show Up followed by an uptime for each container.

Containers not showing as Up?

  • Make sure Docker Desktop is running. Check for the Docker whale icon in your system tray (macOS menu bar or Windows taskbar).
  • If a container shows exited, check its logs by running docker compose logs kafka (replace kafka with the service name that failed).
  • If port 9092, 8123, 9000, or 3000 is already in use, stop the conflicting process or change the port mapping in your Compose file.

help me troubleshoot my Docker Compose containers not starting

Log into Grafana and confirm the ClickHouse plugin

Grafana is now running on port 3000. You will log in with the default credentials and verify that the ClickHouse data source plugin was pre-installed by the GF_PLUGINS_PREINSTALL environment variable.

  • Open your web browser and navigate to http://localhost:3000.
  • Enter admin in the Email or username field.
  • Enter admin in the Password field.
  • Click Log in.

Seeing a password change prompt?

Grafana asks you to change the default password on first login. Click Skip to bypass this for now. This is a local development instance, so keeping the default credentials is fine.

  • In the left sidebar, click the hamburger menu icon (three horizontal lines) to expand the menu.
  • Click Connections.
  • Click Data sources.
  • Click Add data source.
  • In the search bar, type ClickHouse.

You should see ClickHouse appear in the list of available data sources. This confirms the grafana-clickhouse-datasource plugin was installed successfully.

Don't configure the data source yet

You will set up the ClickHouse data source connection in a later step. For now, just confirm the plugin is visible and close this page.

ClickHouse not showing in the data sources list?

  • Check that your docker-compose.yml has GF_PLUGINS_PREINSTALL: grafana-clickhouse-datasource under the grafana service's environment section.
  • If you edited the Compose file after starting, run docker compose down followed by docker compose up -d to recreate the containers with the updated config.

help me get the ClickHouse plugin working in Grafana

Your infrastructure is up and running. Next up, you will create the ClickHouse schema that automatically ingests data from Kafka.

Create the ClickHouse Schema

Your infrastructure is up and running. Kafka is ready to accept messages, ClickHouse is waiting for queries, and Grafana is standing by for visualizations.

Before data can flow through the pipeline, ClickHouse needs to know how to receive and store it. In this step, you'll create a three-part schema that tells ClickHouse exactly how to ingest events directly from Kafka without any external connectors.

In this step, get ready to:
  • Create a MergeTree table to store LLM trace data.
  • Create a Kafka Engine table that reads from the llm-traces topic.
  • Create a Materialized View that glues them together for auto-ingestion.
Create the destination table

ClickHouse uses a columnar storage engine called MergeTree for analytics workloads. This table will be the permanent home for all LLM trace data that arrives through the pipeline.

  • In your code editor, create a new file called init.sql in the same folder as your docker-compose.yml.
  • Add the following SQL to define the destination table:
CREATE TABLE IF NOT EXISTS llm_traces (
    trace_id String,
    timestamp DateTime,
    provider String,
    model String,
    latency_ms UInt32,
    input_tokens UInt32,
    output_tokens UInt32,
    cost_usd Float64,
    quality_score Float32,
    status String
) ENGINE = MergeTree()
ORDER BY (timestamp, provider);

What does this SQL do?

This creates a table called llm_traces with columns for every field in an LLM trace event: the provider name, model, latency, token counts, cost, quality score, and status.

The ENGINE = MergeTree() tells ClickHouse to use its high-performance columnar storage engine. ORDER BY (timestamp, provider) sorts data on disk by time and provider, making time-range and provider-filtered queries fast.

  • Save init.sql.

You won't see a change yet. The next chunk adds the Kafka Engine table that feeds data into this destination.

Create the Kafka Engine table

ClickHouse has a built-in Kafka Engine that acts as a consumer, reading messages directly from a Kafka topic. It does not store data on disk. It serves as a virtual source that makes Kafka messages available to SQL queries.

  • In init.sql, add a blank line after the first statement, then add the following SQL:
CREATE TABLE IF NOT EXISTS llm_traces_queue (
    trace_id String,
    timestamp DateTime,
    provider String,
    model String,
    latency_ms UInt32,
    input_tokens UInt32,
    output_tokens UInt32,
    cost_usd Float64,
    quality_score Float32,
    status String
) ENGINE = Kafka
SETTINGS
    kafka_broker_list = 'kafka:9092',
    kafka_topic_list = 'llm-traces',
    kafka_group_name = 'clickhouse_consumer',
    kafka_format = 'JSONEachRow',
    kafka_num_consumers = 1;

What does this SQL do?

This creates a virtual table called llm_traces_queue that connects to Kafka as a consumer. The columns match the destination table exactly so data can flow between them without transformation.

The kafka_broker_list uses kafka:9092 because containers on the same Docker network reach each other by service name, not localhost. The kafka_format = 'JSONEachRow' tells ClickHouse to expect one JSON object per line.

  • Save init.sql.

You won't see a change yet. The next chunk ties these two tables together with a Materialized View.

Create the Materialized View and run the schema

A Materialized View in ClickHouse acts as a trigger. It watches the Kafka Engine table and automatically moves new rows into the MergeTree destination table the moment they arrive.

  • In init.sql, add a blank line after the second statement, then add the following SQL:
CREATE MATERIALIZED VIEW IF NOT EXISTS llm_traces_mv TO llm_traces AS
SELECT
    trace_id,
    timestamp,
    provider,
    model,
    latency_ms,
    input_tokens,
    output_tokens,
    cost_usd,
    quality_score,
    status
FROM llm_traces_queue;

What does this SQL do?

The TO llm_traces clause tells the Materialized View where to send data. Whenever a row appears in llm_traces_queue (from Kafka), the view selects all columns and inserts them into llm_traces (the MergeTree table).

This three-tier pattern (Kafka Engine + MergeTree + Materialized View) is the canonical ClickHouse ingestion approach. It eliminates the need for external connectors like Kafka Connect.

  • Save init.sql.

✔️ Awesome, I've got everything!

Great. Double check you've saved init.sql with all three SQL statements.

ⓧ I'd like to double check the full code

Here is the complete init.sql file:

CREATE TABLE IF NOT EXISTS llm_traces (
    trace_id String,
    timestamp DateTime,
    provider String,
    model String,
    latency_ms UInt32,
    input_tokens UInt32,
    output_tokens UInt32,
    cost_usd Float64,
    quality_score Float32,
    status String
) ENGINE = MergeTree()
ORDER BY (timestamp, provider);

CREATE TABLE IF NOT EXISTS llm_traces_queue (
    trace_id String,
    timestamp DateTime,
    provider String,
    model String,
    latency_ms UInt32,
    input_tokens UInt32,
    output_tokens UInt32,
    cost_usd Float64,
    quality_score Float32,
    status String
) ENGINE = Kafka
SETTINGS
    kafka_broker_list = 'kafka:9092',
    kafka_topic_list = 'llm-traces',
    kafka_group_name = 'clickhouse_consumer',
    kafka_format = 'JSONEachRow',
    kafka_num_consumers = 1;

CREATE MATERIALIZED VIEW IF NOT EXISTS llm_traces_mv TO llm_traces AS
SELECT
    trace_id,
    timestamp,
    provider,
    model,
    latency_ms,
    input_tokens,
    output_tokens,
    cost_usd,
    quality_score,
    status
FROM llm_traces_queue;
  • Run the schema against ClickHouse by executing this command in your terminal:
docker exec -i clickhouse clickhouse-client < init.sql

What does this command do?

The docker exec -i flag keeps stdin open so you can pipe data into the container. The < init.sql part sends your SQL file to clickhouse-client running inside the clickhouse container, which executes all three statements in sequence.

  • Verify the schema was created by running:
docker exec clickhouse clickhouse-client --query "SELECT count() FROM llm_traces"
  • You should see 0 printed in the terminal. This confirms the table exists and is empty, ready to receive data from Kafka.

Seeing an error when running the schema?

  • Make sure you are running the command from the same directory where init.sql is saved. The < init.sql part reads the file from your current working directory.
  • If you see a connection error, check that the ClickHouse container is still running with docker compose ps.
  • If you see a syntax error, double check that each SQL statement ends with a semicolon and that there are no extra characters between statements.

Help me debug my ClickHouse schema execution error

Your ClickHouse schema is in place. The moment data lands in the llm-traces Kafka topic, it will automatically flow into the llm_traces table. Next up, you'll build the Python producer that generates that data.

Build the Python Trace Producer

Your infrastructure is humming. Kafka is ready to receive messages, ClickHouse is waiting to ingest them, and the Materialized View is the glue connecting both.

Now you need something to actually generate data. In this step, you will build a Python script that simulates LLM API trace events and streams them into Kafka. The moment events land in the llm-traces topic, your ClickHouse pipeline automatically picks them up.

In this step, get ready to:
  • Set up a Python virtual environment and install the Kafka client library.
  • Write a trace generator that produces realistic mock LLM telemetry.
  • Stream 200 events into Kafka and verify they land in ClickHouse automatically.
Set up the virtual environment and install confluent-kafka

The confluent-kafka package is a high-performance Python client for Kafka built on top of the C library librdkafka. It is what production teams use for reliable event streaming.

Why confluent-kafka over kafka-python?

The kafka-python package is pure Python with significant performance limitations for high-throughput use cases. confluent-kafka wraps librdkafka (a C library) and delivers enterprise-grade throughput. It is what professionals use in production pipelines.

  • Create a Python virtual environment in your project folder by running this command:

macOS

python3 -m venv venv
source venv/bin/activate

Your terminal prompt should now show (venv) at the beginning, confirming the environment is active.

Windows

python -m venv venv
venv\Scripts\activate

Your terminal prompt should now show (venv) at the beginning, confirming the environment is active.

Don't see (venv) in your prompt?

Make sure you ran the activation command (the second line) after creating the environment. On macOS, the command is source venv/bin/activate. On Windows, it is venv\Scripts\activate.

If you see a permissions error on Windows, try running your terminal as Administrator. Help me activate my Python virtual environment.

  • Create a requirements.txt file in your project folder with the following content:
confluent-kafka==2.14.2
  • Install the dependency by running:
pip install -r requirements.txt

You should see output ending with Successfully installed confluent-kafka-2.14.2 (or a note that it is already satisfied).

Install failing with a build error?

The confluent-kafka package includes a compiled C extension. On some systems you may need a C compiler installed. On macOS, run xcode-select --install if prompted. On Windows, make sure you have the Visual C++ Build Tools installed.

Help me fix a confluent-kafka installation error.

  • Create a new file called producer.py in your project folder and add the following imports and configuration:
import json
import random
import time
import uuid
from datetime import datetime, timezone

from confluent_kafka import Producer

# Map each provider to its available models
PROVIDERS = {
    "openai": ["gpt-4o", "gpt-4o-mini"],
    "anthropic": ["claude-sonnet-4", "claude-haiku-4"],
    "gemini": ["gemini-2.5-flash", "gemini-2.5-pro"],
}

# Cost per 1,000 input tokens for each model
COST_PER_1K_INPUT = {
    "gpt-4o": 0.0025,
    "gpt-4o-mini": 0.00015,
    "claude-sonnet-4": 0.003,
    "claude-haiku-4": 0.0008,
    "gemini-2.5-flash": 0.0003,
    "gemini-2.5-pro": 0.00125,
}

# Cost per 1,000 output tokens for each model
COST_PER_1K_OUTPUT = {
    "gpt-4o": 0.01,
    "gpt-4o-mini": 0.0006,
    "claude-sonnet-4": 0.015,
    "claude-haiku-4": 0.004,
    "gemini-2.5-flash": 0.0025,
    "gemini-2.5-pro": 0.01,
}

What does this code do?

The imports bring in everything you need: json for serialization, random for generating varied data, uuid for unique trace IDs, and Producer from confluent-kafka for sending messages to Kafka.

The three dictionaries define realistic pricing data for six LLM models across three providers. When the script generates a trace event, it uses these dictionaries to compute a realistic cost based on token counts.

  • Save producer.py.

You won't see a change yet. The next chunk adds the function that builds individual trace events.

Write the generate_trace() function

Each trace event simulates one LLM API call. The function needs to pick a random provider and model, generate realistic metrics, and return a dictionary that matches the ClickHouse schema you created in the previous step.

  • In producer.py, add the following function below the COST_PER_1K_OUTPUT dictionary:

def generate_trace():
    # Pick a random provider and one of its models
    provider = random.choice(list(PROVIDERS.keys()))
    model = random.choice(PROVIDERS[provider])
    input_tokens = random.randint(50, 2000)
    output_tokens = random.randint(10, 1000)
    latency_ms = random.randint(100, 3000)
    # Calculate cost based on token counts and per-model pricing
    cost_usd = (
        input_tokens / 1000 * COST_PER_1K_INPUT[model]
        + output_tokens / 1000 * COST_PER_1K_OUTPUT[model]
    )
    quality_score = round(random.uniform(0.3, 1.0), 4)
    # Weighted random status: 90% success, 7% error, 3% timeout
    status = random.choices(["success", "error", "timeout"], weights=[90, 7, 3])[0]

    return {
        "trace_id": str(uuid.uuid4()),
        "timestamp": datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S"),
        "provider": provider,
        "model": model,
        "latency_ms": latency_ms,
        "input_tokens": input_tokens,
        "output_tokens": output_tokens,
        "cost_usd": round(cost_usd, 6),
        "quality_score": quality_score,
        "status": status,
    }

What does this function do?

Each call to generate_trace() produces one mock LLM API event. It randomly selects a provider (openai, anthropic, or gemini) and one of that provider's models.

The cost calculation uses the per-model pricing dictionaries you defined above, multiplying token counts by the cost-per-1K rate. The returned dictionary has the same fields as the llm_traces table in ClickHouse, so it will ingest cleanly via the Kafka Engine.

  • Save producer.py.

You won't see a change yet. The next chunk adds the main loop that calls this function 200 times and pushes events into Kafka.

Add the main loop and run the producer

The final piece connects everything together. You need a delivery callback for error reporting, a main function that produces 200 events in a loop, and the entry point that kicks it all off.

  • In producer.py, add the following code below the generate_trace() function:

def delivery_report(err, msg):
    # Called once per message to report success or failure
    if err is not None:
        print(f"Delivery failed: {err}")


def main():
    # Connect to Kafka running on localhost
    producer = Producer({"bootstrap.servers": "localhost:9092"})
    topic = "llm-traces"

    print(f"Producing 200 LLM trace events to topic '{topic}'...")
    for i in range(200):
        trace = generate_trace()
        producer.produce(
            topic,
            key=trace["trace_id"],
            value=json.dumps(trace).encode("utf-8"),
            callback=delivery_report,
        )
        # Serve delivery callbacks without blocking
        producer.poll(0)

        # Print progress and pause briefly every 50 events
        if (i + 1) % 50 == 0:
            print(f"  Produced {i + 1} events...")
            time.sleep(0.5)

    # Wait for all outstanding messages to be delivered
    producer.flush()
    print("Done! All events delivered.")


if __name__ == "__main__":
    main()

What does this code do?

delivery_report() is a callback that Kafka invokes for each message. It prints a warning only if delivery fails.

main() creates a Kafka Producer pointing at localhost:9092 (your Docker-mapped Kafka port). It loops 200 times, generating a trace and producing it as a JSON-encoded message to the llm-traces topic. The producer.poll(0) call services delivery callbacks without blocking, and producer.flush() at the end ensures all messages are delivered before the script exits.

  • Save producer.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved producer.py with all three sections (imports/config, generate_trace(), and main()).

ⓧ I'd like to double check the full code

import json
import random
import time
import uuid
from datetime import datetime, timezone

from confluent_kafka import Producer

PROVIDERS = {
    "openai": ["gpt-4o", "gpt-4o-mini"],
    "anthropic": ["claude-sonnet-4", "claude-haiku-4"],
    "gemini": ["gemini-2.5-flash", "gemini-2.5-pro"],
}

COST_PER_1K_INPUT = {
    "gpt-4o": 0.0025,
    "gpt-4o-mini": 0.00015,
    "claude-sonnet-4": 0.003,
    "claude-haiku-4": 0.0008,
    "gemini-2.5-flash": 0.0003,
    "gemini-2.5-pro": 0.00125,
}

COST_PER_1K_OUTPUT = {
    "gpt-4o": 0.01,
    "gpt-4o-mini": 0.0006,
    "claude-sonnet-4": 0.015,
    "claude-haiku-4": 0.004,
    "gemini-2.5-flash": 0.0025,
    "gemini-2.5-pro": 0.01,
}


def generate_trace():
    provider = random.choice(list(PROVIDERS.keys()))
    model = random.choice(PROVIDERS[provider])
    input_tokens = random.randint(50, 2000)
    output_tokens = random.randint(10, 1000)
    latency_ms = random.randint(100, 3000)
    cost_usd = (
        input_tokens / 1000 * COST_PER_1K_INPUT[model]
        + output_tokens / 1000 * COST_PER_1K_OUTPUT[model]
    )
    quality_score = round(random.uniform(0.3, 1.0), 4)
    status = random.choices(["success", "error", "timeout"], weights=[90, 7, 3])[0]

    return {
        "trace_id": str(uuid.uuid4()),
        "timestamp": datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S"),
        "provider": provider,
        "model": model,
        "latency_ms": latency_ms,
        "input_tokens": input_tokens,
        "output_tokens": output_tokens,
        "cost_usd": round(cost_usd, 6),
        "quality_score": quality_score,
        "status": status,
    }


def delivery_report(err, msg):
    if err is not None:
        print(f"Delivery failed: {err}")


def main():
    producer = Producer({"bootstrap.servers": "localhost:9092"})
    topic = "llm-traces"

    print(f"Producing 200 LLM trace events to topic '{topic}'...")
    for i in range(200):
        trace = generate_trace()
        producer.produce(
            topic,
            key=trace["trace_id"],
            value=json.dumps(trace).encode("utf-8"),
            callback=delivery_report,
        )
        producer.poll(0)

        if (i + 1) % 50 == 0:
            print(f"  Produced {i + 1} events...")
            time.sleep(0.5)

    producer.flush()
    print("Done! All events delivered.")


if __name__ == "__main__":
    main()
  • Run the producer script:
python producer.py

You should see output like this:

Producing 200 LLM trace events to topic 'llm-traces'...
  Produced 50 events...
  Produced 100 events...
  Produced 150 events...
  Produced 200 events...
Done! All events delivered.

Seeing a connection refused error?

This means the producer cannot reach Kafka on localhost:9092. Make sure your Docker containers are still running by checking docker compose ps. If the kafka container is not running, restart it with docker compose up -d.

Help me fix a Kafka connection refused error from my Python producer.

  • Verify the events landed in ClickHouse by running this query:
curl http://localhost:8123 --data-binary "SELECT count() FROM llm_traces"

You should see 200 (or a number very close to it) printed in your terminal. This confirms the full pipeline is working: events flowed from your Python script into Kafka, and the Materialized View automatically ingested them into the llm_traces MergeTree table.

Seeing 0 rows?

If the count returns 0, the Materialized View may not have consumed the events yet. Wait a few seconds and try again. ClickHouse's Kafka Engine polls in the background, so there can be a brief delay.

If it still shows 0 after 10 seconds, check that your llm_traces_queue table is pointing at the correct topic name (llm-traces) and broker (kafka:9092).

Help me debug why ClickHouse is not ingesting events from Kafka.

Your end-to-end pipeline is proven. Data flows from Python to Kafka to ClickHouse without a single manual INSERT. Next up, you will visualize this data in a live Grafana dashboard.

Build the Grafana Dashboard

Your pipeline is ingesting data end-to-end. Events flow from the Python producer, through Kafka, and into ClickHouse automatically.

Now you need a way to see that data in real time. In this step, you will connect Grafana to ClickHouse and build a live dashboard with three panels: cost breakdown by provider, latency over time, and quality scores per model.

In this step, get ready to:
  • Connect Grafana to ClickHouse as a data source.
  • Create a dashboard with three visualization panels.
  • Enable auto-refresh and verify live data updates.
Add the ClickHouse data source in Grafana

Grafana needs to know where your data lives before it can visualize anything. You will connect it to your ClickHouse container using the Docker network name.

  • Return to the Grafana UI at http://localhost:3000 in your browser.
  • Click Connections in the left-side menu.
  • Click Add new connection.
  • Type ClickHouse in the search bar.
  • Click ClickHouse from the results.
  • Click Add new data source in the upper right.

Grafana opens the data source settings page. Configure the connection fields:

  • Set Server to clickhouse.
  • Set Protocol to Native.
  • Set Port to 9000.
  • Set Username to default.
  • Leave the Password field empty.
  • Scroll to the bottom and click Save & test.

You should see a green confirmation message that says the data source is working.

Why use clickhouse instead of localhost?

Grafana runs inside a Docker container on the llm-pipeline network. From inside that network, containers reach each other by their service name. Using localhost would point Grafana at itself, not at the ClickHouse container.

Save & test failed?

Make sure all three containers are running. Switch to your terminal and run docker compose ps to check their status.

Confirm the server field is clickhouse (the Docker service name), not localhost. Confirm the port is 9000 (Native protocol), not 8123 (HTTP).

help me connect Grafana to my ClickHouse data source in Docker

Create the dashboard and add a cost breakdown panel

With the data source connected, you can now build panels that query ClickHouse in real time. The first panel is a pie chart showing how much each LLM provider costs.

  • Click Dashboards in the left-side menu.
  • Click New and select New Dashboard.
  • Click the Add new element icon (blue plus sign) and click a panel onto the dashboard.
  • Click Configure visualization on the new panel.

Grafana opens the panel editor. Now set the query and visualization type:

  • In the query editor at the bottom, make sure your ClickHouse data source is selected.
  • Switch the query mode to SQL Editor so you can write raw SQL.
  • Enter the following query:
SELECT provider, sum(cost_usd) FROM llm_traces GROUP BY provider
  • Click Refresh (or press Shift+Enter) to run the query.
  • In the visualization picker on the right side, click All visualizations and select Pie chart.
  • Enter Cost by Provider as the panel title at the top.
  • Click Back to dashboard in the upper left to return to the dashboard view.

You should see a pie chart showing cost segments for openai, anthropic, and gemini.

What does this query do?

The sum(cost_usd) function totals up all the simulated spend for each provider. The GROUP BY provider clause splits the results into one slice per provider. Grafana maps each group to a slice in the pie chart automatically.

Add latency and quality panels

A single panel only tells part of the story. Adding latency and quality panels gives you a full operational picture across all three providers.

Create the latency time series panel:

  • Click the Add new element icon (blue plus sign) and click a panel onto the dashboard.
  • Click Configure visualization.
  • Switch to SQL Editor mode and enter this query:
SELECT toStartOfMinute(timestamp) as time, provider, avg(latency_ms) FROM llm_traces GROUP BY time, provider ORDER BY time
  • Click Refresh to run the query.
  • In the visualization picker, select Time series (this is usually the default suggestion).
  • Set the panel title to Avg Latency by Provider.
  • Click Back to dashboard.

What does toStartOfMinute do?

ClickHouse's toStartOfMinute() function truncates each timestamp down to the minute. This groups all events that happened in the same minute together, giving you a smooth time series line instead of scattered individual points.

Now create the quality score bar gauge panel:

  • Click the Add new element icon and click a new panel onto the dashboard.
  • Click Configure visualization.
  • Switch to SQL Editor mode and enter this query:
SELECT model, avg(quality_score) FROM llm_traces GROUP BY model ORDER BY avg(quality_score) DESC
  • Click Refresh to run the query.
  • In the visualization picker, click All visualizations and select Bar gauge.
  • Set the panel title to Avg Quality Score by Model.
  • Click Back to dashboard.

Your dashboard now has three panels side by side. Click the Save icon at the top and give the dashboard a title like LLM Observability.

Panels show no data?

Make sure you ran producer.py in Step 3. Without events in the llm_traces table, there is nothing to visualize.

Check the query editor for red error text. A common mistake is selecting the wrong data source. Confirm the ClickHouse data source is selected in the query editor dropdown.

help me fix my Grafana panel that shows no data from ClickHouse

Enable auto-refresh and verify live updates

A static dashboard is useful, but the real power of this pipeline is seeing data flow in real time. You will set the dashboard to refresh every 5 seconds, then push a new batch of events to watch the panels update live.

  • At the top of the dashboard, find the refresh interval dropdown (next to the time range picker).
  • Click it and select 5s.
  • Switch back to your terminal where the virtual environment is active.
  • Re-run the producer to generate another batch of events:
python producer.py
  • Switch back to the Grafana dashboard in your browser.
  • Watch the panels update within a few seconds as new data arrives.

The pie chart slices adjust as new cost data arrives. The time series panel shows a new data point for the latest minute. The bar gauge values shift as new quality scores are averaged in.

Why does auto-refresh matter?

In production observability, you want to spot issues the moment they appear. A 5-second refresh means you would see a sudden latency spike or error rate jump almost immediately. This is the same refresh cadence used in real-time operations dashboards.

Dashboard not updating?

Make sure the auto-refresh dropdown shows 5s and not Off. Also confirm the time range picker at the top right is set to Last 15 minutes or a recent window so new data falls within the visible range.

my Grafana dashboard auto-refresh is not picking up new data

Secret mission

Add a Cost-Efficiency Ranking Panel

Your dashboard shows cost, latency, and quality separately. But which model gives you the most bang for your buck? Add a fourth panel that ranks every model by quality-per-dollar spent.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally with Docker, so there are no ongoing cloud costs.

Resources you used:

  • Three Docker containers (Kafka, ClickHouse, Grafana) with associated volumes and the llm-pipeline network.
  • Project folder containing docker-compose.yml, init.sql, producer.py, requirements.txt, and the Python virtual environment.

Keep everything running

No action needed. Choose this if you're still actively building or want to experiment with more trace data and dashboard panels.

  • Your three containers continue running and use approximately 1-2GB of RAM total.
  • Run python producer.py anytime to send more events and watch your dashboard update.
  • The confluent-kafka package remains installed in your virtual environment inside the project folder.

Pause - I'll come back to this later

Shut down running containers to free up memory without losing any data. Your ClickHouse tables and Grafana dashboard are preserved in Docker volumes.

  • In the terminal where your project folder is open, run this command to stop all containers:
docker compose stop
  • When you're ready to resume, run this from the same folder:
docker compose start

All your data, dashboard configurations, and ClickHouse tables will still be there.

Delete - I don't want to use this again

Remove all containers, volumes, networks, and project files. This frees all disk space and removes all data permanently.

  • In the terminal, navigate to your project folder (the one containing docker-compose.yml) and run this command to remove all containers and volumes:
docker compose down -v
  • This removes the Kafka, ClickHouse, and Grafana containers along with all stored data and the llm-pipeline network.
  • Navigate up one level and delete the project folder by running:
cd ..
rm -rf llm-observability-pipeline

Confirm your folder name

Replace llm-observability-pipeline with whatever you named your project folder when you created it in Step 1.

  • On Windows, use this command instead:
cd ..
rmdir /s /q llm-observability-pipeline

Everything is now removed from your machine.

That's a Wrap!

That's a Wrap!

Nice work! You just built a production-grade LLM observability pipeline from scratch, running entirely on your local machine.

You've learned how to:

  • Orchestrate a multi-service stack (Kafka, ClickHouse, Grafana) with a single Docker Compose file.
  • Write a Python Kafka producer that generates realistic mock LLM trace telemetry and streams it into a topic.
  • Create an automatic ingestion pipeline using the Kafka Engine + Materialized View pattern in ClickHouse, and visualize the data live in a Grafana dashboard with cost, latency, and quality panels.
  • Secret Mission: Added a cost-efficiency ranking panel that compares model value by calculating quality per dollar spent.

Ready to quiz yourself?