Observable RAG API with Grafana

Build a local RAG API and monitor it with Prometheus and Grafana dashboards.

Introduction

30 Second Summary

When your AI-powered app takes 10 seconds to respond instead of 2, you need to know why. Is the vector database slow? Is the language model overloaded? Without metrics, you are guessing.

In this project, you will build a fully local RAG (Retrieval-Augmented Generation) API and wire it to a real-time monitoring dashboard. You'll instrument every stage of the pipeline so you can see exactly where time is spent on each request.

What You'll Build

You'll ask a question through a REST API, watch an AI generate an answer grounded in your documents, and see the entire request broken down in a live Grafana dashboard showing retrieval time, generation time, and token throughput.

By the end of this project, you'll have:

  • A local RAG API powered by Ollama and ChromaDB that answers questions using your ingested documents.
  • Prometheus metrics tracking every stage of the pipeline: retrieval latency, generation latency, token counts, and error rates.
  • A real-time Grafana dashboard with live panels and an alert that fires when generation latency spikes.
  • Secret Mission: A token efficiency and cost estimation panel that calculates tokens-per-second and estimates cost-per-query.

Are there any prerequisites?

You should be comfortable writing basic Python code. You'll also need Docker and Docker Compose installed on your machine. The first step will confirm everything is ready to go.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Launch the Multi-Service Stack

Before you can observe a RAG pipeline, you need the pipeline itself. This project runs five services: an LLM (Ollama), a vector database (ChromaDB), a FastAPI application, Prometheus for metrics collection, and Grafana for dashboards.

Docker Compose lets you define all five services in a single file and launch them together with one command. In this step, you will create all the project files and bring the entire stack online.

In this step, get ready to:
  • Create the project directory with all configuration and application files.
  • Start the five-service stack and pull the LLM model.
  • Verify every service is accessible on its port.
Create the project directory structure

You need a project folder containing the Docker Compose configuration, a Prometheus config file, and the FastAPI application source code.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create the project folder structure and move into it by running these commands:
mkdir -p observable-rag/app
cd observable-rag
  • Confirm the directory was created by running:
ls

You should see the app folder listed in the output.

Now you will create the Docker Compose file that defines all five services. This file is the backbone of the project.

  • Create a file called docker-compose.yml in the observable-rag folder and paste in the application services:
services:
  ollama:
    image: ollama/ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama

  chromadb:
    image: chromadb/chroma
    ports:
      - "8000:8000"
    volumes:
      - chroma_data:/data

  app:
    build: ./app
    ports:
      - "8080:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - CHROMA_HOST=chromadb
      - CHROMA_PORT=8000
    depends_on:
      - ollama
      - chromadb

What does this code do?

This defines three services. ollama runs the LLM engine locally. chromadb is the vector database that stores your document embeddings. app is your FastAPI application that connects to both.

The environment block passes connection details so the app knows how to reach Ollama and ChromaDB by their Docker service names.

  • Add the observability services and volume definitions below the existing content in docker-compose.yml:

  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    depends_on:
      - app

  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    depends_on:
      - prometheus

volumes:
  ollama_data:
  chroma_data:

What does this code do?

prometheus scrapes metrics from the app every 15 seconds. grafana reads those metrics and displays them in dashboards.

The volumes block at the bottom creates named volumes so Ollama's downloaded model and ChromaDB's data persist across container restarts.

  • Validate that Docker Compose can parse your file by running:
docker compose config --services

You should see all five service names printed, one per line: ollama, chromadb, app, prometheus, grafana.

Seeing a YAML parsing error?

YAML is sensitive to indentation. Make sure every nested key uses exactly 2 spaces (not tabs). Check that the services: key appears only once at the top of the file.

help me fix a YAML parsing error in my docker-compose.yml

✔️ Awesome, I've got everything!

Great. Make sure you have saved docker-compose.yml before moving on.

ⓧ I'd like to double check the full code

Scroll up and confirm your file contains both chunks in order: the three application services (ollama, chromadb, app) followed by the two observability services (prometheus, grafana) and the volumes block. The services: key should appear only once, at the very top.

Next, create the Prometheus configuration that tells it where to scrape metrics from.

  • Create a file called prometheus.yml in the observable-rag folder (next to docker-compose.yml) and paste this content:
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'rag-app'
    static_configs:
      - targets: ['app:8080']

What does this code do?

This tells Prometheus to check the app service on port 8080 every 15 seconds for new metrics. The target app:8080 uses the Docker Compose service name as the hostname, which Docker's internal networking resolves automatically.

  • Save prometheus.yml.

Now create the FastAPI application files inside the app folder.

  • Create app/requirements.txt and paste the Python dependencies:
fastapi
uvicorn[standard]
httpx
chromadb-client
prometheus-client

What are these packages?

fastapi is the web framework. uvicorn is the ASGI server that runs it. httpx makes HTTP calls to Ollama. chromadb-client is ChromaDB's lightweight Python HTTP client. prometheus-client exposes metrics for Prometheus to scrape.

  • Create app/Dockerfile and paste the container build instructions:
FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY main.py .

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]

What does this Dockerfile do?

It starts from a slim Python 3.11 base image, installs your dependencies, copies your application code, and runs uvicorn on port 8080.

  • Confirm your infrastructure files are in place by running:
ls && ls app/

You should see docker-compose.yml and prometheus.yml in the first listing, and Dockerfile and requirements.txt inside app/.

Now for the main application file. This is the FastAPI app that handles document ingestion and RAG queries.

  • Create app/main.py and paste the imports, configuration, and client setup:
import time
import os

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import httpx
import chromadb
from prometheus_client import (
    Counter,
    Histogram,
    make_asgi_app,
)

app = FastAPI(title="Observable RAG API")

# --- Configuration ---
OLLAMA_BASE_URL = os.getenv("OLLAMA_BASE_URL", "http://localhost:11434")
CHROMA_HOST = os.getenv("CHROMA_HOST", "localhost")
CHROMA_PORT = int(os.getenv("CHROMA_PORT", "8000"))
MODEL_NAME = "llama3.2:1b"

# --- Mount Prometheus metrics endpoint ---
metrics_app = make_asgi_app()
app.mount("/metrics", metrics_app)

# --- ChromaDB Client ---
chroma_client = chromadb.HttpClient(host=CHROMA_HOST, port=CHROMA_PORT)
COLLECTION_NAME = "documents"

What does this code do?

This sets up the FastAPI app with environment-based configuration. It connects to ChromaDB using the thin HTTP client and mounts a /metrics endpoint so Prometheus can scrape data later.

MODEL_NAME is set to llama3.2:1b, a small model (~1.3 GB) that loads quickly and is sufficient for demonstrating RAG.

  • Below the existing code in app/main.py, add the request and response models:


# --- Request/Response Models ---
class IngestRequest(BaseModel):
    documents: list[str]
    ids: list[str]


class QueryRequest(BaseModel):
    question: str


class QueryResponse(BaseModel):
    answer: str
    sources: list[str]
    retrieval_time_ms: float
    generation_time_ms: float

What does this code do?

These Pydantic models define the shape of data going in and out of your API. IngestRequest takes documents and IDs for storage. QueryResponse returns the answer plus timing data so you can see how long each stage takes.

  • Add the /ingest endpoint below the models:


# --- Endpoints ---
@app.post("/ingest")
def ingest_documents(request: IngestRequest):
    collection = chroma_client.get_or_create_collection(name=COLLECTION_NAME)
    collection.add(
        documents=request.documents,
        ids=request.ids,
    )
    return {"status": "ok", "count": len(request.documents)}

What does this code do?

The /ingest endpoint stores documents in ChromaDB. ChromaDB handles embedding internally using its built-in model, so you only need to send raw text.

  • Add the first half of the /query endpoint, which retrieves documents and builds a prompt:


@app.post("/query", response_model=QueryResponse)
def query_rag(request: QueryRequest):
    try:
        retrieval_start = time.time()
        collection = chroma_client.get_or_create_collection(name=COLLECTION_NAME)
        results = collection.query(
            query_texts=[request.question],
            n_results=3,
        )
        retrieval_duration = time.time() - retrieval_start

        documents = results["documents"][0] if results["documents"] else []
        context = "\n\n".join(documents)

        prompt = f"""Based on the following context, answer the question.
If the context doesn't contain relevant information, say so.

Context:
{context}

Question: {request.question}

Answer:"""

What does this code do?

This is the retrieval stage of the RAG pipeline. It queries ChromaDB for the 3 most relevant documents, then assembles them into a prompt that instructs the LLM to answer based on the retrieved context.

  • Complete the /query endpoint by adding the generation stage and health check below the prompt:

        generation_start = time.time()
        with httpx.Client(timeout=60.0) as client:
            response = client.post(
                f"{OLLAMA_BASE_URL}/api/generate",
                json={
                    "model": MODEL_NAME,
                    "prompt": prompt,
                    "stream": False,
                },
            )
            response.raise_for_status()

        generation_duration = time.time() - generation_start

        response_data = response.json()
        answer = response_data.get("response", "")

        return QueryResponse(
            answer=answer,
            sources=documents,
            retrieval_time_ms=round(retrieval_duration * 1000, 2),
            generation_time_ms=round(generation_duration * 1000, 2),
        )

    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))


@app.get("/health")
def health_check():
    return {"status": "healthy"}

What does this code do?

The generation stage sends the prompt to Ollama's /api/generate endpoint and waits for a complete response. It measures how long generation takes and returns both timing values alongside the answer.

The /health endpoint is a simple liveness check you can use to confirm the app container is running.

  • Save app/main.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved all files: docker-compose.yml, prometheus.yml, app/Dockerfile, app/requirements.txt, and app/main.py.

ⓧ I'd like to double check the full code

Your app/main.py should contain all five chunks in order: imports + config, request/response models, the /ingest endpoint, the retrieval + prompt section of /query, and the generation + health check section. Scroll up and confirm each is present.

Start the stack and pull the model

With all files in place, one command builds your app container and starts every service in the background.

  • Build and launch the entire stack by running:
docker compose up -d --build

What does this command do?

up -d starts all services in detached mode so your terminal stays free. --build forces Docker to build the app image from your Dockerfile before starting it.

The first run downloads ~1 GB of container images for Ollama, ChromaDB, Prometheus, and Grafana. This may take a few minutes depending on your internet connection.

  • Confirm all 5 containers are running by running:
docker compose ps

You should see all five services listed with a status of Up.

Seeing a container with status Exited?

If the app container exited, check its logs for a Python import or syntax error:

docker compose logs app

Common causes: a typo in main.py, or indentation errors in docker-compose.yml. Fix the issue, then rebuild with docker compose up -d --build.

help me fix a container that keeps exiting in my docker compose stack

The Ollama container is running, but it does not have a language model loaded yet. You need to pull one into it.

  • Pull the llama3.2:1b model into the Ollama container by running:
docker compose exec ollama ollama pull llama3.2:1b

What does this command do?

docker compose exec runs a command inside an already-running container. This downloads the llama3.2:1b model (~1.3 GB) into Ollama's persistent volume. You only need to do this once.

When the download finishes, you will see a success message.

Download stalling or timing out?

Large model downloads depend on your connection speed. If it appears stuck, wait a few minutes. If it fails completely, re-run the same command to resume the download.

help me troubleshoot an Ollama model pull that is timing out inside Docker

Verify all services are accessible

Each service exposes a port on your machine. Before building anything further, confirm each one responds.

  • Before you run the next command, what do you expect to get back from a running Ollama instance?
  • Check that Ollama is responding by running:
curl http://localhost:11434

You should see Ollama is running printed in your terminal.

  • Check ChromaDB by running:
curl http://localhost:8000/api/v1/heartbeat

You should see a JSON response with a nanosecond timestamp, confirming ChromaDB is alive.

  • Check your FastAPI app's health endpoint by running:
curl http://localhost:8080/health

You should see {"status":"healthy"}.

  • Open http://localhost:9090 in your browser to confirm Prometheus is running. You should see the Prometheus web UI.
  • Open http://localhost:3000 in your browser to confirm Grafana is running. You should see Grafana's login page.

Grafana login credentials

The default username is admin and the password is admin (set in your docker-compose.yml). You will use these in a later step when building the dashboard.

Connection refused on a port?

Run docker compose ps again to check the container status. If a service shows Exited, check its logs with docker compose logs <service-name>.

help me debug a Docker Compose service that is not accessible on its expected port

All five services are running and accessible. Next up, you will feed documents into the RAG pipeline and ask your first AI-generated question.

Build the RAG Pipeline

Your five-service stack is up and running. Now it's time to put it to work.

Before you can observe anything meaningful about the pipeline, you need the pipeline itself to be doing real work. In this step, you will feed documents into ChromaDB, ask questions through the API, and watch Ollama generate answers grounded in your documents.

In this step, get ready to:
  • Ingest sample documents into ChromaDB via the /ingest endpoint.
  • Query the RAG pipeline and receive AI-generated answers via the /query endpoint.
  • Confirm that answers are grounded in the ingested documents.
Ingest sample documents into ChromaDB

A RAG pipeline starts with a knowledge base. Your app's /ingest endpoint accepts a list of documents and stores them in a vector database (ChromaDB). ChromaDB automatically converts each document into an embedding so it can be searched by meaning later.

  • Feed three sample documents into your RAG pipeline by running this command:
curl -X POST http://localhost:8080/ingest \
  -H "Content-Type: application/json" \
  -d '{
    "documents": [
      "FastAPI is a modern Python web framework for building APIs. It uses type hints for automatic validation and generates OpenAPI documentation automatically.",
      "Docker Compose orchestrates multi-container applications. You define services in a YAML file and start everything with a single command.",
      "Prometheus is a time-series database that scrapes metrics from applications. It stores data with timestamps and supports a powerful query language called PromQL."
    ],
    "ids": ["doc1", "doc2", "doc3"]
  }'

You should see this response:

{"status":"ok","count":3}

What just happened?

The /ingest endpoint sent your three documents to ChromaDB. ChromaDB embedded each one (converting text into a numerical vector) and stored them in a collection called documents.

When you query later, ChromaDB will compare your question's meaning against these vectors to find the most relevant documents.

Seeing "Connection refused"?

Make sure your Docker containers are still running. Run docker compose ps and confirm the app container shows as running.

If the app container has restarted or exited, bring it back with:

docker compose up -d app

help me debug a connection refused error when curling my FastAPI app running in Docker Compose

Query the RAG pipeline

Now that your knowledge base has documents, you can ask a question. The /query endpoint retrieves the most relevant documents from ChromaDB, builds a prompt with that context, sends it to Ollama, and returns the generated answer.

  • Ask your first question by running this command:
curl -X POST http://localhost:8080/query \
  -H "Content-Type: application/json" \
  -d '{"question": "What is FastAPI used for?"}'

This may take 10-30 seconds on the first request while Ollama loads the model into memory.

You should see a JSON response containing an answer field with a response that references FastAPI's features, a sources array showing which documents were retrieved, and timing values for retrieval_time_ms and generation_time_ms.

What does the response tell you?

The sources field shows which documents ChromaDB deemed most relevant to your question. The retrieval_time_ms tells you how long ChromaDB took to search, and generation_time_ms tells you how long Ollama took to generate the answer.

These timing values are printed by the app, but they are not recorded as metrics yet. That gap becomes important in the next step.

Request timed out?

The first query can take up to 60 seconds because Ollama needs to load the llama3.2:1b model into memory. Subsequent queries will be much faster.

If it still times out, confirm Ollama has the model by running:

docker compose exec ollama ollama list

You should see llama3.2:1b in the output. If it is missing, pull it again:

docker compose exec ollama ollama pull llama3.2:1b

help me fix Ollama timeout errors in my Docker Compose RAG app

Confirm answers are grounded in your documents

A good RAG pipeline does not just generate any answer. It grounds answers in the documents you provided. Try a few more questions to see how the pipeline uses different parts of your knowledge base.

  • Ask a question about Docker Compose by running:
curl -X POST http://localhost:8080/query \
  -H "Content-Type: application/json" \
  -d '{"question": "How do you start a multi-container application?"}'

The answer should mention defining services in a YAML file and starting everything with a single command. Check the sources array to confirm the Docker Compose document was retrieved.

  • Before you run the next query, predict: will the answer mention Prometheus or FastAPI?
  • Ask a question about monitoring by running:
curl -X POST http://localhost:8080/query \
  -H "Content-Type: application/json" \
  -d '{"question": "What query language does Prometheus use?"}'

The answer should mention PromQL and time-series data. The sources array should contain the Prometheus document you ingested.

Why do the sources matter?

If the sources array contains documents relevant to your question, the pipeline is working correctly. The LLM is answering based on retrieved context, not just its own training data.

This is the core value of RAG: you control what the AI knows by controlling the documents in the vector database.

Answer doesn't reference your documents?

Check the sources field in the response. If it is empty, the ingest step may not have succeeded. Re-run the ingest command from the first substep, then try querying again.

help me debug why my RAG pipeline returns empty sources from ChromaDB

Your RAG pipeline is working end-to-end. Documents go in, questions come out with grounded answers. But did you notice how the response times vary between queries? Some take 2 seconds, others take 10. In the next step, you will try to figure out why.

Confront the Observability Gap

Your RAG pipeline works. You can feed it documents and get AI-generated answers grounded in those documents. But there is a problem lurking beneath the surface: some queries come back in 2 seconds, others take 10.

Is ChromaDB retrieval the bottleneck? Or is Ollama generation eating up the time? Right now you have no way to answer that question. In this step, you will experience that gap firsthand.

In this step, get ready to:
  • Send 5 rapid queries and observe wildly inconsistent response times.
  • Search Prometheus for RAG-specific metrics and find nothing.
  • Identify the observability gap that the next step will fix.
Send 5 rapid queries and observe the variance

The JSON response from your /query endpoint already includes retrieval_time_ms and generation_time_ms fields. Let's fire off 5 different questions in quick succession and see what those numbers look like.

  • Send the first query by running this command:
curl -s -X POST http://localhost:8080/query \
  -H "Content-Type: application/json" \
  -d '{"question": "What is the main topic?"}' | python3 -m json.tool

Look at the retrieval_time_ms and generation_time_ms values in the response. Note them down mentally, then fire the next four.

  • Send 4 more queries back-to-back by running:
curl -s -X POST http://localhost:8080/query \
  -H "Content-Type: application/json" \
  -d '{"question": "Can you summarize the key points?"}' | python3 -m json.tool
curl -s -X POST http://localhost:8080/query \
  -H "Content-Type: application/json" \
  -d '{"question": "What details are mentioned about implementation?"}' | python3 -m json.tool
curl -s -X POST http://localhost:8080/query \
  -H "Content-Type: application/json" \
  -d '{"question": "Are there any recommendations?"}' | python3 -m json.tool
curl -s -X POST http://localhost:8080/query \
  -H "Content-Type: application/json" \
  -d '{"question": "Explain the relationship between the documents."}' | python3 -m json.tool

You should see generation_time_ms values ranging anywhere from under 2000 to over 10000. The retrieval times stay relatively low, but generation times swing wildly depending on prompt complexity and model state.

Why do the times vary so much?

Ollama's inference speed depends on prompt length, context window usage, and whether model layers are still warm in memory. A short answer might generate in 1-2 seconds. A longer answer with more context might take 8-10 seconds. You can see these numbers per-request, but you have no historical record, no aggregation, and no alerting.

Search Prometheus for RAG metrics

Your stack includes Prometheus, and it is already scraping the app at app:8080. Maybe it already has useful metrics about your RAG pipeline?

Before you check, think about this: do you expect Prometheus to have any custom metrics like request counts or latency histograms for your RAG pipeline?

  • Open your browser and navigate to http://localhost:9090.
  • In the expression input field at the top of the page, type {__name__=~"rag_.*"} and click Execute.

The result is empty. No data found. There are zero metrics with the rag_ prefix.

  • Clear the expression and type python_info instead, then click Execute.

This time you get a result. Prometheus IS scraping the app and collecting default Python process metrics. It just has nothing about your RAG pipeline's actual behaviour.

What metrics does Prometheus have right now?

The prometheus_client library automatically exposes default process metrics like python_info, process_cpu_seconds_total, and process_resident_memory_bytes. These tell you that Python is running, but nothing about what your application is actually doing.

You need custom metrics that track each stage of the RAG pipeline: how long retrieval takes, how long generation takes, how many tokens are produced, and whether requests succeed or fail.

Name the gap

Here is the situation: you can see per-request timing in individual JSON responses, but you have no way to track trends over time, no way to compare retrieval vs. generation latency across many requests, and no way to get alerted when performance degrades.

You are flying blind. The next step fixes this by instrumenting every stage of the pipeline with Prometheus metrics.

Why not just log the times?

Logs work for debugging individual requests after the fact. Metrics give you real-time aggregates, percentiles, and alerting out of the box. You can answer questions like "what is the p95 generation latency over the last 5 minutes?" without grepping through thousands of log lines.

Your observability gap is clear. Next up, you will instrument the retrieval and generation stages with histograms, counters, and labels that turn this blind spot into a fully observable pipeline.

Instrument the Pipeline with Prometheus Metrics

In the previous step you saw the problem firsthand: queries taking anywhere from 2 to 10 seconds with no way to tell whether ChromaDB retrieval or Ollama generation is the bottleneck.

Now you will add Prometheus metrics to each stage of the RAG pipeline. By wrapping retrieval and generation in timed observations, you get per-stage latency data that turns guesswork into diagnosis.

In this step, get ready to:
  • Define Prometheus metric objects (Counters and Histograms) in your app.
  • Instrument the retrieval and generation stages with timing observations and token counting.
  • Rebuild the app container and verify metrics appear in Prometheus.
Define the Prometheus metrics

Prometheus needs you to declare metric objects before you can record data into them. Each metric type serves a different purpose.

  • In app/main.py, find the line MODEL_NAME = "llama3.2:1b".
  • Directly below that line, add the following metric definitions:
# --- Prometheus Metrics ---
REQUEST_COUNT = Counter(
    "rag_requests_total",
    "Total RAG requests",
    ["status"],
)
RETRIEVAL_DURATION = Histogram(
    "rag_retrieval_duration_seconds",
    "Time spent retrieving documents from ChromaDB",
    buckets=[0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5],
)
GENERATION_DURATION = Histogram(
    "rag_generation_duration_seconds",
    "Time spent generating a response from Ollama",
    buckets=[0.5, 1.0, 2.5, 5.0, 10.0, 15.0, 30.0],
)
TOKENS_TOTAL = Counter(
    "rag_tokens_total",
    "Total tokens generated by the LLM",
)

What does this code do?

Each metric object maps to a different observability need:

  • REQUEST_COUNT is a Counter labeled by status (success or error). It tracks total requests so you can compute rates and error ratios.
  • RETRIEVAL_DURATION is a Histogram that buckets retrieval times from 10ms to 2.5s. Histograms let Prometheus compute percentiles (p50, p95, p99).
  • GENERATION_DURATION is a Histogram bucketed from 0.5s to 30s, reflecting the longer time LLM generation takes.
  • TOKENS_TOTAL is a Counter that accumulates every token the LLM produces, letting you compute throughput rates over time.

Why Histograms instead of Gauges?

A Gauge only shows the last value. A Histogram accumulates observations into buckets, so Prometheus can answer questions like "what was the 95th percentile retrieval time over the last 5 minutes?" without losing data between scrapes.

Wire metrics into the query endpoint

Now you need to record values into those metrics every time a request flows through the pipeline. The timing code already exists. You just need to observe durations and count tokens.

  • In app/main.py, scroll to the query_rag function.
  • Find the retrieval section inside the try: block. Replace everything from retrieval_start = time.time() down through context = "\n\n".join(documents) with the following:
        # Step 1: Retrieve relevant documents
        retrieval_start = time.time()
        collection = chroma_client.get_or_create_collection(name=COLLECTION_NAME)
        results = collection.query(
            query_texts=[request.question],
            n_results=3,
        )
        retrieval_duration = time.time() - retrieval_start
        RETRIEVAL_DURATION.observe(retrieval_duration)

        # Build context from retrieved documents
        documents = results["documents"][0] if results["documents"] else []
        context = "\n\n".join(documents)

What changed here?

One new line: RETRIEVAL_DURATION.observe(retrieval_duration). This records the retrieval time into the histogram so Prometheus can compute percentiles. The retrieval timing logic was already there. Now it feeds into a metric instead of just appearing in the JSON response.

  • Next, find everything from generation_duration = time.time() - generation_start down through the end of the except block. Replace that entire section with:
        generation_duration = time.time() - generation_start
        GENERATION_DURATION.observe(generation_duration)

        # Extract response data
        response_data = response.json()
        answer = response_data.get("response", "")
        eval_count = response_data.get("eval_count", 0)
        TOKENS_TOTAL.inc(eval_count)

        REQUEST_COUNT.labels(status="success").inc()

        return QueryResponse(
            answer=answer,
            sources=documents,
            retrieval_time_ms=round(retrieval_duration * 1000, 2),
            generation_time_ms=round(generation_duration * 1000, 2),
        )

    except Exception as e:
        REQUEST_COUNT.labels(status="error").inc()
        raise HTTPException(status_code=500, detail=str(e))

What does this code do?

  • GENERATION_DURATION.observe(generation_duration) records how long the LLM took to respond into its own histogram.
  • eval_count comes from Ollama's JSON response. It tells you exactly how many output tokens the model generated for that request.
  • TOKENS_TOTAL.inc(eval_count) adds those tokens to the running counter so you can compute tokens-per-second rates.
  • REQUEST_COUNT.labels(status="success").inc() and the matching error label in the except block give you a success/error split for every request.
  • Save app/main.py.

Not sure where the code goes?

The metric definitions go between MODEL_NAME = "llama3.2:1b" and the # --- Mount Prometheus metrics endpoint --- comment. Inside query_rag, the retrieval observe call goes right after retrieval_duration = time.time() - retrieval_start, and the generation observe call goes right after generation_duration = time.time() - generation_start.

Use the full-file double check below to confirm everything is in the right place. help me find where to place the Prometheus metrics in my main.py file

✔️ Awesome, I've got everything!

Great. Make sure you have saved app/main.py before moving on to the rebuild.

ⓧ I'd like to double check the full code

Your app/main.py is now longer than 30 lines, so confirm each section is present in order:

  • Imports at the top (time, os, FastAPI, BaseModel, httpx, chromadb, Counter, Histogram, make_asgi_app).
  • Configuration section (OLLAMA_BASE_URL, CHROMA_HOST, CHROMA_PORT, MODEL_NAME).
  • Prometheus Metrics section with REQUEST_COUNT, RETRIEVAL_DURATION, GENERATION_DURATION, TOKENS_TOTAL.
  • Mount Prometheus metrics endpoint (make_asgi_app() mounted at /metrics).
  • ChromaDB Client and Request/Response models (unchanged).
  • The /ingest endpoint (unchanged).
  • The /query endpoint with RETRIEVAL_DURATION.observe(), GENERATION_DURATION.observe(), TOKENS_TOTAL.inc(), and REQUEST_COUNT.labels().inc() calls.
  • The except block with REQUEST_COUNT.labels(status="error").inc() before the raise.
  • The /health endpoint at the bottom (unchanged).
Rebuild the app and verify metrics

The running container still has the old code. You need to rebuild the image and restart the app service for your changes to take effect.

  • Rebuild and restart only the app container by running:
docker compose up -d --build app
  • Wait a few seconds for the container to start, then send a couple of queries to generate metric data:
curl -s -X POST http://localhost:8080/query \
  -H "Content-Type: application/json" \
  -d '{"question": "What topics are covered?"}'
  • You should see an AI-generated answer in the JSON response, just like before.
  • Send one or two more queries with different questions so the metrics have multiple data points.

Before you check the metrics endpoint, predict what you will see there. Will it show just the metric names, or names with actual numeric values?

  • Open http://localhost:8080/metrics in your browser.

You should see rag_retrieval_duration_seconds, rag_generation_duration_seconds, rag_tokens_total, and rag_requests_total listed with real numeric values from your queries.

  • Now open Prometheus at http://localhost:9090.
  • In the query input, type rag_requests_total and click **Execute**.

You should see results with your success count. The blank search from Step 3 now returns real data.

Metrics page shows a connection error?

  • Make sure the rebuild finished successfully. Run docker compose ps and confirm the app container is in a running state, not restarting.
  • If the container is restarting, check the logs with docker compose logs app. A SyntaxError or ImportError means a typo in your metric definitions. Compare against the double-check list above.
  • If Prometheus at http://localhost:9090 shows no data for rag_ metrics, wait 15 seconds. Prometheus scrapes the app every 15 seconds per prometheus.yml.

Still stuck? Help me debug why my RAG metrics are not appearing in Prometheus after rebuilding the app container

Your RAG pipeline is no longer invisible. Every query now records exactly how long retrieval and generation take, how many tokens are produced, and whether it succeeded or failed. Next, you will turn these raw numbers into a visual Grafana dashboard.

Build the Grafana Dashboard and Alert

You now have metrics flowing from your RAG pipeline into Prometheus. The problem is that raw Prometheus queries are hard to read at a glance.

A Grafana dashboard transforms those numbers into visual panels that tell the story of your pipeline's health in real time. In this step, you will connect Grafana to Prometheus, build four panels, and set up an alert that fires when generation latency spikes.

In this step, get ready to:
  • Connect Grafana to Prometheus as a data source.
  • Create a dashboard with four panels covering request rate, latency, token throughput, and errors.
  • Configure an alert rule that fires when generation latency exceeds 10 seconds.
Log into Grafana and add the Prometheus data source

Grafana is already running as part of your Docker Compose stack. You need to point it at Prometheus so it can query the metrics your app is producing.

  • Open your browser and navigate to http://localhost:3000.
  • Log in with username admin and password admin.
  • Grafana will prompt you to change the password. Click Skip for now.

You should see the Grafana home page with a welcome banner.

  • Click Connections in the left-side menu.
  • Click Add new connection.
  • Type Prometheus in the search bar.
  • Click Prometheus data source from the results.
  • Click Add new data source in the upper right.

Grafana opens the Settings tab for the new data source.

  • In the Prometheus server URL field, enter:
http://prometheus:9090

Why prometheus:9090 instead of localhost?

Grafana runs inside the Docker network, where each service is accessible by its container name. From Grafana's perspective, Prometheus lives at http://prometheus:9090. Using localhost would point to the Grafana container itself.

  • Scroll to the bottom and click Save & Test.

You should see a green banner confirming the data source is working.

Seeing a connection error?

Make sure all containers are running by checking docker compose ps in your terminal. The Prometheus container must be in a Running state.

Double-check that you typed http://prometheus:9090 exactly. A common mistake is typing https or using localhost instead of the container name.

Still stuck? Help me debug my Grafana Prometheus data source connection error.

Create the dashboard and add panels

With the data source connected, you can now build a dashboard that visualizes your RAG pipeline metrics. Each panel answers a specific question about pipeline health.

  • Click Dashboards in the main menu.
  • Click New, then select New Dashboard.

You now have an empty dashboard ready for panels. You will add four panels, one at a time.

Panel 1: Request Rate.

  • Click the Add new element icon (blue plus sign) and drag a panel onto the dashboard.
  • Click Configure visualization.
  • In the Queries tab, make sure the data source is set to your Prometheus data source.
  • Enter the following PromQL query:
rate(rag_requests_total[5m])
  • Select Time series as the visualization type.
  • Click Refresh to query the data source.
  • In the panel title field on the right sidebar, enter Request Rate.
  • Click Save to save the panel.

What does this panel show?

The rate() function calculates how many requests per second your RAG API is handling, averaged over the last 5 minutes. You will see separate lines for status="success" and status="error" because rag_requests_total is labeled by status.

You should see at least one data point from your earlier queries. If the graph is empty, send a quick query to your API to generate data.

Panel 2: Retrieval vs Generation Latency (p95).

  • Click Back to dashboard to return to the dashboard view.
  • Click the Add new element icon and drag another panel onto the dashboard.
  • Click Configure visualization.
  • Enter the first PromQL query for retrieval latency:
histogram_quantile(0.95, rate(rag_retrieval_duration_seconds_bucket[5m]))
  • Click Add query to add a second query.
  • Enter the second PromQL query for generation latency:
histogram_quantile(0.95, rate(rag_generation_duration_seconds_bucket[5m]))
  • Select Time series as the visualization type.
  • Click Refresh.
  • Set the panel title to Retrieval vs Generation Latency (p95).
  • Click Save.

What does p95 mean?

The histogram_quantile(0.95, ...) function returns the 95th percentile latency. This means 95% of requests complete faster than the value shown. It is more useful than an average because it reveals tail latency that affects user experience.

By overlaying retrieval and generation on the same panel, you can instantly see which stage dominates total response time.

Panel 3: Token Throughput.

  • Click Back to dashboard, add another panel, and click Configure visualization.
  • Enter this PromQL query:
rate(rag_tokens_total[5m])
  • Select Time series as the visualization type and click Refresh.
  • Set the panel title to Token Throughput.
  • Click Save.

Panel 4: Error Rate.

  • Click Back to dashboard, add a final panel, and click Configure visualization.
  • Enter this PromQL query:
rate(rag_requests_total{status="error"}[5m])
  • Select Time series as the visualization type and click Refresh.
  • Set the panel title to Error Rate.
  • Click Save.

Now save the entire dashboard.

  • Click Back to dashboard to see all four panels together.
  • Click Save dashboard (the floppy disk icon at the top).
  • Enter RAG Pipeline Health as the dashboard title.
  • Click Save.

You should now see your dashboard with four panels. Generate some traffic to confirm they display real data.

  • Send a few queries to your RAG API to generate fresh metrics by running:
curl -s -X POST http://localhost:8080/query \
  -H "Content-Type: application/json" \
  -d '{"question": "What is Python used for?"}'
  • Return to your Grafana dashboard and click the refresh icon (circular arrow) at the top right.

You should see the Request Rate panel tick up and the Latency panel show new data points.

Panels showing No data?

Make sure you sent at least one query to /query after the metrics instrumentation was added in the previous step. Metrics only appear after the first instrumented request.

Check that the time range in Grafana (top right corner) is set to Last 15 minutes or Last 1 hour. A narrow range might miss your data.

Still stuck? Help me debug why my Grafana panels show No data even though I see metrics in Prometheus.

Create a Grafana alert rule for generation latency

Dashboards help you see problems, but you will not be staring at them 24/7. An alert rule watches a metric continuously and fires when a threshold is breached. You will create an alert that fires when p95 generation latency exceeds 10 seconds for 1 minute.

  • In the left-side menu, click Alerts & IRM.
  • Click Alert rules.
  • Click + New alert rule.
  • Enter RAG Generation Latency High as the alert rule name.

Now define the query and condition that triggers the alert.

  • In the Define query and condition section, select your Prometheus data source.
  • Enter this PromQL query:
histogram_quantile(0.95, rate(rag_generation_duration_seconds_bucket[1m]))

Why 1m instead of 5m for the alert?

The alert uses a 1-minute rate window instead of 5 minutes so it reacts faster to latency spikes. A 5-minute window smooths out short bursts, which is fine for dashboards but too slow for alerting.

  • Set the alert condition: in the When dropdown, select last as the reducer function.
  • Set the threshold to is above 10.
  • Click Preview to verify the expression.

Now configure how often the alert evaluates.

  • In Configure alert evaluation behavior, set the evaluation interval to 1m (evaluate every 1 minute).
  • Set the pending period to 1m (the condition must hold for 1 minute before firing).
  • Click Save rule.

Your alert rule is now active. Time to trigger it.

Before you send a complex query, what do you think will happen? Will the alert fire immediately, or will it take about a minute to transition from Pending to Firing?

  • Send a complex query that forces a longer generation time:
curl -s -X POST http://localhost:8080/query \
  -H "Content-Type: application/json" \
  -d '{"question": "Explain in great detail every possible use case for Python in data science, web development, machine learning, automation, and scientific computing, with examples for each"}'
  • Wait about 1-2 minutes for the pending period to elapse.
  • In Grafana, click Alerts & IRM in the left-side menu.
  • Click Alert rules.

You should see your RAG Generation Latency High alert rule showing a Firing state (shown in red). This confirms that your alert correctly detected the latency spike.

What if the alert stays in Normal state?

The llama3.2:1b model is small and might generate responses in under 10 seconds even for complex prompts, depending on your hardware. If you have a fast machine, try sending 3-4 complex queries in rapid succession to increase the chance of hitting the threshold.

You can also temporarily lower the threshold to 5 seconds in the alert rule to make it easier to trigger, then set it back to 10 after confirming it works.

Alert not appearing in the list?

Make sure you clicked Save rule at the bottom of the alert configuration page. Unsaved rules do not evaluate.

Check that your Prometheus data source is still connected by returning to Connections and verifying the green status indicator.

Need more help? Help me figure out why my Grafana alert rule is not firing even though my generation latency seems high.

You have built a complete observability stack for your RAG pipeline. Next up, the Secret Mission challenges you to add cost estimation panels that turn this engineering dashboard into a business intelligence tool.

Secret mission

Token Efficiency and Cost Estimation Panel

Your dashboard tracks speed. But in production, teams care about cost too. Add a tokens-per-second metric and a cost-per-query panel that turns your engineering dashboard into a business intelligence tool.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing cloud costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • 5 Docker containers (Ollama, ChromaDB, FastAPI app, Prometheus, Grafana).
  • A downloaded LLM model (~1.3GB stored in the ollama_data Docker volume).
  • ChromaDB data stored in the chroma_data Docker volume.
  • Local project files (docker-compose.yml, prometheus.yml, app/ directory).

Keep everything running

No action needed. Choose this if you are still actively building or experimenting with queries and your Grafana dashboard.

  • Your 5 containers continue running and consuming system resources (CPU and RAM).
  • The Ollama model stays loaded in the ollama_data volume (~1.3GB on disk).
  • You can keep sending queries to http://localhost:8080/query and watching the Grafana dashboard update.

Pause - I'll come back to this later

Shut down running containers to free up CPU and RAM, but keep all data intact so you can resume later.

  • Stop all containers by running:
docker compose stop
  • Your volumes (including the downloaded model and ChromaDB data) remain on disk.
  • When you are ready to resume, navigate back to your project folder and run:
docker compose start

Delete - I don't want to use this again

Remove all containers, volumes, and local files. You will need to re-pull the Ollama model and re-ingest documents if you start over.

  • Remove all containers and volumes by running:
docker compose down -v
  • This removes all 5 containers, the ollama_data volume (with the ~1.3GB model), and the chroma_data volume.
  • Delete the local project files by running:

macOS

cd ~/Desktop
rm -rf observable-rag-api

Windows

cd ~\Desktop
rmdir /s /q observable-rag-api
  • Docker itself remains installed on your machine. It was a prerequisite for this project, not something the project installed.

Nice Work!

Nice Work!

You just built a production-style observability stack for an AI application, entirely on your local machine. Your RAG API answers questions grounded in real documents, and you can see exactly what is happening inside every request.

You've learned how to:

  • Build a fully local RAG pipeline with Ollama, ChromaDB, and FastAPI that ingests documents and generates AI-grounded answers.
  • Instrument an AI pipeline with Prometheus metrics that track retrieval latency, generation latency, token counts, and error rates across every request.
  • Create a real-time Grafana dashboard with alert rules that fire when your RAG pipeline degrades.
  • Secret Mission: Added token efficiency and cost estimation panels that turn your engineering dashboard into a business intelligence tool.

Ready to quiz yourself?