Build a Local RAG Pipeline

Containerize a 5-service AI pipeline with Docker Compose and live metrics.

Introduction

30 Second Summary

AI chatbots are everywhere, but most send your data to someone else's servers. What if you could run the entire AI pipeline on your own machine, with full visibility into how it performs?

In this project, you will build a local RAG (Retrieval-Augmented Generation) pipeline that ingests documents, answers questions using retrieved context, and displays live performance metrics. Everything runs on your machine using Docker Compose to orchestrate five interconnected services.

What You'll Build

You will feed documents into your pipeline, ask it a question, and watch it retrieve relevant context before generating a grounded answer from a local LLM. Then you will open a live dashboard showing exactly how fast it responded, how many tokens it produced, and whether any errors occurred.

By the end of this project, you'll have:

  • A working document ingestion flow where you feed any text into ChromaDB and see it stored as vector embeddings ready for retrieval.
  • A question-answering endpoint where you ask natural language questions and get answers grounded in your stored documents, powered by a local Ollama model.
  • A Grafana dashboard with four panels visualizing token generation speed, retrieval latency, request counts, and error rates scraped by Prometheus.
  • Secret Mission: Add a health check endpoint that verifies connectivity to both services and configure a Grafana alert that fires when your error rate spikes above 50%.

Are there any prerequisites?

No prior experience with Docker or AI tools is needed. This guide walks you through installing everything from scratch, including Docker Desktop, all project files, and the local LLM model.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

You're going to build a local AI-powered question-answering system that ingests documents, retrieves relevant context from a vector database, generates answers with a local LLM, and displays every request on a real-time metrics dashboard.

Install Docker Desktop and Create the Project

Your RAG pipeline needs five services running together: a FastAPI web server, an Ollama language model, a ChromaDB vector database, Prometheus for metrics collection, and Grafana for dashboards.

Docker Compose orchestrates all of them from a single command. In this step, you'll install Docker Desktop (which includes Docker Compose) and scaffold the project folder where all your code will live.

In this step, get ready to:
  • Install Docker Desktop with WSL 2 support.
  • Verify Docker is working by running a test container.
  • Create the project folder and empty files.
Download and install Docker Desktop

Docker provides the container runtime that all five services depend on. Without it, nothing in this project can run.

Windows

  • Open the Docker Desktop for Windows install page: https://docs.docker.com/desktop/setup/install/windows-install/
  • Click the download button and run the installer.
  • On the Configuration page, ensure Use WSL 2 instead of Hyper-V is selected.
  • Choose the per-user installation (no admin required).
  • Once installation finishes, open the Windows Start menu and launch Docker Desktop.
  • Wait for the Docker Engine to finish starting. You will see a green "running" indicator in the bottom-left of the Docker Desktop window.

macOS

  • Open the Docker Desktop for Mac install page: https://docs.docker.com/desktop/setup/install/mac-install/
  • Download the .dmg file for your chip (Apple Silicon or Intel).
  • Double-click Docker.dmg and drag the Docker icon to your Applications folder.
  • Open Docker.app from your Applications folder.
  • Accept the service agreement when prompted. macOS may ask for your password during initial setup.
  • Wait for the Docker Engine to finish starting. You will see a green whale icon in your menu bar.

What is Docker Desktop?

Docker Desktop is an application that bundles the Docker Engine, Docker CLI, and Docker Compose into one install. It runs a lightweight Linux virtual machine where your containers live.

Docker Compose is the specific tool you'll use to spin up all five services at once from a single YAML file.

Verify Docker is running
  • Open a terminal. Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or PowerShell (Windows) and press Enter to open it.

Before you run the next command, what do you expect to see if Docker is working correctly?

  • Test that Docker is working by running this command:
docker run hello-world

You should see a message that says "Hello from Docker!" followed by an explanation of what just happened.

✔️ I see "Hello from Docker!"

Docker pulled the test image and ran it successfully. Your container runtime is ready.

ⓧ I see "command not found" or "is not recognized"

Docker CLI is not on your system PATH. Try these fixes:

  • Make sure Docker Desktop is fully started (look for the green running indicator).
  • Close your terminal completely and open a fresh one. Docker Desktop adds itself to PATH on first launch, but existing terminal sessions won't pick it up.
  • On Windows, restart your computer if a fresh terminal still does not find docker.

Run docker run hello-world again after trying the fixes above.

ⓧ I see a "permission denied" or daemon error

The Docker Engine is not running yet. Try these fixes:

  • Open Docker Desktop and wait for it to fully start (the status should say "running").
  • On Windows, confirm WSL 2 is enabled. Open PowerShell as Administrator and run wsl --install, then restart your machine.
  • On macOS, check that you accepted the service agreement when Docker Desktop launched for the first time.

Run docker run hello-world again after trying the fixes above.

Create the project folder structure

Your RAG pipeline needs a specific folder layout. The app/ directory holds your Python application code, while the root contains Docker and monitoring config files.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and the app subfolder by running these commands:
mkdir rag-pipeline
cd rag-pipeline
mkdir app
  • Confirm the folders exist by running:
ls

You should see app listed in the output.

  • Create the empty project files by running these commands:

macOS

# Create files in the app/ directory
touch app/main.py app/requirements.txt app/Dockerfile

# Create files in the root project directory
touch docker-compose.yml prometheus.yml

Windows

# Create files in the app/ directory
New-Item -ItemType "File" -Path "app/main.py", "app/requirements.txt", "app/Dockerfile"

# Create files in the root project directory
New-Item -ItemType "File" -Path "docker-compose.yml", "prometheus.yml"
  • Confirm all files were created by running:
ls
ls app

The first ls should show app, docker-compose.yml, and prometheus.yml. The second ls app should show Dockerfile, main.py, and requirements.txt.

Don't see all the files?

  • Make sure you ran the file creation commands from inside the rag-pipeline folder. Run pwd (macOS) or cd (Windows, no arguments) to check your current location.
  • On Windows, New-Item prints output for each file created. That is normal and does not indicate an error.

Help me figure out why my project files are missing.

💡 What does this folder structure look like?

Your project now has this layout:

rag-pipeline/
  docker-compose.yml
  prometheus.yml
  app/
    main.py
    requirements.txt
    Dockerfile

All files are empty for now. You will fill them in over the next steps.

Your environment is ready and the project skeleton is in place. Next up, you'll write the FastAPI application that powers the entire RAG pipeline.

Write the FastAPI RAG Application

Your project folder is ready, but the files are empty. Before you can containerize anything, you need the application code itself.

In this step, you will write a complete FastAPI application that implements the full RAG pipeline. It will accept documents, store them in a vector database, retrieve relevant context when queried, and call a local LLM to generate answers. You will also instrument every request with 9 Prometheus metrics so you can observe performance under load later.

In this step, get ready to:
  • Define all Python dependencies and 9 Prometheus metrics for your RAG pipeline.
  • Write a /ingest endpoint that stores documents in ChromaDB.
  • Write a /query endpoint that retrieves context and generates LLM-powered answers.
Write the dependencies and define Prometheus metrics

Your FastAPI app needs five Python packages. The chromadb-client package is a lightweight HTTP-only client that connects to a ChromaDB server. Since ChromaDB will run as its own Docker container, you do not need the full chromadb library.

  • Open app/requirements.txt in VS Code and add your project dependencies by pasting this content:
fastapi==0.141.1
uvicorn
chromadb-client==1.5.9
prometheus-client==0.26.0
httpx

What does each dependency do?

  • fastapi==0.141.1 is the web framework for your API endpoints.
  • uvicorn is the ASGI server that runs your FastAPI app.
  • chromadb-client==1.5.9 is the thin HTTP client that talks to a ChromaDB server without bundling the full database.
  • prometheus-client==0.26.0 provides Counter, Histogram, and the ASGI metrics app.
  • httpx is the HTTP client you will use to call Ollama's generate API.
  • Save app/requirements.txt.
  • Confirm the file saved correctly by running:
cat app/requirements.txt

You should see all five package lines printed in your terminal.

Seeing an empty output?

Make sure you saved the file in VS Code (Cmd+S on macOS, Ctrl+S on Windows). Also confirm your terminal is inside the rag-pipeline folder. Run pwd to check.

help me debug why cat app/requirements.txt shows nothing

Now you will start building app/main.py. The first chunk sets up the FastAPI app and defines the first four Prometheus metrics.

  • Open app/main.py in VS Code and paste the following imports, app creation, and metric definitions:
import time
import httpx
import chromadb
from fastapi import FastAPI
from pydantic import BaseModel
from prometheus_client import Counter, Histogram, make_asgi_app

app = FastAPI(title="RAG Pipeline")

# --- Prometheus Metrics ---
rag_requests_total = Counter(
    "rag_requests_total",
    "Total RAG query requests",
    ["status"],
)
rag_request_duration_seconds = Histogram(
    "rag_request_duration_seconds",
    "Total end-to-end request duration",
)
rag_retrieval_duration_seconds = Histogram(
    "rag_retrieval_duration_seconds",
    "Time spent retrieving documents from ChromaDB",
)
rag_generation_duration_seconds = Histogram(
    "rag_generation_duration_seconds",
    "Time spent generating response from Ollama",
)
rag_ttft_seconds = Histogram(
    "rag_ttft_seconds",
    "Time to first token from Ollama prompt_eval_duration",
)

What does this code do?

The imports pull in everything you need: time for measuring durations, httpx for calling the Ollama API, chromadb for the vector database client, and prometheus_client for instrumenting metrics.

The five metrics defined here track total requests (with a success/error label), end-to-end request duration, document retrieval time, generation time, and time-to-first-token. Each is either a Counter (goes up) or a Histogram (records distributions of values).

  • Save app/main.py. You should see 27 lines of code in your editor.

Next, add the remaining three metrics, mount the Prometheus endpoint, connect to ChromaDB, and define your request models.

  • In app/main.py, paste the following code directly below the rag_ttft_seconds definition:
rag_tokens_total = Counter(
    "rag_tokens_total",
    "Total output tokens generated by Ollama",
)
rag_prompt_tokens_total = Counter(
    "rag_prompt_tokens_total",
    "Total prompt tokens processed by Ollama",
)
rag_tokens_per_second = Histogram(
    "rag_tokens_per_second",
    "Token generation rate from Ollama",
)
rag_tpot_seconds = Histogram(
    "rag_tpot_seconds",
    "Time per output token from Ollama",
)

# Expose metrics at /metrics for Prometheus to scrape
metrics_app = make_asgi_app()
app.mount("/metrics", metrics_app)

# Connect to ChromaDB vector store
chroma_client = chromadb.HttpClient(host="chromadb", port=8000)
collection = chroma_client.get_or_create_collection(name="documents")


# --- Request Models ---
class IngestRequest(BaseModel):
    documents: list[str]
    ids: list[str]


class QueryRequest(BaseModel):
    query: str
    n_results: int = 3

What does this code do?

The four remaining metrics track total output tokens, total prompt tokens, tokens-per-second generation rate, and time-per-output-token (TPOT). TPOT is the inverse of tokens/sec and measures how fast the LLM produces each individual token.

make_asgi_app() creates a small ASGI app that serves all your metrics in Prometheus exposition format. Mounting it at /metrics means Prometheus can scrape that path to collect data.

The ChromaDB client connects to hostname chromadb on port 8000. This hostname will not resolve yet because the ChromaDB container does not exist. Docker Compose will create it in the next step. The two Pydantic models define the shape of incoming request bodies for your endpoints.

  • Save app/main.py. Scroll through the file and count all 9 metric names: rag_requests_total, rag_request_duration_seconds, rag_retrieval_duration_seconds, rag_generation_duration_seconds, rag_ttft_seconds, rag_tokens_total, rag_prompt_tokens_total, rag_tokens_per_second, and rag_tpot_seconds.

Missing a metric or seeing a syntax error?

Make sure the second chunk starts right after the closing parenthesis of rag_ttft_seconds. There should be no blank line or duplicate code between the two chunks. Each Counter and Histogram definition must end with a closing ) on its own line.

help me fix a syntax error in my Prometheus metric definitions in main.py

Write the /ingest endpoint

The /ingest endpoint is how documents enter your RAG pipeline. It accepts a list of text documents and their IDs, then stores them in ChromaDB where they can be retrieved later.

  • In app/main.py, paste the following code at the bottom of the file, below the QueryRequest class:
# --- Endpoints ---
@app.post("/ingest")
def ingest(request: IngestRequest):
    collection.add(documents=request.documents, ids=request.ids)
    return {"status": "ok", "count": len(request.documents)}

What does this code do?

collection.add() sends documents to ChromaDB. ChromaDB automatically generates vector embeddings for each document using its built-in embedding model. The IDs let you reference or update specific documents later.

The response confirms success and tells you how many documents were stored.

  • Save app/main.py. You should now see a def ingest function near the bottom of the file.
Write the /query endpoint

The /query endpoint is the heart of the RAG pipeline. It retrieves relevant documents from ChromaDB, builds a prompt with that context, sends it to the local Ollama LLM, and records detailed timing metrics from every step.

  • In app/main.py, paste the following code below the ingest function. This is the first half of the /query endpoint (retrieval and generation):
@app.post("/query")
def query(request: QueryRequest):
    try:
        # Track end-to-end request time
        request_start = time.time()

        # Retrieve relevant documents from ChromaDB
        start = time.time()
        results = collection.query(
            query_texts=[request.query],
            n_results=request.n_results,
        )
        retrieval_duration = time.time() - start
        rag_retrieval_duration_seconds.observe(retrieval_duration)

        # Build context from retrieved documents
        context = "\n".join(results["documents"][0])

        # Generate answer with Ollama
        prompt = f"Context:\n{context}\n\nQuestion: {request.query}\nAnswer:"

        start = time.time()
        response = httpx.post(
            "http://ollama:11434/api/generate",
            json={
                "model": "llama3.2:1b",
                "prompt": prompt,
                "stream": False,
            },
            timeout=120.0,
        )
        generation_duration = time.time() - start
        rag_generation_duration_seconds.observe(generation_duration)

        data = response.json()

What does this code do?

collection.query() searches ChromaDB for documents similar to the user's question. It returns the top N matches ranked by similarity.

The retrieved documents are joined into a context string and injected into a prompt template. The prompt gives the LLM both the relevant context and the user's question.

The httpx.post call sends this prompt to Ollama's /api/generate endpoint with stream: False. This returns all timing fields (eval_count, eval_duration, prompt_eval_duration) in a single JSON response instead of streaming chunks.

  • Save app/main.py. The file will show an incomplete function at this point. That is expected because you are about to add the second half.
  • Paste the following code directly below the data = response.json() line (still inside the try block, same indentation level):
        # Record Ollama timing metrics (all durations in nanoseconds)
        if data.get("prompt_eval_duration"):
            rag_ttft_seconds.observe(data["prompt_eval_duration"] / 1e9)

        if data.get("eval_count"):
            rag_tokens_total.inc(data["eval_count"])

        if data.get("prompt_eval_count"):
            rag_prompt_tokens_total.inc(data["prompt_eval_count"])

        if data.get("eval_count") and data.get("eval_duration") and data["eval_duration"] > 0:
            tokens_per_sec = data["eval_count"] / data["eval_duration"] * 1e9
            rag_tokens_per_second.observe(tokens_per_sec)
            # TPOT = time per output token (inverse of tokens/sec)
            tpot = data["eval_duration"] / data["eval_count"] / 1e9
            rag_tpot_seconds.observe(tpot)

        # Record end-to-end request duration
        rag_request_duration_seconds.observe(time.time() - request_start)
        rag_requests_total.labels(status="success").inc()

        return {
            "answer": data.get("response", ""),
            "sources": results["documents"][0],
            "metrics": {
                "retrieval_duration_s": round(retrieval_duration, 3),
                "generation_duration_s": round(generation_duration, 3),
                "eval_count": data.get("eval_count"),
                "prompt_eval_count": data.get("prompt_eval_count"),
            },
        }
    except Exception as e:
        rag_requests_total.labels(status="error").inc()
        return {"error": str(e)}

What does this code do?

Ollama returns all durations in nanoseconds. Dividing by 1e9 converts them to seconds for Prometheus. The tokens-per-second formula is eval_count / eval_duration * 1e9 and TPOT is the inverse: eval_duration / eval_count / 1e9.

The rag_request_duration_seconds observation wraps the entire try block, capturing retrieval + generation + overhead in a single number. On success, the endpoint returns the LLM's answer, the source documents, and timing metrics. If anything fails, the except block increments the error counter and returns the error message.

  • Save app/main.py.

Before you move on, predict what you expect to see when you scroll through the file. Can you spot the /ingest endpoint, the /query endpoint, and all 7 metric definitions?

  • Scroll through app/main.py from top to bottom. Confirm you can see: the 7 metric variables, the /metrics mount, the ChromaDB client, the IngestRequest and QueryRequest models, the /ingest endpoint, and the full /query endpoint ending with return {"error": str(e)}.

Seeing indentation errors or misaligned code?

The most common issue is the second /query chunk not being indented correctly. Everything inside the try: block must be indented with 8 spaces (two levels: one for the function, one for try). The except Exception as e: line should align with try: at 4 spaces.

help me fix indentation in my /query endpoint in main.py

✔️ Awesome, I've got everything!

Great. Double check you have saved app/main.py and app/requirements.txt.

ⓧ I'd like to double check the full code

Scroll through app/main.py and confirm each section is present in order:

  • Imports (6 lines): time, httpx, chromadb, FastAPI, BaseModel, Counter/Histogram/make_asgi_app.
  • App creation: app = FastAPI(title="RAG Pipeline").
  • 9 Prometheus metrics: rag_requests_total through rag_tpot_seconds.
  • Metrics mount: app.mount("/metrics", metrics_app).
  • ChromaDB client: chromadb.HttpClient(host="chromadb", port=8000) and collection creation.
  • Two Pydantic models: IngestRequest and QueryRequest.
  • /ingest endpoint: collection.add() returning status and count.
  • /query endpoint: request_start timing, retrieval, prompt building, Ollama call, TTFT + TPOT + tokens/sec recordings, rag_request_duration_seconds observation, success return, and except block.

For app/requirements.txt, confirm you have exactly these 5 lines:

fastapi==0.141.1
uvicorn
chromadb-client==1.5.9
prometheus-client==0.26.0
httpx

Why can't you run this code yet?

The application references chromadb and ollama as hostnames. These names only resolve inside a Docker Compose network where each service is reachable by its service name. Without Docker Compose running, there is no DNS to resolve them.

In the next step, you will write the Dockerfile and docker-compose.yml to spin up all 5 services on a shared network. Then these hostnames will resolve and your app will connect.

Your RAG application code is complete. Next up, you will containerize this app and launch the full 5-service stack with Docker Compose.

Containerize and Launch the Stack

Your FastAPI app references chromadb and ollama as hostnames in its code. Those services do not exist yet, so the app cannot run on its own.

Docker Compose solves this by creating a shared network where each service is reachable by its service name. In this step, you will containerize the FastAPI app and define all 5 services so they can communicate automatically.

In this step, get ready to:
  • Write a Dockerfile to containerize the FastAPI app.
  • Define all 5 services in a Docker Compose configuration.
  • Launch the full stack and pull the LLM model.
Write the Dockerfile

A Dockerfile is a recipe that tells Docker how to build an image for your application. It specifies the base operating system, installs dependencies, copies your code, and defines the command to start the app.

  • In VS Code, open the empty app/Dockerfile file and paste the following code:
FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

What does this Dockerfile do?

  • FROM python:3.12-slim starts from a lightweight Python 3.12 image, keeping the final container small.
  • COPY requirements.txt and RUN pip install install your dependencies first. Docker caches this layer, so rebuilds are fast unless dependencies change.
  • COPY . . copies main.py and everything else into the container.
  • CMD starts uvicorn serving your FastAPI app on port 8000, listening on all interfaces so Docker can route traffic to it.
  • Save app/Dockerfile.
  • Test the build by running this command from the rag-pipeline folder:
docker build -t rag-fastapi ./app

You should see Docker download the Python base image, install your pip packages, and finish with a success status. The build takes about a minute the first time because it downloads the base image.

Seeing a build error?

  • Make sure Docker Desktop is running. You can check by looking for the Docker whale icon in your system tray.
  • Confirm you are running the command from inside the rag-pipeline folder, not from rag-pipeline/app.
  • Check that app/requirements.txt is saved and not empty.

Why is my Docker build failing?

Define the Docker Compose services

Docker Compose lets you define multiple containers in a single YAML file. Each service gets a hostname matching its name, so your FastAPI code can reach chromadb:8000 and ollama:11434 without any manual networking setup.

  • Open the empty prometheus.yml file in the root rag-pipeline folder and paste the following:
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "fastapi-rag"
    static_configs:
      - targets: ["fastapi:8000"]
    metrics_path: "/metrics"

What does this config do?

This tells Prometheus to scrape your FastAPI app's /metrics endpoint every 15 seconds. The target fastapi:8000 uses the Docker Compose service name as a hostname, which Docker resolves automatically inside the shared network.

  • Save prometheus.yml.
  • Now open the empty docker-compose.yml file in the same rag-pipeline folder and paste the core application services:
services:
  fastapi:
    build: ./app
    ports:
      - "8000:8000"
    depends_on:
      - chromadb
      - ollama

  ollama:
    image: ollama/ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama

  chromadb:
    image: chromadb/chroma
    ports:
      - "8001:8000"
    volumes:
      - chroma_data:/data

What are these three services?

  • fastapi builds your app from the Dockerfile you just wrote. It depends on chromadb and ollama so Docker starts those first.
  • ollama runs the Ollama LLM server. The ollama_data volume persists downloaded models across container restarts.
  • chromadb runs the ChromaDB vector database. Host port 8001 maps to container port 8000 to avoid conflicting with FastAPI's port 8000.
  • Directly below the chromadb service (still inside the same file), add the observability services and volume definitions:
  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin

volumes:
  ollama_data:
  chroma_data:

What do these services add?

  • prometheus scrapes metrics from your FastAPI app using the config you just wrote. It mounts prometheus.yml from your host into the container.
  • grafana provides the dashboarding UI. The GF_SECURITY_ADMIN_PASSWORD environment variable sets the admin login password.
  • The volumes: block at the bottom declares named volumes that persist data for Ollama and ChromaDB across container restarts.
  • Save docker-compose.yml.

✔️ Awesome, I've got everything!

Great. Confirm you have saved both docker-compose.yml and prometheus.yml.

ⓧ I'd like to double check the full code

Your docker-compose.yml should contain both code blocks combined in order. Scroll up and confirm:

  • The file starts with services: and the first service is fastapi.
  • The three core services (fastapi, ollama, chromadb) come first.
  • The two observability services (prometheus, grafana) come next.
  • The file ends with the volumes: block declaring ollama_data and chroma_data.

Your prometheus.yml should have a scrape_interval: 15s and a single job targeting fastapi:8000 at the /metrics path.

Launch the stack and pull the model

All three configuration files are ready. Time to bring up the entire stack in one command.

  • Launch all 5 services by running this command from the rag-pipeline folder:
docker compose up --build -d

Docker builds your FastAPI image, pulls the Ollama, ChromaDB, Prometheus, and Grafana images, then starts all 5 containers in the background. This takes a few minutes on the first run while images download.

  • Confirm all containers are running by running:
docker compose ps

You should see 5 services listed, all with a status of "running" or "Up".

Why Docker Compose instead of running each container separately?

Docker Compose creates a shared network where every service can reach the others by name. Without it, you would need to manually create a Docker network, start each container with --network flags, and coordinate startup order yourself.

The depends_on field ensures ChromaDB and Ollama start before FastAPI attempts to connect to them.

The Ollama container starts without any AI models installed. You need to pull a model before the /query endpoint can work.

  • Pull the llama3.2:1b model into the Ollama container by running:
docker compose exec ollama ollama pull llama3.2:1b

What is this command doing?

docker compose exec runs a command inside a running container. Here it tells the Ollama server to download the llama3.2:1b model (about 1.3GB). The model is stored in the ollama_data volume, so it persists even if you restart the container.

You will see a progress bar as the model downloads. Wait for it to complete before continuing.

Before you check the result, what do you think you will see if you open http://localhost:8000/docs in your browser?

  • Open http://localhost:8000/docs in your browser.

You should see the FastAPI Swagger UI with three endpoints listed: /ingest, /query, and /metrics. This confirms your containerized app is running and reachable.

Don't see the Swagger UI?

  • Run docker compose ps and check that the fastapi container status is "running". If it exited, run docker compose logs fastapi to see the error.
  • Make sure nothing else is using port 8000 on your machine. Stop any other local servers first.
  • If you see a connection error, wait 10-15 seconds. The FastAPI container may still be starting up.

Why can't I see the FastAPI Swagger UI at localhost:8000?

Your 5-service stack is live. The FastAPI app can now reach ChromaDB and Ollama by their service names, and Prometheus is already scraping your metrics endpoint. Next up, you will test the full RAG pipeline by ingesting documents and asking questions.

Test the RAG Pipeline End-to-End

Your Docker Compose stack is running five services, and the FastAPI Swagger UI confirms the app is reachable. But you have not actually proven the RAG pipeline works end-to-end.

This step exercises the full flow: query without context, ingest documents, then query again to see how retrieval-augmented generation produces better answers. The difference is the core reason RAG exists.

In this step, get ready to:
  • Query the LLM without any documents and observe its vague answer.
  • Ingest sample documents into ChromaDB via the /ingest endpoint.
  • Query again and confirm the answer is grounded in your documents, then verify Prometheus is scraping metrics.
Query without any documents

Right now, the ChromaDB collection is empty. The /query endpoint will still work because it retrieves whatever it can from ChromaDB, builds a prompt, and sends it to the LLM.

Before you run this, think: what kind of answer do you expect the LLM to give about a fictional internal project it has never seen before?

  • Send a query to the /query endpoint by running this command:
curl -X POST http://localhost:8000/query \
  -H "Content-Type: application/json" \
  -d '{"query": "What does Project Aurora do?"}'

The response comes back with an answer field, but the LLM's answer is vague or entirely made up. It might hallucinate details about a space mission or an unrelated software project.

Why is the answer so unreliable?

The LLM has no context about "Project Aurora" because ChromaDB returned nothing useful. Without retrieved documents, the model falls back on its training data and guesses.

This is exactly the problem RAG solves. By feeding relevant documents into the prompt, you ground the LLM's answer in real information instead of hallucinations.

Ingest sample documents

Now you will give the pipeline something to retrieve. The /ingest endpoint accepts a list of documents and IDs, then stores them as vector embeddings in ChromaDB.

  • Ingest three documents about a fictional project by running this command:
curl -X POST http://localhost:8000/ingest \
  -H "Content-Type: application/json" \
  -d '{
    "documents": [
      "Project Aurora is an internal tool that automates weekly team reports by pulling data from Jira and Slack.",
      "Project Aurora was started in March 2025 by the platform engineering team to reduce manual reporting overhead.",
      "Project Aurora sends a summary email every Friday at 5pm with sprint velocity, blocked tickets, and team sentiment scores."
    ],
    "ids": ["doc1", "doc2", "doc3"]
  }'

What does this command do?

Each string in the documents array gets converted into a vector embedding and stored in ChromaDB's documents collection. The ids array provides unique identifiers so you can reference or delete them later.

You should see a JSON response confirming the ingestion:

{"status": "ok", "count": 3}

Seeing a connection error?

If you get a connection refused or timeout error, the ChromaDB container may have restarted. Check container status with docker compose ps and confirm the chromadb service shows as running.

help me debug a connection error when calling my FastAPI /ingest endpoint

Query again with documents ingested

The same question, but now ChromaDB has three documents about Project Aurora to retrieve. The /query endpoint will find the most relevant ones and inject them into the prompt before the LLM generates its answer.

What do you expect the answer to look like this time?

  • Run the same query again by running this command:
curl -X POST http://localhost:8000/query \
  -H "Content-Type: application/json" \
  -d '{"query": "What does Project Aurora do?"}'

This time the response includes a grounded answer mentioning automating weekly reports, pulling data from Jira and Slack, and sending Friday emails. The sources field shows which documents were retrieved, and metrics shows eval_count and generation timing.

What changed between the two queries?

The LLM itself did not change. The only difference is that ChromaDB now returns relevant context that gets injected into the prompt. The LLM reads that context and generates an answer based on it instead of guessing.

This is the "Retrieval" in Retrieval-Augmented Generation. Better documents in, better answers out.

  • Confirm Prometheus is scraping your metrics by opening http://localhost:9090 in your browser.
  • In the query input field at the top, type rag_requests_total and click **Execute**.

You should see at least two entries: one with status="success" showing a value of 2 or higher (from your two /query calls). This confirms Prometheus is successfully scraping the FastAPI /metrics endpoint every 15 seconds.

Not seeing any metrics in Prometheus?

Wait 15-20 seconds and click **Execute** again. Prometheus scrapes at 15-second intervals, so your most recent requests may not appear immediately.

If nothing appears after a minute, check that the Prometheus target is healthy by navigating to **Status** then **Targets** in the Prometheus UI. The fastapi-rag target should show state UP.

help me debug why Prometheus is not scraping metrics from my FastAPI container

Your RAG pipeline is proven end-to-end. Documents go in, grounded answers come out, and every request generates metrics. Next up, you will turn those raw metrics into a visual Grafana dashboard.

Build a Grafana Dashboard

Your RAG pipeline is generating metrics with every request, and Prometheus is collecting them behind the scenes. But raw metric values in a query interface are hard to read and impossible to monitor at a glance.

Grafana turns those numbers into a live visual dashboard. In this step, you will connect Grafana to Prometheus and build panels that show request rates, latency percentiles, and token throughput in real time.

In this step, get ready to:
  • Connect Prometheus as a data source in Grafana.
  • Build a dashboard with 4 metric visualization panels.
  • Generate traffic and confirm the panels display live data.
Log in to Grafana and add Prometheus as a data source

Grafana is already running as part of your Docker Compose stack. You need to tell it where to find the metric data by connecting it to Prometheus.

  • Open your browser and navigate to http://localhost:3000.
  • Log in with username admin and password admin.
  • Grafana may prompt you to change the password. You can click Skip for now since this is a local development instance.
  • In the left-side menu, click Connections.
  • Click Data sources.
  • Click Add new data source.
  • Select Prometheus from the list.
  • In the Prometheus server URL field, enter http://prometheus:9090.
  • Scroll down and click Save & Test.

You should see a green success message confirming the data source is working.

Why http://prometheus:9090 and not localhost?

Grafana is running inside a Docker container. From inside the container, localhost refers to Grafana's own container, not your host machine. The service name prometheus resolves to the Prometheus container's IP address through Docker's internal DNS.

🙋‍♀️ Save & Test failed?

Make sure all containers are still running. Return to your terminal and run docker compose ps to check. If prometheus is not listed as running, restart the stack with docker compose up -d.

Also double-check the URL field. It must be exactly http://prometheus:9090 with no trailing slash.

help me troubleshoot my Grafana Prometheus data source connection

Create a dashboard with 4 metric panels

Now that Grafana can read from Prometheus, you will build a dashboard with four panels. Each panel uses a PromQL query to visualize a different aspect of your RAG pipeline's performance.

  • In the left-side menu, click Dashboards.
  • Click New and select New Dashboard.
  • Click the Add new element icon (blue plus sign) and click a panel onto the dashboard.
  • On the new panel, click Configure visualization.

The panel editor opens. This is where you write PromQL queries and choose how to display the results.

  • In the query editor at the bottom, switch to Code mode (if not already selected).
  • Enter the following PromQL query for your first panel:
rate(rag_requests_total[5m])
  • Click Run queries (or press Shift+Enter) to see a preview.
  • In the Legend field below the query, enter {{status}} to split the graph by the status label.
  • Set the panel title to Request Rate in the panel options on the right.
  • Click Save at the top of the panel editor.

Your first panel now shows the rate of requests to your RAG pipeline, split by success and error.

What does rate(rag_requests_total[5m]) mean?

The rate() function calculates the per-second increase of a counter over a time window. [5m] means it looks at the last 5 minutes of data to compute the average rate.

The {{status}} legend template splits the line into separate series for each label value (success and error), so you can see them independently on the graph.

Now add the remaining three panels. You will repeat a similar process for each one.

  • Click Back to dashboard to return to the dashboard view.
  • Click the Add new element icon again and click a new panel onto the dashboard.
  • Click Configure visualization on the new panel.
  • In Code mode, enter this query for retrieval latency:
histogram_quantile(0.95, rate(rag_retrieval_duration_seconds_bucket[5m]))
  • Set the panel title to Retrieval Latency (p95).
  • Click Save.
  • Click Back to dashboard, add another panel, and click Configure visualization.
  • Enter this query for generation latency:
histogram_quantile(0.95, rate(rag_generation_duration_seconds_bucket[5m]))
  • Set the panel title to Generation Latency (p95).
  • Click Save.
  • Click Back to dashboard, add one final panel, and click Configure visualization.
  • Enter this query for token throughput:
histogram_quantile(0.5, rate(rag_tokens_per_second_bucket[5m]))
  • Set the panel title to Tokens Per Second.
  • Click Save.
  • Click Back to dashboard.
  • Enter a title for the dashboard such as RAG Pipeline Metrics and click Save.
  • Click Exit edit.

Your dashboard now has 4 panels. They may show "No data" if not enough recent requests have generated metrics. The next substep fixes that.

What do p95 and p50 mean?

The histogram_quantile() function calculates percentiles from histogram data. A p95 value means 95% of requests completed faster than this number. A p50 (median) shows the typical value.

You use p95 for latency panels because it shows the worst-case experience for most users. You use p50 for tokens per second because it shows the typical throughput rate.

Generate traffic and verify the dashboard

The panels need recent metric data to display anything useful. You will send several queries to the RAG pipeline, then watch the dashboard come alive.

  • Send 5-10 queries to your RAG pipeline by running these commands in your terminal:
curl -s -X POST http://localhost:8000/query \
  -H "Content-Type: application/json" \
  -d '{"query": "What topics are covered?"}'

curl -s -X POST http://localhost:8000/query \
  -H "Content-Type: application/json" \
  -d '{"query": "Give me a summary."}'

curl -s -X POST http://localhost:8000/query \
  -H "Content-Type: application/json" \
  -d '{"query": "What are the key points?"}'

curl -s -X POST http://localhost:8000/query \
  -H "Content-Type: application/json" \
  -d '{"query": "Tell me more details."}'

curl -s -X POST http://localhost:8000/query \
  -H "Content-Type: application/json" \
  -d '{"query": "What else can you tell me?"}'

What is happening with each request?

Each request hits the /query endpoint, which retrieves documents from ChromaDB, sends the context to Ollama, and records all 7 metrics. Prometheus scrapes these metrics every 15 seconds, making them available to Grafana.

Each request takes 5-30 seconds because the local LLM needs time to generate a response. Wait for each one to return before checking the dashboard.

Before you check the dashboard, what do you expect to see on the Request Rate panel?

  • Return to your Grafana dashboard at http://localhost:3000.
  • Navigate to Dashboards in the left menu, then click your RAG Pipeline Metrics dashboard.
  • Click the refresh icon in the top-right toolbar to reload all panels.

You should see the Request Rate panel showing a line for success requests. The Retrieval Latency and Generation Latency panels display timing data. The Tokens Per Second panel shows the LLM's throughput rate.

Panels still showing No data?

Wait at least 30 seconds after your last query for Prometheus to scrape the updated metrics (scrape interval is 15 seconds). Then click the refresh icon again.

If the panels remain empty, check that the time range in the top-right corner of Grafana is set to Last 15 minutes or Last 5 minutes. A time range that is too narrow may miss the data.

help me debug why my Grafana panels show no data

Your RAG pipeline now has production-grade observability. You can see request rates, spot slow queries, and monitor throughput at a glance. Next up, the Secret Mission challenges you to add a health check endpoint and automated alerting.

Secret mission

Health Check Endpoint and Error Alerting

Your RAG pipeline runs, but how would you know if one of its dependencies goes down? Add a /health endpoint that checks connectivity to both Ollama and ChromaDB, then configure a Grafana alert that fires when your error rate exceeds 50%.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally, so there are no ongoing costs beyond the RAM your containers use while running.

Resources you used:

  • Docker containers: 5 services (FastAPI, Ollama, ChromaDB, Prometheus, Grafana).
  • Docker volumes: ollama_data (~1.3GB for the LLM model) and chroma_data (stores your ingested documents).
  • Docker images: custom FastAPI image, plus pulled images for Ollama, ChromaDB, Prometheus, and Grafana.
  • Local project files: the rag-pipeline folder on your Desktop.

Keep everything running

No action needed. Choose this if you are still actively building or experimenting with your RAG pipeline.

  • Your 5 containers continue running and use approximately 2-3GB of RAM while idle.
  • The Ollama model stays cached in the ollama_data volume, so queries remain fast.
  • Your Grafana dashboard and alert rule persist across Docker Desktop restarts.

Pause - I'll come back to this later

Shut down running containers to free up memory. Your data and configuration remain intact for when you return.

  • Open your terminal and navigate to the project folder by running:
  • Stop all containers without removing them.
cd ~/Desktop/rag-pipeline
docker compose stop
  • When you are ready to resume, start everything back up by running:
cd ~/Desktop/rag-pipeline
docker compose start

Your volumes, images, and Grafana configuration all remain intact.

Delete - I don't want to use this again

Remove all project containers, volumes, images, and local files to reclaim disk space.

  • Navigate to the project folder and remove all containers, networks, and volumes by running:
cd ~/Desktop/rag-pipeline
docker compose down -v

What does this command do?

The -v flag tells Docker Compose to also remove the named volumes (ollama_data and chroma_data). Without it, your ~1.3GB model and ingested documents would remain on disk.

  • Remove all unused Docker images to reclaim additional disk space by running:
docker system prune -a
  • When prompted to confirm, type y and press Enter.
  • Delete the local project folder. On macOS or Linux, run:
rm -rf ~/Desktop/rag-pipeline
  • On Windows, run:
rmdir /s /q %USERPROFILE%\Desktop\rag-pipeline
  • Confirm deletion by running ls ~/Desktop (macOS/Linux) or dir %USERPROFILE%\Desktop (Windows). The rag-pipeline folder should no longer appear.

Nice Work!

Nice Work!

You just built a fully local AI-powered question-answering system with production-grade observability. Your RAG pipeline ingests documents, retrieves relevant context, generates grounded answers, and reports every metric to a live dashboard.

You learned how to:

  • Install Docker Desktop and use Docker Compose to orchestrate multi-service applications with shared networking and persistent volumes.
  • Build a FastAPI application that implements a full RAG pipeline: ingest documents into ChromaDB, retrieve relevant context, and generate answers with a local LLM.
  • Instrument a Python API with 7 custom Prometheus metrics including counters, histograms, and computed rates from Ollama's response timing fields.
  • Configure Prometheus to scrape your application and build a Grafana dashboard visualizing request rates, latency percentiles, and token throughput.
  • Secret Mission: Add a health check endpoint and configure Grafana alerting for production-readiness.

Ready to quiz yourself?