Build a RAG-Powered Q&A API

Build a RAG pipeline with a local LLM, vector DB, and FastAPI REST endpoint.

Introduction

30 Second Summary

Every AI chatbot you've ever used that gave a wrong answer was missing one thing: access to the right information at the right time. Retrieval-Augmented Generation solves this by fetching relevant documents before the AI generates its response.

In this project, you will build a RAG-powered Q&A API that answers questions about a knowledge base of documents. Everything runs locally on your machine with no cloud costs or API keys required.

What You'll Build

A locally-running RAG API that ingests markdown documents into a vector database, retrieves semantically relevant chunks at query time, and generates grounded answers using a local LLM.

By the end of this project, you'll have:

  • A document ingestion pipeline that chunks text, generates embeddings, and stores everything in ChromaDB.
  • A retrieval and generation system that connects semantic search to a local Mistral 7B model running on Ollama.
  • A FastAPI REST endpoint that serves the full RAG pipeline with structured JSON responses and source citations.
  • Secret Mission: Add a retrieval evaluation system that measures how well your pipeline finds relevant chunks.

Are there any prerequisites?

You should be comfortable with Python fundamentals like functions, loops, and importing modules. A basic understanding of what machine learning models do is helpful, but you don't need any prior experience with RAG, embeddings, or LLMs.

Before We Start

Before We Start...

Set Up Your Environment

A RAG pipeline needs several pieces working together: a vector database to store document chunks, an embedding model to turn text into numbers, a large language model to generate answers, and a web framework to serve it all as an API.

In this step, you will install Ollama to run an LLM locally on your GPU, set up a Python project with all the required packages, and create sample knowledge base documents for your RAG system to answer questions about.

In this step, get ready to:
  • Install Ollama and pull the Mistral 7B model.
  • Create the project directory, virtual environment, and install Python dependencies.
  • Create sample knowledge base documents for testing.
Install Ollama and pull the Mistral model

Ollama lets you run large language models directly on your machine instead of sending requests to a cloud API. Your RTX 4060 GPU will handle the heavy lifting of generating text responses.

  • Open a terminal. Press Ctrl+` (backtick) in VS Code to open the integrated terminal.

What terminal shell should I use?

Make sure your VS Code terminal is running PowerShell. You can check the dropdown in the top-right corner of the terminal panel. On Windows, PowerShell is the default.

  • Install Ollama by running this command:

Windows

irm https://ollama.com/install.ps1 | iex

What does this command do?

This PowerShell command downloads and runs the official Ollama installer. It installs in your user account without needing Administrator rights and adds ollama to your PATH so you can use it in the same terminal session.

macOS

curl -fsSL https://ollama.com/install.sh | sh

What does this command do?

This downloads and runs the official Ollama install script. It sets up the ollama binary and makes it available in your terminal.

Once the install finishes, you should see a message confirming Ollama is ready.

  • Verify Ollama installed correctly by running:
ollama -v

✔️ I see a version number

Ollama is installed and ready. Continue to the next command below.

ⓧ Command not found or error

If the command is not recognized, try closing and reopening your terminal to refresh the PATH. On Windows, you can also try opening a new PowerShell window outside VS Code.

If you still see an error, visit ollama.com/download and download the installer manually.

  • Pull the Mistral 7B model by running:
ollama pull mistral

What is happening here?

This downloads the Mistral 7B model (~4GB) to your local machine. Mistral is a powerful open-source model that fits comfortably in your GPU's 8GB VRAM when quantized.

The download may take a few minutes depending on your internet speed. You will see a progress bar as the model layers download.

  • Test that the model works by running:
ollama run mistral "Say hello"

Mistral should respond with a greeting. This confirms your GPU is running inference correctly.

Exit the interactive mode

If Ollama drops you into an interactive chat prompt after responding, type /bye and press Enter to exit back to your terminal.

Model not responding or very slow?

If the model takes more than 60 seconds to respond, it may be running on CPU instead of GPU. Check that your NVIDIA drivers are up to date. You need driver version 531 or later for your RTX 4060.

If you see an out-of-memory error, make sure no other GPU-heavy applications are running.

Ask for help: Why isn't Ollama using my GPU for inference?

Create the project and install Python dependencies

Now that Ollama is ready, you need a Python project with the libraries that handle document storage, embeddings, and the API layer.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir rag-qa-api
cd rag-qa-api
  • Create a Python virtual environment by running:
python -m venv venv

What is a virtual environment?

A virtual environment is an isolated space for your project's Python packages. It keeps this project's dependencies separate from other Python projects on your machine, preventing version conflicts.

  • Activate the virtual environment by running:

Windows

.\venv\Scripts\activate

macOS

source venv/bin/activate

You should see (venv) appear at the beginning of your terminal prompt. This confirms the virtual environment is active.

Seeing an execution policy error on Windows?

If PowerShell blocks the activation script, run this command first to allow local scripts:

Set-ExecutionPolicy -Scope CurrentUser RemoteSigned

Then try the activation command again.

Ask for help: How do I fix PowerShell execution policy errors when activating a Python venv?

  • Create the requirements.txt file. In VS Code's Explorer sidebar, right-click the rag-qa-api folder and select New File.
  • Name the file requirements.txt and paste in the following content:
chromadb==1.5.9
sentence-transformers==5.5.1
fastapi[standard]==0.136.3

What are these packages?

  • chromadb is a vector database that stores document chunks and their embeddings for fast similarity search.
  • sentence-transformers provides pre-trained models that convert text into vector embeddings (numerical representations capturing meaning).
  • fastapi[standard] is a modern Python web framework for building REST APIs, bundled with a development server.
  • Save requirements.txt.
  • Install all dependencies by running:
pip install -r requirements.txt

This install takes a few minutes

The sentence-transformers package pulls in PyTorch and several other large dependencies. The total download is several hundred MB. Wait for the install to finish completely before continuing.

  • Verify all packages imported correctly by running:
python -c "import chromadb; import sentence_transformers; import fastapi; print('All imports OK')"

You should see All imports OK printed to the terminal.

Import errors or install failures?

Make sure your virtual environment is active (you see (venv) in your prompt). If you ran pip install without the venv active, deactivate, reactivate, and run the install again.

If a specific package fails to build, check that you are running Python 3.11 by running python --version in your activated venv.

Ask for help: Why are my Python package imports failing after pip install?

✔️ Awesome, I've got everything!

Great. Make sure you have saved requirements.txt and that all imports passed.

ⓧ I'd like to double check the full code

chromadb==1.5.9
sentence-transformers==5.5.1
fastapi[standard]==0.136.3
Create sample knowledge base documents

Your RAG system needs documents to search through when answering questions. You will create a small knowledge base of markdown files covering Python basics and web development.

  • Create a documents directory inside your rag-qa-api folder by running:
mkdir documents
  • Confirm the folder exists by running:
ls

You should see documents, requirements.txt, and venv listed in the output.

  • In VS Code's Explorer sidebar, right-click the documents folder and select New File.
  • Name the file python_basics.md and paste in the following content:
# Python Basics

## Variables and Data Types

Python supports several built-in data types including integers, floats, strings, and booleans. Variables in Python are dynamically typed, meaning you don't need to declare their type explicitly. You simply assign a value to a variable name and Python infers the type.

Common data types include:
- int: whole numbers like 42 or -7
- float: decimal numbers like 3.14 or -0.5
- str: text enclosed in quotes like "hello" or 'world'
- bool: True or False values
- list: ordered collections like [1, 2, 3]
- dict: key-value pairs like {"name": "Alice", "age": 30}

## Functions

Functions in Python are defined using the def keyword. They allow you to encapsulate reusable logic into named blocks of code. Functions can accept parameters and return values using the return statement.

A function definition includes:
- The def keyword
- A function name following Python naming conventions
- Parentheses containing zero or more parameters
- A colon followed by an indented function body
- An optional return statement

Example concepts: default parameters allow you to specify fallback values, and keyword arguments let callers specify arguments by name for clarity.

## Control Flow

Python uses if, elif, and else statements for conditional execution. The conditions are evaluated as boolean expressions, and the indented block under the matching condition executes.

Loops in Python come in two forms:
- for loops iterate over sequences like lists, strings, or ranges
- while loops repeat as long as a condition remains True

The break statement exits a loop early, and continue skips to the next iteration.
  • Save python_basics.md.
  • Right-click the documents folder again and select New File.
  • Name the file web_development.md and paste in the following content:
# Web Development with Python

## HTTP and APIs

HTTP (Hypertext Transfer Protocol) is the foundation of web communication. It defines methods like GET, POST, PUT, and DELETE for interacting with resources. APIs (Application Programming Interfaces) use HTTP to allow programs to communicate with each other.

REST (Representational State Transfer) is an architectural style for designing web APIs. RESTful APIs use HTTP methods meaningfully:
- GET retrieves data without modifying it
- POST creates new resources
- PUT updates existing resources
- DELETE removes resources

Status codes indicate the result: 200 means success, 404 means not found, 500 means server error.

## FastAPI Framework

FastAPI is a modern Python web framework for building APIs. It provides automatic data validation using Python type hints and Pydantic models. FastAPI generates interactive API documentation automatically and supports asynchronous request handling.

Key features of FastAPI:
- Type-based request validation
- Automatic OpenAPI documentation
- High performance comparable to NodeJS and Go
- Native async/await support
- Dependency injection system

## Databases and ORMs

Databases store persistent data for web applications. SQL databases like PostgreSQL use structured tables with relationships. NoSQL databases like MongoDB store flexible document structures.

ORMs (Object-Relational Mappers) let you interact with databases using Python objects instead of raw SQL queries. Popular Python ORMs include SQLAlchemy and the Django ORM. They handle connection pooling, migrations, and query building.
  • Save web_development.md.

Why these documents?

These markdown files form the knowledge base your RAG system will search through. When you ask a question later, the system will find the most relevant chunks from these documents and use them as context for generating an answer.

The content covers two distinct topics so you can verify the retrieval system returns the correct source document for each question.

  • Confirm both files exist by running:
ls documents

You should see python_basics.md and web_development.md in the output.

Files not showing up?

Make sure you created the files inside the documents folder, not in the root rag-qa-api folder. Check the Explorer sidebar in VS Code to see the folder structure. The files should be nested under documents/.

Ask for help: My document files are in the wrong location, how do I move them?

✔️ Awesome, I've got everything!

Great. Double check you have saved both files inside the documents/ directory.

ⓧ I'd like to double check the full code

Here is documents/python_basics.md:

# Python Basics

## Variables and Data Types

Python supports several built-in data types including integers, floats, strings, and booleans. Variables in Python are dynamically typed, meaning you don't need to declare their type explicitly. You simply assign a value to a variable name and Python infers the type.

Common data types include:
- int: whole numbers like 42 or -7
- float: decimal numbers like 3.14 or -0.5
- str: text enclosed in quotes like "hello" or 'world'
- bool: True or False values
- list: ordered collections like [1, 2, 3]
- dict: key-value pairs like {"name": "Alice", "age": 30}

## Functions

Functions in Python are defined using the def keyword. They allow you to encapsulate reusable logic into named blocks of code. Functions can accept parameters and return values using the return statement.

A function definition includes:
- The def keyword
- A function name following Python naming conventions
- Parentheses containing zero or more parameters
- A colon followed by an indented function body
- An optional return statement

Example concepts: default parameters allow you to specify fallback values, and keyword arguments let callers specify arguments by name for clarity.

## Control Flow

Python uses if, elif, and else statements for conditional execution. The conditions are evaluated as boolean expressions, and the indented block under the matching condition executes.

Loops in Python come in two forms:
- for loops iterate over sequences like lists, strings, or ranges
- while loops repeat as long as a condition remains True

The break statement exits a loop early, and continue skips to the next iteration.

Here is documents/web_development.md:

# Web Development with Python

## HTTP and APIs

HTTP (Hypertext Transfer Protocol) is the foundation of web communication. It defines methods like GET, POST, PUT, and DELETE for interacting with resources. APIs (Application Programming Interfaces) use HTTP to allow programs to communicate with each other.

REST (Representational State Transfer) is an architectural style for designing web APIs. RESTful APIs use HTTP methods meaningfully:
- GET retrieves data without modifying it
- POST creates new resources
- PUT updates existing resources
- DELETE removes resources

Status codes indicate the result: 200 means success, 404 means not found, 500 means server error.

## FastAPI Framework

FastAPI is a modern Python web framework for building APIs. It provides automatic data validation using Python type hints and Pydantic models. FastAPI generates interactive API documentation automatically and supports asynchronous request handling.

Key features of FastAPI:
- Type-based request validation
- Automatic OpenAPI documentation
- High performance comparable to NodeJS and Go
- Native async/await support
- Dependency injection system

## Databases and ORMs

Databases store persistent data for web applications. SQL databases like PostgreSQL use structured tables with relationships. NoSQL databases like MongoDB store flexible document structures.

ORMs (Object-Relational Mappers) let you interact with databases using Python objects instead of raw SQL queries. Popular Python ORMs include SQLAlchemy and the Django ORM. They handle connection pooling, migrations, and query building.

Your environment is ready. You have Ollama running a local LLM, a Python project with all the right packages, and a set of documents for your RAG system to search through. Next up, you will build the ingestion pipeline that turns these documents into searchable vector embeddings.

Build the Document Ingestion Pipeline

Your environment is ready and your knowledge base documents are waiting in the documents/ folder. Now you need a way to feed that text into a system that can search it by meaning, not just keywords.

In this step, you will build an ingestion pipeline that reads your markdown files, splits them into manageable chunks, converts those chunks into numerical representations called vector embeddings, and stores everything in ChromaDB for fast semantic retrieval later.

In this step, get ready to:
  • Build document loading and text chunking functions.
  • Generate vector embeddings and store them in ChromaDB.
  • Run the pipeline and verify the stored data.
Implement document loading and chunking logic

Before your RAG system can search anything, it needs to break documents into smaller pieces. Large language models have limited context windows, so you feed them only the most relevant chunks rather than entire files.

  • In VS Code, create a new file called ingest.py in your project folder.
  • Add the imports, configuration constants, and the load_documents() function by pasting this code:
import os
import chromadb
from sentence_transformers import SentenceTransformer

CHROMA_PATH = "./chroma_db"
COLLECTION_NAME = "documents"
DOCUMENTS_DIR = "./documents"
EMBEDDING_MODEL = "all-MiniLM-L6-v2"
CHUNK_SIZE = 500
CHUNK_OVERLAP = 50


def load_documents(directory: str) -> list[dict]:
    """Load all markdown files from a directory."""
    documents = []
    for filename in os.listdir(directory):
        if filename.endswith(".md"):
            filepath = os.path.join(directory, filename)
            with open(filepath, "r", encoding="utf-8") as f:
                content = f.read()
            documents.append({"filename": filename, "content": content})
    return documents

What does this code do?

The configuration constants at the top define where your database lives (./chroma_db), the embedding model name, and how text will be split (500 characters per chunk, 50 character overlap).

The load_documents() function scans a directory for .md files and returns each one as a dictionary containing the filename and full text content.

Next, you need a function to split those full documents into smaller, overlapping chunks. Overlapping means each chunk shares some text with its neighbors so important ideas at chunk boundaries are not lost.

  • Below the load_documents() function, add the chunk_text() function:

def chunk_text(text: str, chunk_size: int = CHUNK_SIZE, overlap: int = CHUNK_OVERLAP) -> list[str]:
    """Split text into overlapping chunks by character count."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        if chunk.strip():
            chunks.append(chunk.strip())
        start += chunk_size - overlap
    return chunks

What does this code do?

The chunk_text() function walks through a string using a sliding window of 500 characters, advancing by 450 characters each step (500 minus the 50-character overlap). This produces chunks that share border text with their neighbors, preserving context across boundaries.

  • Save ingest.py.

Seeing a syntax error?

Check that the def chunk_text line is at the top level of the file (no extra indentation), and that the blank line between functions is present.

help me debug a syntax error in my ingest.py file

Add embedding generation and ChromaDB storage

Your loading and chunking functions handle the text processing side. Now you need the function that orchestrates the full pipeline: load, chunk, embed, and store.

  • Below chunk_text() in ingest.py, add the ingest_documents() function and the main entry point:

def ingest_documents():
    """Load, chunk, embed, and store documents in ChromaDB."""
    print("Loading embedding model...")
    embedder = SentenceTransformer(EMBEDDING_MODEL)

    print("Loading documents...")
    documents = load_documents(DOCUMENTS_DIR)

    if not documents:
        print("No documents found. Add .md files to the documents/ directory.")
        return

    print("Chunking documents...")
    all_chunks = []
    all_metadatas = []
    all_ids = []

    for doc in documents:
        chunks = chunk_text(doc["content"])
        for i, chunk in enumerate(chunks):
            all_chunks.append(chunk)
            all_metadatas.append({"source": doc["filename"], "chunk_index": i})
            all_ids.append(f"{doc['filename']}_{i}")

    print(f"Generating embeddings for {len(all_chunks)} chunks...")
    embeddings = embedder.encode(all_chunks).tolist()

    print("Storing in ChromaDB...")
    client = chromadb.PersistentClient(path=CHROMA_PATH)
    collection = client.get_or_create_collection(name=COLLECTION_NAME)
    collection.add(
        documents=all_chunks,
        embeddings=embeddings,
        metadatas=all_metadatas,
        ids=all_ids,
    )

    print(f"Successfully ingested {len(all_chunks)} chunks from {len(documents)} documents.")


if __name__ == "__main__":
    ingest_documents()

What does this code do?

The ingest_documents() function is the orchestrator. It loads the all-MiniLM-L6-v2 sentence-transformers model, reads all markdown files, splits them into chunks, and generates a vector embedding for each chunk.

Each chunk gets metadata tracking its source filename and position. The PersistentClient saves the database to disk at ./chroma_db so your data survives between script runs.

The if __name__ == "__main__" block means the ingestion runs when you execute the file directly, but the functions stay importable from other modules.

  • Save ingest.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved ingest.py and move on to running it.

ⓧ I'd like to double check the full code

import os
import chromadb
from sentence_transformers import SentenceTransformer

CHROMA_PATH = "./chroma_db"
COLLECTION_NAME = "documents"
DOCUMENTS_DIR = "./documents"
EMBEDDING_MODEL = "all-MiniLM-L6-v2"
CHUNK_SIZE = 500
CHUNK_OVERLAP = 50


def load_documents(directory: str) -> list[dict]:
    """Load all markdown files from a directory."""
    documents = []
    for filename in os.listdir(directory):
        if filename.endswith(".md"):
            filepath = os.path.join(directory, filename)
            with open(filepath, "r", encoding="utf-8") as f:
                content = f.read()
            documents.append({"filename": filename, "content": content})
    return documents


def chunk_text(text: str, chunk_size: int = CHUNK_SIZE, overlap: int = CHUNK_OVERLAP) -> list[str]:
    """Split text into overlapping chunks by character count."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        if chunk.strip():
            chunks.append(chunk.strip())
        start += chunk_size - overlap
    return chunks


def ingest_documents():
    """Load, chunk, embed, and store documents in ChromaDB."""
    print("Loading embedding model...")
    embedder = SentenceTransformer(EMBEDDING_MODEL)

    print("Loading documents...")
    documents = load_documents(DOCUMENTS_DIR)

    if not documents:
        print("No documents found. Add .md files to the documents/ directory.")
        return

    print("Chunking documents...")
    all_chunks = []
    all_metadatas = []
    all_ids = []

    for doc in documents:
        chunks = chunk_text(doc["content"])
        for i, chunk in enumerate(chunks):
            all_chunks.append(chunk)
            all_metadatas.append({"source": doc["filename"], "chunk_index": i})
            all_ids.append(f"{doc['filename']}_{i}")

    print(f"Generating embeddings for {len(all_chunks)} chunks...")
    embeddings = embedder.encode(all_chunks).tolist()

    print("Storing in ChromaDB...")
    client = chromadb.PersistentClient(path=CHROMA_PATH)
    collection = client.get_or_create_collection(name=COLLECTION_NAME)
    collection.add(
        documents=all_chunks,
        embeddings=embeddings,
        metadatas=all_metadatas,
        ids=all_ids,
    )

    print(f"Successfully ingested {len(all_chunks)} chunks from {len(documents)} documents.")


if __name__ == "__main__":
    ingest_documents()
Run the pipeline and verify the stored data

Your ingestion script is complete. Time to run it and confirm that your documents are chunked, embedded, and stored correctly.

  • Run the ingestion pipeline by executing this command in your terminal:
python ingest.py

What should I see?

The script prints progress messages as it loads the embedding model, reads documents, chunks them, generates embeddings, and stores them. The first run downloads the all-MiniLM-L6-v2 model (~80MB), so it may take a minute.

The final line should report successfully ingesting multiple chunks from 2 documents.

Seeing a ModuleNotFoundError?

Make sure your virtual environment is activated. You should see the environment name in parentheses at the start of your terminal prompt. If not, activate it again and rerun the command.

help me fix a ModuleNotFoundError when running ingest.py

Now verify that ChromaDB actually stored your chunks by querying it directly.

  • Open a Python REPL by running:
python
  • In the REPL, check the number of stored chunks by running these lines:
import chromadb
client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_collection(name="documents")
print(collection.count())
  • Confirm the printed number is greater than zero. This is the total number of chunks stored across both documents.
  • Exit the REPL by typing exit() and pressing Enter.

Why use get_collection here?

In the ingestion script, you used get_or_create_collection() because the collection might not exist yet. Here you use get_collection() to confirm the collection already exists from the ingestion run. If it does not exist, this call raises an error, which tells you the ingestion did not succeed.

Getting a collection not found error?

This means ingest.py did not run successfully. Check the output from the earlier python ingest.py command for error messages. Make sure you are running the REPL from the same directory as your project (the folder containing chroma_db/).

help me debug why my ChromaDB collection was not created after running ingest.py

Your documents are now embedded and searchable. Next up, you will build the retrieval and generation logic that turns a user's question into a grounded answer using these stored chunks.

Build the RAG Pipeline

Your documents are chunked, embedded, and stored in ChromaDB. The knowledge base is ready to be searched.

Now you need two capabilities: finding the most relevant chunks for a question, and feeding those chunks to your local LLM so it can generate a grounded answer. In this step, you will build the retrieval and generation functions that form the core of your RAG pipeline.

In this step, get ready to:
  • Implement a retrieval function that queries ChromaDB for semantically similar chunks.
  • Implement a generation function that sends context-injected prompts to Ollama.
  • Wire retrieval and generation together into a complete RAG pipeline.
Implement the retrieval function

The retrieval function is the "R" in RAG. It takes a user's question, converts it into an embedding using the same model that embedded your documents, and finds the closest matching chunks in ChromaDB.

  • Create a new file called rag_pipeline.py in your project folder.
  • Add the imports, configuration constants, and module-level initialization by pasting this code:
import chromadb
import httpx
from sentence_transformers import SentenceTransformer

# Configuration constants
CHROMA_PATH = "./chroma_db"
COLLECTION_NAME = "documents"
EMBEDDING_MODEL = "all-MiniLM-L6-v2"
OLLAMA_BASE_URL = "http://localhost:11434"
LLM_MODEL = "mistral"
TOP_K = 3

# Load models and connect to ChromaDB once at module level
embedder = SentenceTransformer(EMBEDDING_MODEL)
chroma_client = chromadb.PersistentClient(path=CHROMA_PATH)
collection = chroma_client.get_or_create_collection(name=COLLECTION_NAME)

What does this code do?

The constants at the top define where ChromaDB lives, which embedding model to use, and how to reach your local Ollama instance. TOP_K = 3 means you will retrieve the 3 most similar chunks per query.

The three lines at the bottom load the embedding model and connect to ChromaDB once when the module is imported. This avoids re-loading on every request.

Why load models at module level?

Loading a model from disk takes several seconds. By initializing at module level, you pay that cost once. Every subsequent function call reuses the already-loaded objects instantly.

  • Below the module-level initialization, add the retrieve() function:
def retrieve(query: str, top_k: int = TOP_K) -> list[dict]:
    """Retrieve relevant document chunks for a query."""
    # Encode the question using the same model that embedded the documents
    query_embedding = embedder.encode([query]).tolist()
    # Search ChromaDB for the closest matching chunks
    results = collection.query(
        query_embeddings=query_embedding,
        n_results=top_k,
    )
    # Package each result into a dictionary
    retrieved = []
    for i in range(len(results["documents"][0])):
        retrieved.append({
            "content": results["documents"][0][i],
            "metadata": results["metadatas"][0][i],
            "distance": results["distances"][0][i],
        })
    return retrieved

What does this code do?

The retrieve() function encodes the user's question into an embedding vector, then asks ChromaDB to find the top-K chunks whose embeddings are closest to the query vector. This is semantic search.

Each result includes the chunk text, its metadata (source filename and chunk index), and a distance score. Lower distance means higher relevance.

  • Save rag_pipeline.py.

You won't see a visible change yet. The next chunk adds the generation logic that uses these retrieved results.

Seeing an import error?

Make sure your virtual environment is activated. If httpx is not found, run pip install -r requirements.txt again.

help me debug an import error in rag_pipeline.py

Implement the generation function

Retrieval alone gives you relevant text chunks. The generation function takes those chunks, injects them into a prompt, and sends the prompt to your local Mistral model via Ollama's chat API.

  • In rag_pipeline.py, add the generate() function below retrieve():
def generate(query: str, context_chunks: list[dict]) -> str:
    """Generate an answer using Ollama with retrieved context."""
    # Combine all retrieved chunks into one context block
    context = "\n\n---\n\n".join([chunk["content"] for chunk in context_chunks])

    # Build a prompt that instructs the LLM to use the context
    prompt = (
        "Use the following context to answer the question. "
        "If the context doesn't contain enough information to answer, say so.\n\n"
        f"Context:\n{context}\n\n"
        f"Question: {query}\n\n"
        "Answer:"
    )

    # Send the prompt to Ollama's chat endpoint
    response = httpx.post(
        f"{OLLAMA_BASE_URL}/api/chat",
        json={
            "model": LLM_MODEL,
            "messages": [{"role": "user", "content": prompt}],
            "stream": False,
        },
        timeout=120.0,
    )
    response.raise_for_status()
    return response.json()["message"]["content"]

What does this code do?

The function joins the retrieved chunks with separator lines, then wraps them in a prompt template that tells the model to answer based on the provided context. This grounding instruction is what prevents the LLM from hallucinating.

The httpx.post call sends the prompt to Ollama's /api/chat endpoint at http://localhost:11434. The timeout=120.0 gives the model up to 2 minutes to respond, which accounts for longer answers on the first run when the model is loading into GPU memory.

Why httpx instead of requests?

The httpx library is already included when you install fastapi[standard]. It has an API similar to requests but avoids adding an extra dependency.

  • Save rag_pipeline.py.

You won't see a change yet. The next chunk connects retrieval and generation into a single function you can call.

Ollama not responding?

Make sure Ollama is running. Check your system tray for the Ollama icon. If it's not there, open a new terminal and run ollama serve.

help me troubleshoot why Ollama is not responding to API requests on localhost:11434

Wire retrieval and generation together

You have a retrieval function and a generation function. Now you need one orchestrator that calls both in sequence and packages the result into a structured response with source citations.

  • In rag_pipeline.py, add the query_rag() function and the test block below generate():
def query_rag(question: str) -> dict:
    """Full RAG pipeline: retrieve context and generate answer."""
    # Step 1: Find relevant chunks
    retrieved_chunks = retrieve(question)
    # Step 2: Generate an answer using those chunks as context
    answer = generate(question, retrieved_chunks)
    # Package the result with truncated source citations
    return {
        "question": question,
        "answer": answer,
        "sources": [
            {
                "content": chunk["content"][:200],
                "source": chunk["metadata"]["source"],
                "distance": chunk["distance"],
            }
            for chunk in retrieved_chunks
        ],
    }


if __name__ == "__main__":
    result = query_rag("What are the main data types in Python?")
    print(f"Question: {result['question']}")
    print(f"Answer: {result['answer']}")
    print(f"Sources: {[s['source'] for s in result['sources']]}")

What does this code do?

The query_rag() function is the complete pipeline. It retrieves relevant chunks, passes them to the LLM for generation, and returns a dictionary with the original question, the generated answer, and a list of sources.

Each source includes the first 200 characters of the chunk (for preview), the source filename, and the distance score. This gives the caller transparency into what the model based its answer on.

The if __name__ == "__main__" block runs a test question so you can verify the full pipeline directly from the terminal.

  • Save rag_pipeline.py.
  • Test the full pipeline by running:
python rag_pipeline.py

The first run may take 10-30 seconds as Ollama loads the Mistral model into GPU memory. You should see a question, a generated answer that references Python data types from your knowledge base, and a sources list showing python_basics.md.

Seeing a connection refused error?

This means Ollama is not running. Check your system tray for the Ollama icon. If it is missing, open a new terminal and run ollama serve. Then try running the script again.

If you see a timeout error instead, the model may still be loading. Wait 30 seconds and run the command again.

help me fix a connection or timeout error when running rag_pipeline.py

✔️ Awesome, I've got everything!

Great. Double check you have saved rag_pipeline.py and confirmed the output includes a grounded answer with sources.

ⓧ I'd like to double check the full code

Here is the complete rag_pipeline.py file:

import chromadb
import httpx
from sentence_transformers import SentenceTransformer

CHROMA_PATH = "./chroma_db"
COLLECTION_NAME = "documents"
EMBEDDING_MODEL = "all-MiniLM-L6-v2"
OLLAMA_BASE_URL = "http://localhost:11434"
LLM_MODEL = "mistral"
TOP_K = 3

embedder = SentenceTransformer(EMBEDDING_MODEL)
chroma_client = chromadb.PersistentClient(path=CHROMA_PATH)
collection = chroma_client.get_or_create_collection(name=COLLECTION_NAME)


def retrieve(query: str, top_k: int = TOP_K) -> list[dict]:
    """Retrieve relevant document chunks for a query."""
    query_embedding = embedder.encode([query]).tolist()
    results = collection.query(
        query_embeddings=query_embedding,
        n_results=top_k,
    )
    retrieved = []
    for i in range(len(results["documents"][0])):
        retrieved.append({
            "content": results["documents"][0][i],
            "metadata": results["metadatas"][0][i],
            "distance": results["distances"][0][i],
        })
    return retrieved


def generate(query: str, context_chunks: list[dict]) -> str:
    """Generate an answer using Ollama with retrieved context."""
    context = "\n\n---\n\n".join([chunk["content"] for chunk in context_chunks])

    prompt = (
        "Use the following context to answer the question. "
        "If the context doesn't contain enough information to answer, say so.\n\n"
        f"Context:\n{context}\n\n"
        f"Question: {query}\n\n"
        "Answer:"
    )

    response = httpx.post(
        f"{OLLAMA_BASE_URL}/api/chat",
        json={
            "model": LLM_MODEL,
            "messages": [{"role": "user", "content": prompt}],
            "stream": False,
        },
        timeout=120.0,
    )
    response.raise_for_status()
    return response.json()["message"]["content"]


def query_rag(question: str) -> dict:
    """Full RAG pipeline: retrieve context and generate answer."""
    retrieved_chunks = retrieve(question)
    answer = generate(question, retrieved_chunks)
    return {
        "question": question,
        "answer": answer,
        "sources": [
            {
                "content": chunk["content"][:200],
                "source": chunk["metadata"]["source"],
                "distance": chunk["distance"],
            }
            for chunk in retrieved_chunks
        ],
    }


if __name__ == "__main__":
    result = query_rag("What are the main data types in Python?")
    print(f"Question: {result['question']}")
    print(f"Answer: {result['answer']}")
    print(f"Sources: {[s['source'] for s in result['sources']]}")

Your RAG pipeline is working end-to-end. You ask a question, the system finds relevant chunks from your knowledge base, and the local LLM generates a grounded answer. Next up, you will wrap this pipeline in a REST API so other applications can query it over HTTP.

Expose the Pipeline as a FastAPI Endpoint

Your RAG pipeline retrieves relevant chunks and generates grounded answers. Right now it only works as a Python script you run from the terminal.

In this step, you will wrap the pipeline in a REST API using FastAPI. This gives any client (a frontend app, a mobile app, or another service) a clean HTTP endpoint to ask questions and receive structured JSON responses with source citations.

In this step, get ready to:
  • Define request and response schemas using Pydantic models.
  • Create API endpoints for querying the RAG pipeline and health checking.
  • Run and test the API with auto-generated interactive documentation.
Define request/response models and create the API

FastAPI uses Pydantic models to validate incoming request data and shape outgoing responses. You define Python classes that describe the exact structure of your JSON, and FastAPI handles serialization, validation, and documentation automatically.

  • Create a new file called main.py in your project folder.
  • Add the imports, app instance, and Pydantic models by pasting this code:
from fastapi import FastAPI
from pydantic import BaseModel
from rag_pipeline import query_rag

app = FastAPI(title="RAG Q&A API", version="1.0.0")


class QueryRequest(BaseModel):
    question: str


class SourceInfo(BaseModel):
    content: str
    source: str
    distance: float


class QueryResponse(BaseModel):
    question: str
    answer: str
    sources: list[SourceInfo]

What does this code do?

The first three lines import FastAPI itself, Pydantic's BaseModel for schema definitions, and the query_rag function you built in the previous step.

The app line creates your FastAPI application with a title and version that appear in the auto-generated docs.

The three classes define the shape of your API's data. QueryRequest expects a question string from the caller. SourceInfo describes each retrieved chunk. QueryResponse combines the question, generated answer, and list of sources into one structured response.

  • Now add the two API endpoints below the Pydantic models in main.py:
@app.post("/query", response_model=QueryResponse)
def ask_question(request: QueryRequest):
    """Ask a question and get an answer based on the document knowledge base."""
    result = query_rag(request.question)
    return result


@app.get("/health")
def health_check():
    """Check if the API is running."""
    return {"status": "healthy"}

What does this code do?

The @app.post("/query") decorator creates a POST endpoint at /query. When a client sends a JSON body with a question field, FastAPI validates it against QueryRequest, passes it to ask_question(), and formats the return value as QueryResponse.

The @app.get("/health") endpoint is a lightweight check that confirms the server is running. Production systems use health endpoints for monitoring and load balancer checks.

  • Save main.py.

Seeing a red underline on the import?

If your editor flags from rag_pipeline import query_rag with a warning, make sure your virtual environment is activated and that rag_pipeline.py exists in the same folder as main.py.

help me fix my import error in main.py

✔️ Awesome, I've got everything!

Great. Double check you have saved main.py before moving on.

ⓧ I'd like to double check the full code

from fastapi import FastAPI
from pydantic import BaseModel
from rag_pipeline import query_rag

app = FastAPI(title="RAG Q&A API", version="1.0.0")


class QueryRequest(BaseModel):
    question: str


class SourceInfo(BaseModel):
    content: str
    source: str
    distance: float


class QueryResponse(BaseModel):
    question: str
    answer: str
    sources: list[SourceInfo]


@app.post("/query", response_model=QueryResponse)
def ask_question(request: QueryRequest):
    """Ask a question and get an answer based on the document knowledge base."""
    result = query_rag(request.question)
    return result


@app.get("/health")
def health_check():
    """Check if the API is running."""
    return {"status": "healthy"}
Run and test the API

FastAPI includes a built-in development server with auto-reload. It also generates interactive API documentation (powered by Swagger UI) where you can send test requests directly from your browser.

  • Start the development server by running this command:
fastapi dev main.py

What does this command do?

The fastapi dev command starts a local development server with auto-reload enabled. Any time you save changes to your code, the server restarts automatically. It serves your API at http://127.0.0.1:8000.

  • Wait for the terminal to show that the server has started. You should see output mentioning the server is running at http://127.0.0.1:8000.
  • Open your browser and navigate to http://127.0.0.1:8000/docs.

You should see the Swagger UI interactive documentation page with your API title "RAG Q&A API" and both endpoints listed.

What is Swagger UI?

Swagger UI is an interactive documentation interface that FastAPI generates automatically from your code. It reads your Pydantic models and route decorators to build a page where you can try out each endpoint without writing any client code.

  • Click the POST /query row to expand it.
  • Click Try it out.
  • Replace the default request body with this JSON:
{
  "question": "What are the main data types in Python?"
}
  • Click Execute.

The response takes a few seconds while Ollama generates the answer. You should see a JSON response with three fields: question, answer, and sources. The answer references information from your ingested documents, and each source includes a content preview, the filename, and a distance score.

Test the health endpoint too

Scroll down to the GET /health row, expand it, click Try it out, then Execute. You should see {"status": "healthy"} in the response.

Getting a connection error or timeout?

Make sure Ollama is still running in the background. The /query endpoint calls the local Ollama server to generate answers. If Ollama was stopped, open a new terminal and run ollama serve to restart it.

If the response takes more than two minutes and times out, your GPU may be under heavy load. Close other GPU-intensive applications and try again.

help me debug my FastAPI RAG endpoint timeout

Secret mission

Add Retrieval Evaluation

Your RAG pipeline returns answers, but how do you know it's retrieving the right chunks? Build an evaluation system that measures retrieval quality with hit rate and Mean Reciprocal Rank metrics. Then experiment with different parameters to see how chunking strategy impacts performance.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing cloud costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Ollama with the Mistral model downloaded (~4GB disk space).
  • ChromaDB database stored in ./chroma_db/ inside your project folder.
  • Sentence-transformers model cache (~80MB in ~/.cache/huggingface/).
  • Project directory on your Desktop with Python virtual environment and source files.

Keep everything running

No action needed. Choose this if you plan to keep experimenting with your RAG pipeline or want to reuse Ollama for other projects.

  • Ollama stays installed and the Mistral model remains cached on disk. You can run it anytime.
  • Your ChromaDB database and project files stay in place, ready for future queries.
  • The sentence-transformers model cache stays in ~/.cache/huggingface/ for faster loading next time.

Pause - I'll come back to this later

Shut down Ollama to free up system resources. Your files and models stay on disk so you can resume later.

  • Right-click the Ollama icon in the system tray and select Quit.
  • To prevent Ollama from starting automatically, open Task Manager, go to the Startup apps tab, find ollama, and click Disable.
  • When you want to resume, restart Ollama by running:
ollama serve
  • Everything else (project files, ChromaDB, model cache) remains on disk and ready to use.

Delete - I don't want to use this again

Remove all project resources to reclaim disk space. Follow these steps in order.

  • Remove the Mistral model from Ollama by running:
ollama rm mistral
  • Uninstall Ollama from Windows. Open Settings, then Apps, then Add or remove programs. Find Ollama in the list and click Uninstall.
  • Delete the project directory from your Desktop by running:
cd ~\Desktop
Remove-Item -Recurse -Force rag-qa-api
  • Delete the sentence-transformers model cache by running:
Remove-Item -Recurse -Force ~\.cache\huggingface
  • If Ollama did not remove its model storage during uninstall, delete it manually by running:
Remove-Item -Recurse -Force ~\.ollama

That's a Wrap!

That's a Wrap!

Nice work! You've just built a complete Retrieval-Augmented Generation system from scratch and served it as a production-style REST API.

You've learned how to:

  • Build a full RAG pipeline end-to-end: document ingestion, chunking, vector embedding with sentence-transformers, semantic retrieval from ChromaDB, and grounded answer generation.
  • Run a 7B parameter LLM locally on your GPU using Ollama, proving you can work with self-hosted models.
  • Expose an AI system as a FastAPI REST endpoint with structured JSON responses and source citations.
  • Secret Mission: Implement retrieval evaluation with hit rate and MRR metrics to measure pipeline quality.

Ready to quiz yourself?