Build a RAG API with FastAPI and ChromaDB

Build a REST API that answers questions using your own documents as context.

Introduction

30 Second Summary

Ever asked an AI chatbot a question and gotten a confident answer that was completely wrong? That happens because the AI is guessing from its training data instead of checking actual documents. The fix is a pattern called Retrieval-Augmented Generation (RAG), which forces the AI to look up real information before responding.

In this project, you will build a REST API that implements the full RAG pipeline. You will feed it a knowledge base of text documents, and it will answer questions by retrieving the most relevant pieces and sending them as context to an LLM.

What You'll Build

A locally-running API that ingests documents into a vector database, retrieves relevant chunks via semantic search, and generates grounded answers using a free LLM.

By the end of this project, you'll have:

  • A ChromaDB vector store populated with document embeddings you can query semantically.
  • A FastAPI endpoint that retrieves relevant document chunks and generates grounded LLM responses via Groq.
  • Interactive Swagger UI docs to test your RAG system without needing any external tools.
  • Secret Mission: Add a /upload endpoint that accepts new documents via the API and indexes them into ChromaDB on the fly.

Are there any prerequisites?

None. This guide includes full setup instructions for Python, ChromaDB, FastAPI, and the Groq API. The Groq free tier requires no credit card, so you can complete this entire project at zero cost.

Set Up Your Development Environment

Before you can build your RAG API, you need the right tools in place. This step gets your development environment ready with Python, a code editor, and the packages that power retrieval and generation.

You will also sign up for Groq, a free LLM API provider. Groq gives you access to powerful language models without any cost or credit card requirement.

In this step, get ready to:
  • Install Python 3.11+ and create your project folder.
  • Create a virtual environment and install required packages.
  • Get a free Groq API key for LLM inference.
Install Python and create a project folder

Python is the language your entire RAG system runs on. You need version 3.11 or higher for compatibility with the packages you will install.

  • Check your Python version by running this command in your terminal:

macOS

python3 --version

Windows

python --version

✔️ I see version 3.11 or higher

Your Python version is ready. Move on to the next instruction below.

ⓧ I see an older version

Your system has an older Python installed. macOS often ships with Python 3.9 by default, which is too old for this project.

  • Download the latest Python 3.11+ installer from python.org/downloads.
  • Run the installer. On Windows, check the box that says Add Python to PATH before clicking Install Now.
  • Close and reopen your terminal after installation.
  • Run the version check command again to confirm it shows 3.11 or higher.

macOS showing the old version still?

After installing a new Python, macOS may still resolve python3 to the old system version. Try running python3.11 --version or python3.12 --version to use the version you just installed directly.

ⓧ Command not found

Python is not installed on your machine yet.

  • Download the latest Python 3.11+ installer from python.org/downloads.
  • Run the installer. On Windows, check the box that says Add Python to PATH before clicking Install Now.
  • Close and reopen your terminal after installation.
  • Run the version check command again to confirm you see a version number.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir rag-api
cd rag-api
  • Confirm the folder exists by running:
ls

You should see an empty output since the folder is brand new. The important thing is that no error appeared.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Visual Studio Code and press Enter to open it.
  • In VS Code, click File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop, select the rag-api folder, and click Open.
  • Open the integrated terminal by pressing Ctrl+` (backtick key, same on macOS and Windows).

Why use the integrated terminal?

VS Code's built-in terminal opens directly in your project folder. This means every command you run is already pointed at the right place.

Create a virtual environment and install dependencies

A virtual environment is an isolated space for your project's packages. It keeps this project's dependencies separate from anything else on your machine.

  • Create a virtual environment by running this command in your VS Code terminal:

macOS

python3 -m venv venv

Windows

python -m venv venv
  • Confirm the venv folder was created by running:
ls

You should see a venv folder listed in the output.

  • Activate the virtual environment by running:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate

You should see (venv) appear at the start of your terminal prompt. This confirms the virtual environment is active.

Not seeing (venv) in your prompt?

On Windows PowerShell, you may see an error about execution policies. Run Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned and then try the activate command again.

On macOS, make sure you typed source before the path. Without it, the command will not work.

Help me activate my Python virtual environment.

  • Install all the packages your project needs by running:
pip install fastapi uvicorn chromadb groq python-dotenv

What are these packages?

  • FastAPI is a modern Python web framework for building APIs with automatic documentation.
  • uvicorn is the server that runs your FastAPI app.
  • ChromaDB is a vector database that stores and searches document embeddings locally.
  • groq is the Python SDK for calling Groq's LLM API.
  • python-dotenv loads environment variables from a .env file so you can keep secrets out of your code.

The install may take a minute or two. When it finishes, you should see a line that says Successfully installed followed by a list of package names.

Seeing an error during install?

Make sure (venv) is showing in your prompt. If you see a permissions error, it usually means the virtual environment is not active.

If you see a network error, check your internet connection and try running the command again.

Help me fix my pip install error.

Get your Groq API key

Groq provides free access to large language models through their API. You will use this API to generate answers from the context your RAG system retrieves.

  • Open your browser and go to console.groq.com.
  • Sign up for a free account. No credit card is required.
  • Once logged in, navigate to the API Keys section in the left sidebar.
  • Click Create API Key.
  • Give your key a name (for example, rag-api) and click Submit.
  • Copy the API key that appears. This is the only time you will see the full key, so copy it now.

Keep your API key safe

Never share your API key publicly or commit it to version control. You will store it in a .env file that stays on your local machine only.

  • In VS Code's Explorer sidebar, right-click inside the file list and select New File.
  • Name the file .env and press Enter.
  • Paste the following content into the file, replacing your-api-key-here with the API key you just copied:
GROQ_API_KEY=your-api-key-here

What does this file do?

The .env file stores sensitive configuration outside your code. The python-dotenv package reads this file at runtime so your application can access the key without hardcoding it.

  • Save the file with Cmd+S (macOS) or Ctrl+S (Windows).

✔️ Awesome, I've got everything!

Great. Double check you have saved your .env file with your real API key in place of the placeholder.

ⓧ I'd like to double check the full code

Here is what your .env file should look like (with your own key substituted):

GROQ_API_KEY=your-api-key-here

Your development environment is fully set up. You have Python, your packages, and a Groq API key ready to go. Next up, you will create a knowledge base and build the script that turns your documents into searchable vector embeddings.

Create Your Knowledge Base

A RAG system needs real documents to retrieve information from. Without a knowledge base, the LLM has nothing to ground its answers in and will fall back to its training data or refuse to answer.

In this step, you'll create a knowledge base text file and build a Python script that splits it into chunks and stores those chunks as vector embeddings in ChromaDB. This is the "retrieval" foundation that the rest of your API depends on.

In this step, get ready to:
  • Write a knowledge base text file with sample content.
  • Build an ingestion script that chunks text and stores it in ChromaDB.
  • Run the ingestion and verify documents are stored.
Write your knowledge base content

Your RAG system needs a source of truth to search through. You'll create a text file with paragraphs of information that ChromaDB will later index for semantic search.

  • Create a new file called knowledge_base.txt inside your rag-api folder.
  • Paste the following content into the file:
NextWork is an online learning platform that helps people build real-world tech skills through hands-on projects. Founded in 2023, NextWork focuses on practical experience rather than passive video watching.

The platform offers projects across cloud computing, AI, web development, and DevOps. Each project takes between 30 and 90 minutes to complete and results in a tangible portfolio piece.

NextWork uses a project-based learning approach where learners follow step-by-step guides to build real applications. This is different from traditional courses that rely heavily on lectures and quizzes.

The platform is designed for career changers, bootcamp graduates, and self-taught developers who want to build proof of their skills. Employers value hands-on experience over certificates.

NextWork projects cover AWS, Azure, Google Cloud, Python, JavaScript, Docker, Kubernetes, and many other technologies. New projects are added regularly based on industry demand.

Each project includes a detailed guide with screenshots, commands, and explanations. Learners can work at their own pace and revisit projects as many times as they want.

The community aspect of NextWork allows learners to share their completed projects, get feedback, and connect with others on similar learning paths.

NextWork offers both free and premium projects. Free projects cover foundational skills while premium projects dive into advanced topics and real-world scenarios.

The platform tracks learner progress and provides badges for completed projects. These badges can be shared on LinkedIn to demonstrate skills to potential employers.

NextWork was built to solve the gap between knowing theory and being able to do the work. Many learners report landing jobs within months of building their project portfolios.

What is this content for?

This text file acts as the knowledge source your RAG system searches through. Each paragraph covers a different aspect of the topic, giving the retrieval system distinct facts to find.

The more varied the content, the better you'll see semantic search in action later. The system will match questions to the most relevant paragraphs, not just keyword matches.

  • Save knowledge_base.txt.
Build the ingestion script

Raw text is not searchable by meaning. You need to split it into smaller pieces (chunks) and store them as vector embeddings that ChromaDB can compare mathematically.

What is document chunking?

Chunking splits a large document into smaller overlapping pieces. Each chunk becomes a separate entry in the vector database that can be independently retrieved.

The overlap ensures that sentences at chunk boundaries are not lost. If a fact spans the border between two chunks, both chunks contain part of it.

  • Create a new file called ingest.py inside your rag-api folder.
  • Add the imports and the chunking function by pasting this code:
import chromadb
from chromadb.config import Settings


def chunk_text(text, chunk_size=500, overlap=50):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        if chunk.strip():
            chunks.append(chunk.strip())
        start += chunk_size - overlap
    return chunks

What does this code do?

The chunk_text() function takes a string of text and slides a window of 500 characters across it, stepping forward 450 characters each time. The 50-character overlap means neighboring chunks share some content at their edges.

Each non-empty chunk gets stripped of whitespace and added to the list. The function returns all chunks ready to be embedded and stored.

  • Save ingest.py.

Seeing a red underline on the import?

If VS Code shows a warning on import chromadb, make sure your virtual environment is selected as the Python interpreter. Click the Python version in the bottom status bar and select the one inside your venv folder.

Why is VS Code not recognizing my chromadb import?

  • Now add the main() function below the chunk_text() function by pasting this code at the bottom of the file:
def main():
    # Read the knowledge base
    with open("knowledge_base.txt", "r", encoding="utf-8") as f:
        content = f.read()

    # Split into chunks
    chunks = chunk_text(content)
    print(f"Split knowledge base into {len(chunks)} chunks")

    # Initialize ChromaDB with persistent storage
    client = chromadb.PersistentClient(path="./chroma_db")

    # Create or get the collection
    collection = client.get_or_create_collection(name="knowledge_base")

    # Add chunks to the collection
    collection.add(
        documents=chunks,
        ids=[f"chunk_{i}" for i in range(len(chunks))],
        metadatas=[{"source": "knowledge_base.txt", "chunk_index": i} for i in range(len(chunks))]
    )

    print(f"Successfully added {collection.count()} chunks to ChromaDB")


if __name__ == "__main__":
    main()

What does this code do?

The main() function runs the full ingestion pipeline. It reads knowledge_base.txt, splits the content into chunks, then stores them in ChromaDB.

  • chromadb.PersistentClient(path="./chroma_db") creates a database that saves to disk so your embeddings survive between script runs.
  • get_or_create_collection(name="knowledge_base") creates a named collection (like a table) to hold your documents.
  • collection.add() stores the chunks with unique IDs and metadata. ChromaDB automatically generates vector embeddings for each chunk using its built-in model.
  • Save ingest.py.

Not sure if you pasted in the right spot?

The main() function should start on the line immediately after the return chunks line of chunk_text(), with a blank line separating them. The if __name__ block should be the very last thing in the file.

Help me check if my ingest.py is structured correctly

✔️ Awesome, I've got everything!

Great. Double check you've saved ingest.py before moving on.

ⓧ I'd like to double check the full code

Here is the complete ingest.py file for comparison:

import chromadb
from chromadb.config import Settings


def chunk_text(text, chunk_size=500, overlap=50):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        if chunk.strip():
            chunks.append(chunk.strip())
        start += chunk_size - overlap
    return chunks


def main():
    # Read the knowledge base
    with open("knowledge_base.txt", "r", encoding="utf-8") as f:
        content = f.read()

    # Split into chunks
    chunks = chunk_text(content)
    print(f"Split knowledge base into {len(chunks)} chunks")

    # Initialize ChromaDB with persistent storage
    client = chromadb.PersistentClient(path="./chroma_db")

    # Create or get the collection
    collection = client.get_or_create_collection(name="knowledge_base")

    # Add chunks to the collection
    collection.add(
        documents=chunks,
        ids=[f"chunk_{i}" for i in range(len(chunks))],
        metadatas=[{"source": "knowledge_base.txt", "chunk_index": i} for i in range(len(chunks))]
    )

    print(f"Successfully added {collection.count()} chunks to ChromaDB")


if __name__ == "__main__":
    main()
Run the ingestion and verify

Your script and knowledge base are ready. Time to run the ingestion and confirm that ChromaDB stored your document chunks.

  • Run the ingestion script in your terminal by running this command:
python ingest.py

You should see two lines of output: one reporting how many chunks were created, and one confirming they were added to ChromaDB.

What should I see?

Your terminal prints the number of chunks the text was split into, then confirms that the same number of chunks were added to the ChromaDB collection. You'll also notice a new chroma_db folder appear in your project directory. This is where ChromaDB persists your embeddings on disk.

Seeing a FileNotFoundError?

This means Python cannot find knowledge_base.txt. Make sure the file is saved in the same folder as ingest.py (your rag-api folder). Also confirm your terminal is in the rag-api directory by checking the path shown in your terminal prompt.

Help me fix my FileNotFoundError when running ingest.py

Your knowledge base is now stored as searchable vector embeddings. Next up, you'll build the API that queries these embeddings and sends the results to an LLM for answer generation.

Build the RAG API

Your knowledge base is ingested and sitting in ChromaDB as vector embeddings, ready to be searched.

Now you need an API that accepts a question, retrieves the most relevant chunks from your vector store, and sends them to an LLM to generate a grounded answer. This is the core of the RAG pattern. You will build it using FastAPI, which auto-generates interactive docs so you can test your API without writing a separate client.

In this step, get ready to:
  • Create a FastAPI application with Pydantic data models.
  • Build a /query endpoint that retrieves chunks and generates LLM answers via Groq.
  • Run and test the API using auto-generated Swagger UI docs.
Create the FastAPI app and data models

Before writing endpoint logic, you need the application skeleton. This includes importing your libraries, connecting to the ChromaDB collection you already populated, initializing the Groq client, and defining the request/response shapes.

  • Create a new file called main.py in your rag-api folder.
  • Add the following imports, app initialization, and data model code:
import os
from dotenv import load_dotenv
from fastapi import FastAPI
from pydantic import BaseModel
import chromadb
from groq import Groq

load_dotenv()

app = FastAPI(title="RAG API", description="A Retrieval-Augmented Generation API")

# Initialize ChromaDB client
chroma_client = chromadb.PersistentClient(path="./chroma_db")
collection = chroma_client.get_or_create_collection(name="knowledge_base")

# Initialize Groq client
groq_client = Groq(api_key=os.environ.get("GROQ_API_KEY"))


class QueryRequest(BaseModel):
    question: str
    n_results: int = 3


class QueryResponse(BaseModel):
    answer: str
    sources: list[str]

What does this code do?

load_dotenv() reads your .env file so your Groq API key is available as an environment variable.

PersistentClient(path="./chroma_db") connects to the same database directory your ingestion script created. get_or_create_collection retrieves the knowledge_base collection you already populated with chunks.

QueryRequest defines what the API accepts (a question and an optional result count). QueryResponse defines what the API returns (an answer and the source chunks used to generate it).

  • Save main.py.

You will not see output yet. The next substep adds the endpoint that uses these models.

Implement the RAG query endpoint

The /query endpoint is where the full RAG pipeline comes together. It takes a question, finds the most relevant chunks via semantic search, then passes those chunks as context to the LLM so the answer is grounded in your documents.

  • In main.py, add the following /query endpoint below your QueryResponse class:
@app.post("/query", response_model=QueryResponse)
def query_rag(request: QueryRequest):
    """Query the RAG system with a question."""
    # Retrieve relevant chunks from ChromaDB
    results = collection.query(
        query_texts=[request.question],
        n_results=request.n_results
    )

    # Extract the document texts
    retrieved_docs = results["documents"][0]

    # Build the context from retrieved chunks
    context = "\n\n".join(retrieved_docs)

    # Create the prompt for the LLM
    chat_completion = groq_client.chat.completions.create(
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a helpful assistant that answers questions based only on the provided context. "
                    "If the context does not contain enough information to answer the question, "
                    "say that you do not have enough information. Do not make up answers."
                )
            },
            {
                "role": "user",
                "content": f"Context:\n{context}\n\nQuestion: {request.question}"
            }
        ],
        model="llama-3.3-70b-versatile",
        temperature=0.3,
        max_tokens=1024
    )

    answer = chat_completion.choices[0].message.content

    return QueryResponse(answer=answer, sources=retrieved_docs)

What does this code do?

collection.query() performs a semantic search against your ChromaDB collection. It finds the chunks most similar in meaning to the question, not just keyword matches.

The retrieved chunks get joined into a single context string and passed to the Groq API inside the user message. The system prompt tells the LLM to only answer from the provided context and to admit when it does not have enough information.

The model llama-3.3-70b-versatile is a powerful open-source LLM available for free on Groq. Setting temperature=0.3 keeps responses focused and factual.

  • Now add the health check endpoint below query_rag:
@app.get("/health")
def health_check():
    """Check if the API is running and ChromaDB has documents."""
    count = collection.count()
    return {"status": "healthy", "documents_indexed": count}

What does this endpoint do?

The /health endpoint gives you a quick way to confirm your API is running and that ChromaDB has documents loaded. It returns the count of indexed chunks so you can verify your ingestion worked.

  • Save main.py.

Your API code is complete. The next substep starts the server so you can test it.

Seeing red underlines in VS Code?

If your editor shows import errors for chromadb or groq, make sure your virtual environment is selected as the Python interpreter. Press Cmd+Shift+P (macOS) or Ctrl+Shift+P (Windows), type Python: Select Interpreter, and choose the one inside your venv folder.

help me fix import errors in my RAG API project

✔️ Awesome, I've got everything!

Great. Double check you have saved main.py before moving on.

ⓧ I'd like to double check the full code

import os
from dotenv import load_dotenv
from fastapi import FastAPI
from pydantic import BaseModel
import chromadb
from groq import Groq

load_dotenv()

app = FastAPI(title="RAG API", description="A Retrieval-Augmented Generation API")

# Initialize ChromaDB client
chroma_client = chromadb.PersistentClient(path="./chroma_db")
collection = chroma_client.get_or_create_collection(name="knowledge_base")

# Initialize Groq client
groq_client = Groq(api_key=os.environ.get("GROQ_API_KEY"))


class QueryRequest(BaseModel):
    question: str
    n_results: int = 3


class QueryResponse(BaseModel):
    answer: str
    sources: list[str]


@app.post("/query", response_model=QueryResponse)
def query_rag(request: QueryRequest):
    """Query the RAG system with a question."""
    # Retrieve relevant chunks from ChromaDB
    results = collection.query(
        query_texts=[request.question],
        n_results=request.n_results
    )

    # Extract the document texts
    retrieved_docs = results["documents"][0]

    # Build the context from retrieved chunks
    context = "\n\n".join(retrieved_docs)

    # Create the prompt for the LLM
    chat_completion = groq_client.chat.completions.create(
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a helpful assistant that answers questions based only on the provided context. "
                    "If the context does not contain enough information to answer the question, "
                    "say that you do not have enough information. Do not make up answers."
                )
            },
            {
                "role": "user",
                "content": f"Context:\n{context}\n\nQuestion: {request.question}"
            }
        ],
        model="llama-3.3-70b-versatile",
        temperature=0.3,
        max_tokens=1024
    )

    answer = chat_completion.choices[0].message.content

    return QueryResponse(answer=answer, sources=retrieved_docs)


@app.get("/health")
def health_check():
    """Check if the API is running and ChromaDB has documents."""
    count = collection.count()
    return {"status": "healthy", "documents_indexed": count}
Run and test your API

FastAPI comes with auto-generated interactive documentation called Swagger UI. You can test every endpoint directly in your browser without needing a separate tool like Postman.

  • Start the development server by running this command in your terminal:
uvicorn main:app --reload

What does this command do?

uvicorn is the server that runs your FastAPI app. main:app tells it to look in main.py for the object named app. The --reload flag restarts the server automatically whenever you save a file change.

You should see output showing the server started on http://127.0.0.1:8000.

Server not starting?

Make sure your virtual environment is activated. If you see a "module not found" error, run pip install fastapi uvicorn chromadb groq python-dotenv again inside your activated environment.

If you see a GROQ_API_KEY error, check that your .env file is in the same folder as main.py and contains your key.

help me fix my uvicorn server startup error

  • Open your browser and go to http://127.0.0.1:8000/docs.
  • You should see the Swagger UI page with your two endpoints listed: POST /query and GET /health.
  • Click on POST /query to expand it.
  • Click Try it out.
  • Replace the example JSON body with a question about your knowledge base:
{
  "question": "What is NextWork?",
  "n_results": 3
}
  • Click Execute.

You should see a JSON response with an answer field containing a grounded answer and a sources field showing which chunks were used.

What just happened?

Your API received the question, searched ChromaDB for the 3 most semantically similar chunks, assembled them into a context string, and sent everything to Groq's llama-3.3-70b-versatile model. The model read the context and generated an answer grounded in your documents.

This is the complete RAG pipeline: retrieve, augment the prompt with context, then generate.

Getting an error response?

If you see a 500 error, check the terminal running uvicorn for the full traceback. Common causes include an invalid or missing GROQ_API_KEY in your .env file, or the chroma_db folder not existing (re-run python ingest.py if needed).

help me debug my RAG API 500 error

Your RAG API is live and answering questions from your knowledge base. Next up, you will stress-test it with different types of questions to see how retrieval quality affects the answers.

Test and Refine Your RAG System

Your RAG API is live and handling queries. The real test of a RAG system is how well it retrieves the right context for different types of questions.

In this step, you'll put your API through its paces by asking questions it can answer, questions it cannot, and then tweaking the retrieval settings to see how they affect response quality.

In this step, get ready to:
  • Test the API with questions your knowledge base can answer.
  • Test the API with questions outside your knowledge base.
  • Tune retrieval parameters to observe quality differences.
Test with relevant questions

The best way to evaluate a RAG system is to ask questions you already know the answer to. That way you can verify the API retrieves the correct chunks and generates an accurate response.

  • Open the Swagger UI in your browser at http://127.0.0.1:8000/docs.
  • Click the POST /query row to expand it.
  • Click Try it out in the top right corner of the endpoint panel.
  • Replace the request body with this JSON:
{
  "question": "What is NextWork?",
  "n_results": 3
}
  • Click Execute.

You should see a JSON response with an answer field containing a summary of what NextWork is, plus a sources array showing the 3 chunks that were retrieved from your knowledge base.

What should I see?

The answer should accurately describe NextWork using only information from your knowledge base. It should mention things like hands-on projects, practical experience, and career building.

The sources array shows you exactly which chunks the vector search selected as most relevant to your question. This is the "retrieval" part of Retrieval-Augmented Generation in action.

  • Try a few more relevant questions to see how the retrieval adapts. Replace the question in the request body with each of these and click Execute each time:
{
  "question": "What technologies do NextWork projects cover?",
  "n_results": 3
}
{
  "question": "How long do projects take to complete?",
  "n_results": 3
}

Notice how each question retrieves different chunks from your knowledge base. The semantic search finds paragraphs related to the meaning of your question, not just keyword matches.

Getting an error or empty response?

  • Check that your uvicorn server is still running in the terminal. If you closed it, restart it with uvicorn main:app --reload.
  • Verify your .env file contains a valid Groq API key. An invalid key returns an authentication error.
  • Make sure you ran python ingest.py in the previous step. The /health endpoint should show a non-zero documents_indexed count.

Help me debug my RAG API query response

Test with irrelevant questions

A well-built RAG system should refuse to answer when it does not have relevant context. Your system prompt instructs the model to say it does not have enough information rather than making things up.

  • In the Swagger UI, replace the request body with a question that has nothing to do with your knowledge base:
{
  "question": "What is the capital of France?",
  "n_results": 3
}
  • Click Execute.

The model should respond saying it does not have enough information to answer the question. The sources array still shows 3 chunks because ChromaDB always returns the closest matches, even when they are not actually relevant.

Why does it still return sources?

Vector search always returns the top N closest results by distance. It has no concept of "relevant enough" on its own. The intelligence here comes from your system prompt, which tells the LLM to only answer from the context provided.

This is why prompt engineering matters in RAG. Without that instruction, the model would happily answer "Paris" from its training data, completely ignoring your knowledge base.

  • Try one more irrelevant question to confirm the behaviour is consistent:
{
  "question": "How do I bake a chocolate cake?",
  "n_results": 3
}
  • Click Execute and confirm the model again declines to answer.
Tune the retrieval

The quality of a RAG response depends heavily on how many chunks you retrieve and how those chunks are sized. Retrieving too few chunks might miss important context. Retrieving too many might dilute the signal with noise.

  • In the Swagger UI, change n_results from 3 to 5 to retrieve more context:
{
  "question": "What is NextWork and who is it for?",
  "n_results": 5
}
  • Click Execute and compare the response to when you used 3 results.

With 5 chunks, the answer should be more comprehensive because the model has more context to draw from. You should see 5 items in the sources array.

How does n_results affect answer quality?

More chunks means more context for the LLM, but there's a trade-off. Too many chunks can include less-relevant text that confuses the model or uses up token limits.

For a small knowledge base like yours, 3-5 chunks works well. In production systems with thousands of documents, teams carefully tune this number alongside chunk size to find the sweet spot.

You can also experiment with chunk sizes. The current ingest.py uses a chunk_size of 500 characters with an overlap of 50.

  • In ingest.py, find this line:
def chunk_text(text, chunk_size=500, overlap=50):
  • Try changing chunk_size to 200 to create smaller, more granular chunks:
def chunk_text(text, chunk_size=200, overlap=50):
  • Save ingest.py.
  • Delete the existing chroma_db folder so you start fresh. In your terminal (open a new terminal if uvicorn is running), run:
rm -rf chroma_db
  • Re-run the ingestion script by running:
python ingest.py

You should see a higher chunk count printed to the terminal, since smaller chunks produce more pieces from the same text.

  • Go back to the Swagger UI and try the same question again. Compare how the answer quality and retrieved sources differ with smaller chunks.

What is the trade-off with chunk size?

Smaller chunks give more precise retrieval. Each chunk covers a narrower topic, so the top results are more focused. However, each individual chunk contains less context, which can make answers feel fragmented.

Larger chunks provide more surrounding context per result. The trade-off is that irrelevant text may get pulled in alongside the relevant parts. Finding the right balance depends on your content and use case.

  • Set chunk_size back to 500 and overlap back to 50 in ingest.py when you're done experimenting:
def chunk_text(text, chunk_size=500, overlap=50):
  • Delete chroma_db and re-run python ingest.py one more time to restore the original chunks.
rm -rf chroma_db
python ingest.py

API returning stale results after re-ingestion?

  • If uvicorn was running with --reload, it should pick up the new chroma_db automatically. If results still look wrong, stop the server with Ctrl+C and restart it.
  • Make sure you deleted the chroma_db folder before re-running ingest.py. Without deleting it, the old chunks remain alongside the new ones.

Help me fix stale results after re-ingesting documents

You now understand how retrieval parameters directly affect answer quality in a RAG system. Next up, you can take your API further with a Secret Mission that adds dynamic document uploads.

Secret mission

Add a Document Upload Endpoint

Your RAG system works great with pre-loaded documents. But what if you want to add new knowledge without rerunning the ingestion script? Build a /upload endpoint that accepts new documents via the API and indexes them into ChromaDB in real time.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. The Groq API free tier does not charge you and does not require a credit card. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • The rag-api project folder (includes venv, chroma_db, and all code files).
  • Your Groq API key (stored in .env and in the Groq console).
  • Your Groq account (free tier, no charges).

Keep everything running

No action needed. Choose this if you are still actively building or plan to extend this project soon.

  • Your rag-api folder stays on your Desktop.
  • Your Groq account remains on the free tier with no charges.
  • The uvicorn server only runs when you start it manually, so it does not consume resources in the background.

Pause - I'll come back to this later

Shut down any running processes to free up memory. Your files and Groq account stay intact for when you return.

  • If the uvicorn server is still running, press Ctrl+C in the terminal to stop it.
  • Deactivate your virtual environment by running:
  • Your project folder, ChromaDB data, and Groq account all remain ready for next time.

Delete - I don't want to use this again

Remove all project resources from your machine and clean up your Groq account.

  • If the uvicorn server is still running, press Ctrl+C to stop it.
  • Deactivate the virtual environment by running:
deactivate
  • Delete the entire project folder by running this command (macOS/Linux):
rm -rf ~/Desktop/rag-api
  • On Windows, run this instead:
rmdir /s /q %USERPROFILE%\Desktop\rag-api
  • Delete your Groq API key. In the Groq console, navigate to API Keys and delete the key you created for this project.
  • To delete your Groq account entirely, visit https://privacy.groq.com/, click Deletion Request, fill out the form, and click Request review. This permanently removes all API keys, usage history, and account data.

That's a Wrap!

That's a Wrap!

Nice work! You just built a fully functional RAG API from scratch.

You've learned how to:

  • Built a document ingestion pipeline that chunks text and stores vector embeddings in ChromaDB.
  • Created a FastAPI REST endpoint that implements the full RAG pattern: retrieve, augment, generate.
  • Used Groq's free API to get fast LLM inference with Llama 3.3 70B.
  • Added a dynamic document upload endpoint for real-time knowledge base expansion.

Ready to quiz yourself?