Build a Local RAG API

Build a local API that grounds answers in private notes with source passages.

Introduction

30 Second Summary

A general-purpose assistant can answer broad questions. Your private notes stay invisible to it until you choose how they are used.

In this project, you will build a private question-answering REST API that keeps its processing and storage on your Mac. FastAPI, Chroma, and Ollama power a local retrieval-augmented generation flow.

What You'll Build

Your final demo shows the /ask endpoint answering a question about Blue Lantern with the matching passage from knowledge.txt displayed as evidence.

By the end of this project, you'll have:

  • An interactive question endpoint where sending JSON to /ask returns a response from Gemma 3 on your Mac.
  • A persistent knowledge index created from knowledge.txt that stays available after the API restarts.
  • A grounded response with sources that answers with Blue Lantern from the matching local passage.
  • Secret Mission: Add a /documents endpoint that embeds new knowledge while the API is running.

Are there any prerequisites?

You need a Mac running macOS 14 or newer. Setup also needs internet access plus free storage for the tools and approximately 1.44 GB of model files.

Before We Start

This is your moment to commit to building a private local API before the hands-on work begins. You will answer questions from personal text without sending that text to a hosted AI service.

Check Your Mac and Install the Local Runtimes

Your private API keeps personal text on your Mac. The Mac needs a compatible operating system before it can run the local model service.

The Ollama runtime requires macOS Sonoma 14 or newer. Python 3.13.15 runs the API code you build in later steps.

In this step, get ready to:
  • Confirm your Mac runs macOS Sonoma 14 or newer.
  • Install Python 3.13.15 with working pip.
  • Install Ollama 0.34.4 with a responsive local service.
Confirm your macOS version

Ollama needs macOS Sonoma 14 or newer. This compatibility check prevents an installation that your operating system cannot run.

  • Click the Apple menu in the top-left corner of your screen.
  • Select About This Mac.
  • Read the macOS name and version shown in the window.

✔️ I see macOS 14 or newer

Good, your Mac clears the compatibility gate. You can install the local runtimes without updating macOS.

ⓧ I see macOS 13 or earlier

A macOS update restarts your Mac. Saving your work first keeps the update uneventful.

  • Save your open work.
  • Click the Apple menu.
  • Select System Settings.
  • Select General in the sidebar.
  • Select Software Update.
  • Install an available update to macOS 14 or newer.
  • Click the Apple menu after your Mac restarts.
  • Select About This Mac to confirm the updated version.

No Compatible Update Available?

Your Mac must offer macOS 14 or newer before you continue. Use a compatible Mac if Software Update cannot provide that version.

Install Python and verify pip

Python runs the FastAPI service you build later. The pinned release keeps your environment consistent with the package versions used throughout this project.

  • Press Cmd+Space to open Spotlight.
  • Type Terminal and press Enter to open it.
  • Check the Python version currently available by running this command:
python3 --version

What Does This Check?

The --version option prints the release used by the python3 command. This reveals whether your terminal already points to Python 3.13.15.

Your result fits one of these three installation states.

✔️ I see Python 3.13.15

  • Confirm pip is available through this Python installation by running:
python3 -m pip --version

What Does This Check?

The command asks Python to run its pip module. The output confirms which pip installation your python3 command uses.

You should see a pip version followed by its installed path. That is Python sorted for the rest of the project.

ⓧ I see an older Python version

The Python.org installation can coexist with the older Python supplied by macOS. Its default installer settings make the newer python3 command available in fresh terminal sessions.

  • Open the official release page for Python 3.13.15.
  • Download the macOS installer from the release page.
  • Open the downloaded installer package.
  • Complete the installer using its default options.
  • Quit Terminal after the installation finishes.
  • Press Cmd+Space to open Spotlight.
  • Type Terminal and press Enter to open a fresh session.
  • Verify Python and pip by running these commands:
python3 --version
python3 -m pip --version

What Do These Checks Prove?

The first command confirms which Python release your terminal uses. The second confirms that the same installation can run pip.

You should see Python 3.13.15 first. The next line should show a pip version with its installed path.

ⓧ Command not found

Your terminal cannot currently find Python. Installing the pinned Python.org release adds the runtime needed by this project.

  • Open the official release page for Python 3.13.15.
  • Download the macOS installer from the release page.
  • Open the downloaded installer package.
  • Complete the installer using its default options.
  • Quit Terminal after the installation finishes.
  • Press Cmd+Space to open Spotlight.
  • Type Terminal and press Enter to open a fresh session.
  • Verify Python and pip by running these commands:
python3 --version
python3 -m pip --version

What Do These Checks Prove?

The first command confirms that the new Python runtime is on your shell path. The second confirms that Python can run its pip module.

You should see Python 3.13.15 first. The next line should show a pip version with its installed path.

Still Seeing the Wrong Python?

  • Quit every Terminal window before opening a fresh session through Spotlight.
  • Confirm that the Python installer reached its completion screen.
  • Rerun both checks after the new terminal session opens.

Still stuck? Help me troubleshoot my Python installation on macOS.

Install Ollama and verify the service

Ollama runs the generation and embedding models directly on your Mac. Its desktop application starts the local service while its CLI gives your Python code a way to use that service.

  • Check whether an Ollama version is already available by running:
ollama --version

What Does This Check?

This command asks the Ollama CLI to print its installed version. The result identifies whether the required release is already on your Mac.

Your result fits one of these three installation states.

✔️ I see Ollama 0.34.4

  • Press Cmd+Space to open Spotlight.
  • Type Ollama and press Enter to start the local service.
  • Allow Ollama to add its CLI link if macOS requests permission.
  • Confirm the installed version and local model list by running:
ollama --version
ollama ls

What Do These Checks Prove?

The first command confirms the installed Ollama release. The second asks the running local service for its model list.

You should see Ollama 0.34.4 followed by a local model list. An empty list is expected before you download models in the next step.

ⓧ I see an older Ollama version

The current application replaces the older release in Applications. The first launch may also request permission to add the Ollama CLI to your path.

  • Open the official Ollama macOS page.
  • Download the Ollama macOS disk image.
  • Mount the downloaded ollama.dmg file.
  • Drag the Ollama application into Applications.
  • Press Cmd+Space to open Spotlight.
  • Type Ollama and press Enter to start the updated application.
  • Allow Ollama to add its CLI link when macOS requests permission.
  • Verify the updated version and local service by running:
ollama --version
ollama ls

What Do These Checks Prove?

The version check confirms that the updated application supplies the CLI. The model-list command confirms that the local service accepts requests.

You should see Ollama 0.34.4 followed by a local model list. The list can be empty at this point.

ⓧ Command not found

Ollama is not available on your shell path yet. The macOS application installs the local service and offers to create the CLI link during its first launch.

  • Open the official Ollama macOS page.
  • Download the Ollama macOS disk image.
  • Mount the downloaded ollama.dmg file.
  • Drag the Ollama application into Applications.
  • Press Cmd+Space to open Spotlight.
  • Type Ollama and press Enter to start the application.
  • Allow Ollama to add its CLI link when macOS requests permission.
  • Verify the installed version and local service by running:
ollama --version
ollama ls

What Do These Checks Prove?

The version check confirms that the CLI link works. The model-list command confirms that the local Ollama service is responding.

You should see Ollama 0.34.4 followed by a local model list. An empty list is expected before the model downloads begin.

Ollama Command Still Unavailable?

  • Confirm that the Ollama application is inside Applications.
  • Quit Ollama before opening it again through Spotlight.
  • Allow the CLI-link permission request if it appears again.

Need another pair of eyes? Help me troubleshoot my Ollama installation.

Before you run the final check, do you expect both local runtimes to respond from the same terminal session?

  • Verify pip and the Ollama service together by running:
python3 -m pip --version
ollama ls

What Does the Final Check Prove?

The pip command proves that Python can load its package installer. The Ollama command proves that the local service is reachable through the CLI.

You should see pip's installed path followed by Ollama's local model list. Those two results confirm that both runtimes are ready.

Your Mac now meets the compatibility requirement. Next up, you'll isolate the project dependencies and download the two local models.

Prepare the Project and Download the Models

Your Mac now has Python 3.13.15 plus a running Ollama service. The local RAG API can use both runtimes.

A virtual environment keeps this project's packages separate from the rest of your Mac. Pinned dependencies make the build repeatable.

Two local models divide the AI work. Gemma 3 generates text.

EmbeddingGemma creates embeddings for semantic retrieval. This step prepares both models inside a Visual Studio Code workspace.

In this step, get ready to:
  • Create the local-rag-api workspace with an active virtual environment.
  • Install the pinned FastAPI, Pydantic, Chroma, and Ollama Python dependencies.
  • Download the Gemma 3 chat model plus the EmbeddingGemma embedding model.
Create the project environment

The named project folder gives every later file a predictable home. The virtual environment stores this project's packages inside .venv.

  • Press Command-Space to open Spotlight.
  • Type Terminal in the search field.
  • Press Return to open Terminal.
  • Create the local-rag-api folder on your Desktop by running these commands:
cd ~/Desktop
mkdir local-rag-api
cd local-rag-api
pwd

What do these commands do?

  • The first cd command moves the terminal to your Desktop.
  • The mkdir command creates the local-rag-api folder.
  • The second cd command enters the new folder.
  • The pwd command prints the folder's full path so you can confirm your location.

You should see a path ending in Desktop/local-rag-api. This confirms the project folder exists in the expected location.

  • Press Command-Space to return to Spotlight.
  • Type Visual Studio Code in the search field.
  • Press Return to open Visual Studio Code.
  • Click File in the menu bar.
  • Click Open Folder....
  • Select the local-rag-api folder on your Desktop.
  • Click Open.

You should see local-rag-api at the top of the Explorer sidebar. Visual Studio Code now treats this folder as your workspace.

  • Click View in the menu bar.
  • Click Terminal to open the integrated terminal.
  • Create the virtual environment plus activate it by running these commands:
python3 -m venv .venv
source .venv/bin/activate

What does this environment do?

The first command creates an isolated Python environment inside .venv. The second command directs Python package installations into that environment.

Good progress. Your terminal prompt should now begin with (.venv), which confirms the project environment is active.

Virtual environment not active?

  • Check that the integrated terminal is inside the local-rag-api folder.
  • Run the activation command again if (.venv) is missing from the prompt.
  • Confirm that .venv appears in the Explorer sidebar.

Still stuck? Help me activate the virtual environment in my local-rag-api folder on macOS.

Install the pinned dependencies

The requirements.txt file records the exact top-level package versions used by this project. Installing from this file keeps your local environment aligned with the code ahead.

  • Click the New File button in the Explorer sidebar.
  • Enter requirements.txt as the file name.
  • Press Return to create the file.

You should see an empty requirements.txt editor plus the new file in the Explorer sidebar.

  • Add the pinned dependencies by copying this content into requirements.txt:
fastapi[standard-no-fastapi-cloud-cli]==0.141.1
pydantic==2.13.5
chromadb==1.5.9
ollama==0.6.2

What are these dependencies?

  • FastAPI provides the web framework plus the local development server tools.
  • Pydantic validates the JSON request bodies sent to your API.
  • Chroma stores embeddings on disk for similarity searches.
  • Ollama Python lets your Python files call the running Ollama service.
  • Save requirements.txt.

Package installation can take a few minutes while Python downloads compatible files. A busy terminal during this wait means the installer is still working.

  • Install the pinned dependencies into .venv by running:
python3 -m pip install -r requirements.txt

What does this command do?

Python asks pip to read every package specification from requirements.txt. The active virtual environment keeps the resulting installation inside .venv.

That's the dependency layer ready. The terminal should return to the (.venv) prompt without an installation error.

Dependency installation failed?

  • Confirm that the terminal prompt begins with (.venv).
  • Check that requirements.txt contains all four lines exactly as shown.
  • Confirm that the terminal is inside the local-rag-api folder.

Need another pair of eyes? Help me diagnose my requirements installation error.

✔️ Awesome, I've got everything!

Your saved requirements.txt now records the four pinned project dependencies.

ⓧ I'd like to double check the full code

  • Compare your complete requirements.txt file with this reference:
fastapi[standard-no-fastapi-cloud-cli]==0.141.1
pydantic==2.13.5
chromadb==1.5.9
ollama==0.6.2

What should match?

The package names plus version pins should match this reference character for character. That exact set produces the dependency state used throughout the project.

Download the local models

The project uses gemma3:1b to generate answers. It uses embeddinggemma to convert text into vectors for retrieval.

  • Click the New File button in the Explorer sidebar.
  • Enter download_models.py as the file name.
  • Press Return to create the file.
  • Add the model download script by copying this code into download_models.py:
import ollama

MODELS = ("gemma3:1b", "embeddinggemma")

for model in MODELS:
    print(f"Downloading {model}...")
    ollama.pull(model)

print("Models ready.")

What does this script do?

  • The ollama import gives the script access to the local model service.
  • The MODELS tuple stores both model identifiers in one place.
  • The loop reports the current model before ollama.pull() downloads it.
  • The final message appears only after both pull operations finish.
  • Save download_models.py.

The first run downloads approximately 1.44 GB of model files. Download time varies with your connection speed, so this may take several minutes.

Before you run the script, do you think the readiness line appears before or after both downloads finish?

  • Download both models through the running Ollama service by running:
python3 download_models.py

What does this command run?

Python executes download_models.py inside the active virtual environment. The script sends each model identifier to the local Ollama service in sequence.

You should see Downloading gemma3:1b... followed by Downloading embeddinggemma.... The terminal ends with Models ready. after both downloads complete.

Model download interrupted?

  • Confirm that the Ollama application from the previous step is still running.
  • Check your internet connection before running the script again.
  • Keep enough free storage available for approximately 1.44 GB of model files.

Still unable to finish the downloads? Help me troubleshoot my Ollama model download script.

Before you inspect the environment, which four top-level package pins do you expect to find?

  • Verify the installed package versions by running:
python3 -m pip freeze

What does this command show?

The pip freeze output lists installed package specifiers plus versions from the active virtual environment. It lets you compare the environment with the pins in requirements.txt.

You should find these pinned packages in the output:

  • FastAPI 0.141.1.
  • Pydantic 2.13.5.
  • Chroma 1.5.9.
  • Ollama Python 0.6.2.

Your project now has its isolated dependencies plus both local models. The environment is ready for the first API endpoint.

✔️ Awesome, I've got everything!

Your saved download_models.py downloads both required models before printing its readiness message.

ⓧ I'd like to double check the full code

  • Compare your complete download_models.py file with this reference:
import ollama

MODELS = ("gemma3:1b", "embeddinggemma")

for model in MODELS:
    print(f"Downloading {model}...")
    ollama.pull(model)

print("Models ready.")

What should match?

The import, model tuple, download loop, plus final message should match this reference exactly. This sequence guarantees that both model pulls finish before the script reports success.

Your workspace now has the packages plus models required for a local AI service. Next up, you'll build the first working API and expose the missing private context problem.

Build the Baseline API

The two local models are ready. Your project still needs a FastAPI entry point that can accept an HTTP request.

Your private codename sits outside Ollama's built-in model knowledge. This baseline makes that missing context visible in every response.

In this step, get ready to:
  • Create the application foundation for typed questions.
  • Add a health route that proves the service is running.
  • Send a private question through the baseline /ask route.
Create the API foundation

Pydantic defines the required shape of the JSON request. The first version requires one string named question.

  • In the VS Code Explorer sidebar, create main.py inside the local-rag-api folder with the code below:
import ollama
from fastapi import FastAPI
from pydantic import BaseModel

CHAT_MODEL = "gemma3:1b"

app = FastAPI(title="Local RAG API")


class Question(BaseModel):
    question: str

What does this code do?

  • The imports provide the web framework, request model, and local model client.
  • CHAT_MODEL keeps the generation model ID in one place.
  • app creates the API with the title shown in its interactive documentation.
  • Question requires every request to contain a string named question.
  • Save main.py.
  • Return to the terminal from the previous step.
  • Start the development server by running this command:
fastapi dev

What does this command do?

fastapi dev finds the app object in main.py. It starts the local server with automatic reloads for saved changes.

You should see output showing that the server is available at http://127.0.0.1:8000. The busy terminal confirms that your API process is running.

  • Leave this terminal running while you continue through the step.

Server not starting?

  • Confirm that main.py is saved inside the local-rag-api folder.
  • Confirm that the terminal is still using the virtual environment from the previous step.
  • Check the imports in main.py for spelling differences.

Still stuck? Help me debug why fastapi dev cannot start my main.py application.

Add the health route

A health route gives you a quick way to prove that the server can receive a request. This route returns a fixed message without calling the model.

  • Add the health route below the Question model in main.py by copying this code:
@app.get("/")
def root() -> dict:
    return {"message": "Local RAG API is running"}

What does this code do?

  • @app.get("/") connects requests for the root path to root().
  • The returned dictionary becomes a JSON response that confirms the API process is reachable.
  • Save main.py.
  • Open http://127.0.0.1:8000 in your browser.

You should see a JSON response with a message field containing Local RAG API is running. Your browser has now completed a request to the local service.

Health message missing?

  • Confirm that the development server is still running in the terminal.
  • Check that the route uses @app.get("/") above root().
  • Save main.py again if the browser still shows the earlier state.

Need another pair of eyes? Help me find why my FastAPI root route does not return its health message.

Add and test the baseline ask route

The baseline /ask route sends each question straight to the chat model. It also reports whether private passages supported the answer.

  • Add the baseline route below root() in main.py by copying this code:
@app.post("/ask")
def ask(request: Question) -> dict:
    response = ollama.chat(
        model=CHAT_MODEL,
        messages=[{"role": "user", "content": request.question}],
    )
    return {
        "answer": response.message.content,
        "context_used": False,
    }

What does this code do?

  • @app.post("/ask") creates an endpoint that accepts the validated Question body.
  • ollama.chat() sends request.question to the model stored in CHAT_MODEL.
  • response.message.content extracts the model's answer for the response.
  • context_used records whether private source passages took part in generating that answer.
  • Save main.py.
  • In the browser from earlier, go to http://127.0.0.1:8000/docs.
  • Expand the POST /ask row.
  • Click Try it out.
  • Replace the example request body with this private question:
{"question":"What is my weekend project codename?"}

Why use this question?

This question targets a personal fact that has not been supplied to the endpoint. The answer text can vary because the model only receives the question.

The context flag provides a consistent signal. It tells you whether retrieved private passages supported the response.

Before you send the request, do you think the API can prove that it used private knowledge?

  • Click Execute to send the request.

You will see a JSON response containing an answer field. The context_used field is false.

That result is the intended shortfall. No private passage supported the model's response.

That's a solid checkpoint: your local model now responds through a typed API.

Request not completing?

  • Confirm that Ollama remains open on your Mac.
  • Check that CHAT_MODEL is set to gemma3:1b.
  • Confirm that the request body contains a string under question.

Still seeing a failed request? Help me debug my baseline POST /ask route and its Ollama chat call.

✔️ Awesome, I've got everything!

Your baseline API is running. Keep the development server active for the next step.

ⓧ I'd like to double check the full code

  • Compare main.py with the complete baseline below.
import ollama
from fastapi import FastAPI
from pydantic import BaseModel

CHAT_MODEL = "gemma3:1b"

app = FastAPI(title="Local RAG API")


class Question(BaseModel):
    question: str


@app.get("/")
def root() -> dict:
    return {"message": "Local RAG API is running"}


@app.post("/ask")
def ask(request: Question) -> dict:
    response = ollama.chat(
        model=CHAT_MODEL,
        messages=[{"role": "user", "content": request.question}],
    )
    return {
        "answer": response.message.content,
        "context_used": False,
    }

Your API now exposes a working local question endpoint. Next, you will turn private text into searchable knowledge that the endpoint can use.

Build the Persistent Knowledge Index

The baseline FastAPI service accepts questions. The local model still has no way to find facts from your private notes.

This step turns three local paragraphs into embeddings with Ollama. Chroma stores those vectors on disk so the next step can retrieve relevant evidence.

In this step, get ready to:
  • Create a private knowledge file with three focused paragraphs.
  • Build an ingestion script that embeds each paragraph.
  • Run ingestion twice to verify the collection stays at three records.
Create the private knowledge file

Each paragraph becomes a focused retrieval chunk. Later, vector similarity search can compare a question with these chunks to find relevant evidence.

  • Use the Explorer sidebar in Visual Studio Code to create knowledge.txt inside the local-rag-api folder.
  • Add the three private facts by copying this text into knowledge.txt:
My weekend project codename is Blue Lantern. It is a tiny local assistant that answers questions from my own notes.

My preferred coding language for this project is Python because I want a clear, readable API implementation.

I want the project data to stay on this Mac, and I want every answer to include the local passages used as evidence.

Why separate the facts?

  • The first paragraph holds the private codename Blue Lantern.
  • The second paragraph records your language preference.
  • The third paragraph records your local-data requirement.
  • The blank lines give the paragraph splitter clear chunk boundaries.
  • Save knowledge.txt.
  • Confirm the editor shows exactly three paragraphs with a blank line between each one.

Your private knowledge source is ready. Each fact now has its own clean passage for retrieval.

Do the paragraphs run together?

  • Check that each paragraph matches the text above.
  • Add one empty line between each paragraph if the facts appear as a single block.
  • Confirm the file is named knowledge.txt inside local-rag-api.

Still stuck? Help me check the formatting and location of my knowledge.txt file.

Build the ingestion script

The ingestion script converts every paragraph into an embedding. Stable record IDs let Chroma update the same entries whenever you rerun the script.

  • Use the Explorer sidebar to create ingest.py inside the local-rag-api folder.
  • Set up the imports, storage settings, and paragraph splitter by copying this code into ingest.py:
from pathlib import Path

import chromadb
import ollama

DB_PATH = "./chroma_data"
COLLECTION_NAME = "learner_profile"
EMBEDDING_MODEL = "embeddinggemma"


def split_paragraphs(text: str) -> list[str]:
    return [paragraph.strip() for paragraph in text.split("\n\n") if paragraph.strip()]

What does this code do?

  • Path gives the script a direct way to read knowledge.txt.
  • DB_PATH points Chroma at the local chroma_data folder.
  • COLLECTION_NAME keeps all three records together in learner_profile.
  • split_paragraphs() separates the file at blank lines. It also removes empty chunks.
  • Save ingest.py.
  • Confirm the Explorer sidebar lists ingest.py beside knowledge.txt.

Seeing import warnings?

  • Check that Visual Studio Code is using the Python interpreter from .venv.
  • Confirm requirements.txt still lists the pinned Chroma package and Ollama Python package.
  • Confirm your terminal prompt still shows the active virtual environment.

Need a hand? Help me resolve the import warnings in ingest.py.

  • Add the ingestion workflow below split_paragraphs() in ingest.py:
def main() -> None:
    text = Path("knowledge.txt").read_text(encoding="utf-8")
    documents = split_paragraphs(text)
    embeddings = ollama.embed(model=EMBEDDING_MODEL, input=documents).embeddings

    client = chromadb.PersistentClient(path=DB_PATH)
    collection = client.get_or_create_collection(
        name=COLLECTION_NAME,
        embedding_function=None,
    )
    collection.upsert(
        ids=[f"profile-{index}" for index in range(len(documents))],
        embeddings=embeddings,
        documents=documents,
        metadatas=[{"source": "knowledge.txt"} for _ in documents],
    )

    print(f"Indexed {collection.count()} chunks.")


if __name__ == "__main__":
    main()

How does ingestion work?

  • main() reads the private text. It passes the result through split_paragraphs().
  • ollama.embed() generates one embedding for each document with embeddinggemma.
  • PersistentClient saves the collection inside ./chroma_data.
  • collection.upsert() stores stable IDs from profile-0 through profile-2. Each record receives {"source": "knowledge.txt"} metadata.
  • Save ingest.py.
  • Return to the terminal running the FastAPI development server.
  • Press Control-C to stop the server process.

The development server currently owns your terminal. Stopping that process leaves main.py intact.

Before you build the index, how many chunks do you expect the three paragraphs to create?

  • Build the first persistent index by running this command:
python3 ingest.py

What does this command do?

The script executes main(). It reads the private text before embedding each chunk.

Chroma creates chroma_data on disk. The final count reports how many records the collection contains.

You should see Indexed 3 chunks. in the terminal. That is the storage layer working with all three private passages.

Did ingestion fail?

  • Confirm knowledge.txt sits beside ingest.py inside local-rag-api.
  • Confirm Ollama is still running with embeddinggemma available.
  • Check that the terminal prompt shows the active .venv environment if Python cannot import a package.

Still blocked? Help me troubleshoot why ingest.py does not print Indexed 3 chunks.

✔️ Awesome, I've got everything!

Great. Save ingest.py before checking persistence one more time.

ⓧ I'd like to double check the full code

from pathlib import Path

import chromadb
import ollama

DB_PATH = "./chroma_data"
COLLECTION_NAME = "learner_profile"
EMBEDDING_MODEL = "embeddinggemma"


def split_paragraphs(text: str) -> list[str]:
    return [paragraph.strip() for paragraph in text.split("\n\n") if paragraph.strip()]


def main() -> None:
    text = Path("knowledge.txt").read_text(encoding="utf-8")
    documents = split_paragraphs(text)
    embeddings = ollama.embed(model=EMBEDDING_MODEL, input=documents).embeddings

    client = chromadb.PersistentClient(path=DB_PATH)
    collection = client.get_or_create_collection(
        name=COLLECTION_NAME,
        embedding_function=None,
    )
    collection.upsert(
        ids=[f"profile-{index}" for index in range(len(documents))],
        embeddings=embeddings,
        documents=documents,
        metadatas=[{"source": "knowledge.txt"} for _ in documents],
    )

    print(f"Indexed {collection.count()} chunks.")


if __name__ == "__main__":
    main()

How the full file fits together

  • The top section defines the storage path, collection name, and embedding model.
  • The helper turns blank-line-separated text into focused documents.
  • The main workflow embeds each document before upserting it into persistent storage.
  • The final guard runs main() when you execute ingest.py.
Verify the index persists

The stable IDs make ingestion repeatable. Upserting the same IDs updates the existing records instead of creating duplicates.

Before you run the script again, do you expect the collection count to grow past three?

  • Test the persistent collection by running the ingestion script again:
python3 ingest.py

Why run ingestion twice?

The second run reconnects to the collection stored in chroma_data. It submits the same three stable IDs.

The unchanged count proves that each existing record was updated. No duplicate records were added.

You should see Indexed 3 chunks. again. Your private index now survives repeated ingestion without growing duplicate entries.

  • Confirm the Explorer sidebar now lists the chroma_data folder inside local-rag-api.

That folder is your persistent vector storage. It contains the learner_profile collection with all three embedded passages.

Did the count change?

  • Check that the ID expression still starts with profile-.
  • Confirm knowledge.txt still contains exactly three paragraphs.
  • Compare ingest.py with the full-file reference above if the count is not three.

Need help with the unexpected count? Help me find why my Chroma collection does not stay at three records.

Your private knowledge is embedded and stored on disk. Next, you will connect this index to the API so every answer can use retrieved evidence.

Build the Grounded RAG API

Your persistent Chroma index now holds three private passages, including the Blue Lantern codename. The baseline API still sends each question to Gemma 3 without retrieving those passages.

A retrieval-augmented generation loop solves that gap by fetching the closest passages before the model answers.

In this step, you'll connect FastAPI to the stored collection. You'll return each answer with the local passages that supported it.

In this step, get ready to:
  • Connect the API to the persistent knowledge collection.
  • Retrieve two nearby passages to ground the model prompt.
  • Verify the private fact response plus the missing fact fallback.
Connect the API to Chroma

The API needs to open the same collection that ingest.py populated. A persistent client loads the records already stored inside chroma_data.

  • In main.py, find the section from import ollama through app = FastAPI(title="Local RAG API").
  • Replace that section with the connection setup below:
import chromadb
import ollama
from fastapi import FastAPI
from pydantic import BaseModel

DB_PATH = "./chroma_data"
COLLECTION_NAME = "learner_profile"
EMBEDDING_MODEL = "embeddinggemma"
CHAT_MODEL = "gemma3:1b"

app = FastAPI(title="Local RAG API")
client = chromadb.PersistentClient(path=DB_PATH)
collection = client.get_or_create_collection(
    name=COLLECTION_NAME,
    embedding_function=None,
)

What does this code do?

  • The chromadb import gives the API access to the persistent vector store.
  • The three new constants match the database path, collection name, and embedding model used by ingest.py.
  • PersistentClient loads the collection from disk whenever the API starts.
  • embedding_function=None keeps embedding generation under your control through Ollama.
  • Save main.py.
  • Switch back to the terminal from the previous step.
  • Start the development server by running:
fastapi dev

What does this command do?

The fastapi CLI finds the app object in main.py. Development mode reloads the service after saved code changes.

The server stays available at http://127.0.0.1:8000 while this terminal remains active.

  • Open http://127.0.0.1:8000 in your browser.

You'll see a JSON message containing Local RAG API is running. This confirms the API can load its persistent Chroma connection during startup.

Server not loading?

  • Check that the terminal prompt still shows (.venv) before you start the server.
  • Stop any older development server with Control+C if the port is already occupied.
  • Confirm that chroma_data remains inside the local-rag-api folder.

Still stuck? Help me debug why my FastAPI server cannot load the persistent Chroma collection.

Ground each answer in retrieved passages

Each question needs an embedding in the same vector space as the stored passages. Ollama creates that query vector with embeddinggemma.

Chroma compares the query vector with the stored vectors. The two closest passages become the context for the chat model.

  • In main.py, find the existing response assignment inside ask().
  • Replace that assignment with the retrieval and grounded generation code below:
    query_embedding = ollama.embed(
        model=EMBEDDING_MODEL,
        input=request.question,
    ).embeddings[0]

    results = collection.query(
        query_embeddings=[query_embedding],
        n_results=2,
        include=["documents"],
    )
    document_batch = results["documents"][0] if results["documents"] else []
    documents = [document for document in document_batch if document]
    context = "\n\n".join(documents)

    prompt = (
        "Answer only from the context below. "
        "If the answer is not in the context, say: "
        "I don't know from the provided context.\n\n"
        f"Context:\n{context}\n\n"
        f"Question: {request.question}"
    )
    response = ollama.chat(
        model=CHAT_MODEL,
        messages=[{"role": "user", "content": prompt}],
    )

How does retrieval work?

  • query_embedding converts the incoming question into a vector using the same model as the knowledge index.
  • collection.query() requests the two nearest stored documents.
  • context joins the retrieved passages into one block for the model.
  • prompt limits the answer to the retrieved evidence. It also supplies a fallback for missing facts.
  • Find the existing return block directly below the new response call.
  • Replace that return block with the response shape below:
    return {
        "answer": response.message.content,
        "sources": documents,
    }

Why return sources?

The answer field contains the model output. The sources field exposes the passages supplied as evidence.

That evidence makes the grounding visible to anyone testing the API.

  • Save main.py.

The first local generation may pause while Gemma 3 becomes active. That pause is normal.

Before you submit the private question, do you expect the response to contain only an answer or an answer with supporting evidence?

  • Return to the browser tab from earlier.
  • Go to http://127.0.0.1:8000/docs.
  • Expand POST /ask in the interactive docs.
  • Enable editing for the request body.
  • Enter What is my weekend project codename? as the question value.
  • Submit the request.

You'll see an answer containing Blue Lantern. The sources array includes the matching paragraph from knowledge.txt.

You have crossed the key threshold. Your API can now show the local evidence behind its answer.

Blue Lantern missing?

  • Confirm that the development server reloaded after you saved main.py.
  • Confirm that knowledge.txt still contains the Blue Lantern paragraph.
  • Confirm that chroma_data contains the collection created by ingest.py.

Need another pair of eyes? Help me find why my grounded /ask response does not contain Blue Lantern.

Verify grounded answers and the fallback

Grounding also controls what happens when the retrieved passages lack an answer. The prompt tells the model to use a fixed fallback instead of inventing a private fact.

Before you submit a missing fact, do you think the model will invent a preference or follow the context rule?

  • Replace the current question value with What is my favorite database?.
  • Submit the request.

You'll see I don't know from the provided context. in the answer. This confirms the model does not claim an unsupported preference.

The sources array still shows the nearest retrieved passages. You can inspect that evidence to confirm the missing fact does not appear there.

Fallback not appearing?

  • Confirm that the prompt in main.py contains the context-only instruction.
  • Confirm that your test question asks about a fact absent from knowledge.txt.
  • Save main.py so the development server reloads the latest prompt.

Still getting an unsupported answer? Help me debug why Gemma 3 is not following the missing-context fallback in my RAG prompt.

✔️ Awesome, I've got everything!

Great work. Keep the FastAPI development server running so you can continue testing the grounded endpoint.

ⓧ I'd like to double check the full code

  • Compare your complete main.py with this reference:
import chromadb
import ollama
from fastapi import FastAPI
from pydantic import BaseModel

DB_PATH = "./chroma_data"
COLLECTION_NAME = "learner_profile"
EMBEDDING_MODEL = "embeddinggemma"
CHAT_MODEL = "gemma3:1b"

app = FastAPI(title="Local RAG API")
client = chromadb.PersistentClient(path=DB_PATH)
collection = client.get_or_create_collection(
    name=COLLECTION_NAME,
    embedding_function=None,
)


class Question(BaseModel):
    question: str


@app.get("/")
def root() -> dict:
    return {"message": "Local RAG API is running"}


@app.post("/ask")
def ask(request: Question) -> dict:
    query_embedding = ollama.embed(
        model=EMBEDDING_MODEL,
        input=request.question,
    ).embeddings[0]

    results = collection.query(
        query_embeddings=[query_embedding],
        n_results=2,
        include=["documents"],
    )
    document_batch = results["documents"][0] if results["documents"] else []
    documents = [document for document in document_batch if document]
    context = "\n\n".join(documents)

    prompt = (
        "Answer only from the context below. "
        "If the answer is not in the context, say: "
        "I don't know from the provided context.\n\n"
        f"Context:\n{context}\n\n"
        f"Question: {request.question}"
    )
    response = ollama.chat(
        model=CHAT_MODEL,
        messages=[{"role": "user", "content": prompt}],
    )

    return {
        "answer": response.message.content,
        "sources": documents,
    }

What should the full file contain?

The file opens the persistent collection before defining the request model and routes. The ask() route embeds each question before querying two passages.

The final response contains answer plus sources. Those fields are the visible result of the complete retrieval loop.

That's the full local RAG loop working. Your API retrieves private passages, grounds Gemma 3 with them, and returns its evidence.

Secret mission

Add Documents Through the API

Your API can retrieve facts that were indexed before startup. Extend it with a runtime endpoint that embeds new text and makes it searchable immediately.

Clean Up Your Resources

Clean Up Your Resources

Everything in this project runs locally on your Mac, so there are no ongoing cloud costs. Decide whether to keep your resources running, pause them for later, or delete them entirely.

Resources you used:

  • A FastAPI development server running at http://127.0.0.1:8000.
  • The local-rag-api project folder containing .venv, chroma_data, main.py, ingest.py, download_models.py, knowledge.txt, and requirements.txt.
  • A persistent Chroma database inside chroma_data containing the three profile records plus demo-4.
  • Approximately 1.44 GB of Ollama model files for gemma3:1b and embeddinggemma.

Keep everything running

No action is needed. Choose this if you want to keep testing the API or adding knowledge through the interactive docs.

  • Keep the FastAPI development server running in its current terminal.
  • Leave Ollama running so the local models remain available.
  • Keep the local-rag-api folder on your Mac so the virtual environment and indexed knowledge remain available.

Pause - I'll come back to this later

Shut down the development server to free its port. Your files, indexed knowledge, and downloaded models stay ready for another session.

  • Switch back to the terminal running the FastAPI development server.
  • Press Control-C to stop the server.

The API stops accepting requests. Your chroma_data database and Ollama models remain on disk.

Delete - I don't want to use this again

Remove the project files and downloaded models if you want a clean start. Removing the models means you need to download them again before rebuilding the API.

  • Switch back to the terminal running the FastAPI development server.
  • Press Control-C to stop the server.

The server stops using the project files, so the folder is ready to remove.

  • In Finder, move the local-rag-api folder to Trash.

The .venv environment and chroma_data database move to Trash with the project folder.

  • Remove both downloaded model files by running these commands:
ollama rm gemma3:1b
ollama rm embeddinggemma

What Gets Removed?

These commands delete the two downloaded model files from Ollama's local storage. They leave the Ollama application installed on your Mac.

  • Confirm the local model list has updated by running:
ollama ls

What Should You See?

You should no longer see gemma3:1b or embeddinggemma in the model list. This confirms that their local model storage has been reclaimed.

Nice Work!

Nice Work!

Mission complete! You built a private RAG API that runs locally on your Mac. It retrieves evidence from Chroma before Gemma 3 answers through FastAPI.

You've learned how to:

  • Built a typed FastAPI service with Pydantic request validation. Exercised the /ask endpoint through the interactive docs.
  • Generated local embeddings through Ollama with EmbeddingGemma. Stored those vectors in a persistent Chroma collection that survives API restarts.
  • Used vector similarity search to retrieve the nearest private passages before generation. Returned those passages in the sources array as visible evidence for Blue Lantern.
  • Completed the optional Secret Mission by adding runtime document ingestion through /documents. Embedded and retrieved demo-4 without rerunning ingest.py.

Ready to quiz yourself?