RAG Microservice with FastAPI (Phase 1)

Build a local RAG API that answers questions over your own documents.

Introduction

30 Second Summary

Most AI chatbots can only answer from what they were trained on. If you want an AI that knows about your specific documents, you need a way to feed it the right context at the right time.

In this project, you will build a local RAG microservice that answers questions over your own documents using FastAPI, LangChain, and a locally running Llama 3.2 model.

What You'll Build

You'll POST a question to your API, watch it pull the most relevant chunks from your document library using vector search, and get back a grounded answer with source citations and latency metrics.

By the end of this project, you'll have:

  • A RAG pipeline that loads your documents, chunks them, embeds them into vectors, and retrieves the most relevant pieces when you ask a question.
  • A FastAPI microservice with typed Pydantic models where you POST a question and get back a JSON response with the answer, source chunks, and per-request performance timing.
  • Structured logging middleware that shows you exactly how long retrieval vs. generation takes on every request, so you always know where the bottleneck is.
  • Secret Mission: Add a live document ingestion endpoint that accepts new documents on the fly and rebuilds the vector index without restarting the server.

Are there any prerequisites?

You should be comfortable writing Python and have a basic understanding of REST APIs. Step 1 walks you through verifying your local tools, so no prior Ollama or LangChain experience is needed.

Before We Start

A RAG microservice needs several pieces working together: a local LLM to generate answers, vector search to find relevant documents, and a web framework to expose it all as an API.

Before we build anything, we need to confirm Ollama is serving a local model and get a Python environment ready with all the libraries the pipeline requires.

In this step, get ready to:
  • Verify Ollama is installed and running on your machine.
  • Pull the llama3.2 model for local inference.
  • Create the project directory and install all Python dependencies.
Verify Ollama is installed

Ollama runs large language models locally on your machine as a background service. It's what lets you query an LLM without any cloud API calls or costs.

  • Open your terminal by pressing Cmd+Space (macOS) or the Windows key (Windows), typing Terminal, and pressing Enter.
  • Check if Ollama is installed by running this command:
ollama --version

You should see a version number printed to the terminal.

✔️ I see a version number

Ollama is installed and ready. Move on to the next substep.

ⓧ Command not found

Ollama is not installed yet. Follow these steps to install it.

  • Download the Ollama installer from https://ollama.com/download/Ollama.dmg.
  • Open the downloaded DMG file and drag Ollama into your Applications folder.
  • Launch Ollama from your Applications folder. You will see a llama icon appear in your menu bar. This means Ollama is running as a background service.
  • Return to your terminal and run ollama --version again to confirm it works.

What about Windows?

On Windows, download the installer from https://ollama.com/download and run the .exe installer instead.

macOS version requirement

Ollama requires macOS Sonoma (v14) or newer. If you are on an older version of macOS, you will need to update before installing Ollama.

Pull the llama3.2 model

Ollama needs a model downloaded before it can generate text. The llama3.2 model is a 3B parameter model that runs well on most machines.

  • Download the llama3.2 model by running this command:
ollama pull llama3.2

What does this command do?

This downloads approximately 2 GB of model weights to your machine. The download may take a few minutes depending on your internet connection.

Once complete, the model is stored locally and ready to use without any further downloads.

  • Confirm the model is available by running:
ollama list

You should see llama3.2 listed in the output.

Before you test the model, what do you think will happen when you ask it to say hello?

  • Test that the model responds by running:
ollama run llama3.2 "Say hello"

You should see a text response from the model printed to your terminal. This confirms the full local LLM stack is working.

Exit the interactive prompt

If Ollama drops you into an interactive chat session after responding, type /bye and press Enter to exit back to your terminal.

Not seeing a response?

Make sure Ollama is running as a background service. Look for the llama icon in your menu bar (macOS). If it is missing, launch Ollama from your Applications folder.

If you see a connection error, Ollama may not have finished starting. Wait a few seconds and try again.

help me troubleshoot why ollama run llama3.2 is not returning a response

Create the project and install dependencies

Now that the LLM is working, you need a Python project with the libraries that connect everything together.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir rag-service
cd rag-service
  • Confirm you are in the new directory by running:
pwd

You should see a path ending in /Desktop/rag-service.

  • Create a Python virtual environment by running:
python3 -m venv .venv

What is a virtual environment?

A virtual environment is an isolated Python installation specific to this project. Packages you install here will not affect any other Python projects on your machine.

  • Activate the virtual environment:

macOS

source .venv/bin/activate

Windows

.venv\Scripts\activate

You should see (.venv) appear at the beginning of your terminal prompt. This confirms the virtual environment is active.

Don't see (.venv) in your prompt?

Make sure you ran the activate command from inside the rag-service folder. Run pwd to confirm your location, then try the activate command again.

help me activate my Python virtual environment

  • Install all the project dependencies by running:
pip install "fastapi[standard]" langchain-ollama langchain-huggingface langchain-community faiss-cpu langchain-text-splitters

What are these packages?

  • fastapi[standard] is the web framework that will serve your API endpoints.
  • langchain-ollama connects LangChain to your local Ollama model.
  • langchain-huggingface provides the embedding model that turns text into vectors.
  • langchain-community includes the FAISS vector store integration.
  • faiss-cpu is the vector similarity search library that finds relevant document chunks.
  • langchain-text-splitters breaks documents into smaller chunks for embedding.

The install will take a minute or two. When it finishes, you should see a line that starts with Successfully installed followed by a list of all the packages.

Seeing an installation error?

Make sure (.venv) is visible in your prompt before running the install. Without an active virtual environment, you may hit permissions errors.

If you see a version conflict, make sure you are using Python 3.11 by running python3 --version. All packages require Python 3.10 or newer.

help me fix a pip install error in my RAG service project

Your local LLM is responding and your Python environment has everything it needs. Next up, you will build a RAG chain that loads documents, embeds them, and generates answers.

Build a Minimal RAG Chain

You have a working local LLM and all your dependencies installed. Now you need to prove the full RAG pipeline works end-to-end before wrapping it in an API.

A RAG pipeline has four stages: load documents, split them into chunks, embed those chunks into a vector store, and connect a retriever to an LLM that uses the retrieved context to answer questions. In this step you will build all four stages in a single script and see a grounded answer printed to your terminal.

In this step, get ready to:
  • Create sample documents that your RAG pipeline can search over.
  • Build a LangChain LCEL chain connecting FAISS vector search to Ollama.
  • Run the chain and see an answer grounded in your document content.
Create sample documents

Your RAG pipeline needs documents to search over. You will create two short markdown files with distinct facts so you can verify the pipeline retrieves the correct content for a given question.

  • Create a docs/ folder inside your rag-service/ directory by running this command:
mkdir docs
  • Confirm the folder was created by running:
ls

You should see docs listed alongside your .venv folder.

  • In VS Code, create a new file called company_policy.md inside the docs/ folder and paste this content:
# Remote Work Policy

All employees may work remotely up to 3 days per week. Remote work days must be agreed upon with your direct manager at least one week in advance.

## Equipment

The company provides a laptop and one external monitor for remote workers. Additional equipment can be requested through the IT portal with manager approval.

## Communication

All remote workers must be available on Slack during core hours (10am - 4pm local time). Video calls are expected for all team meetings.

## Expense Reimbursement

Remote workers can claim up to $50/month for internet costs and $25/month for electricity. Submit receipts through the expense portal by the 5th of each month.
  • Create a second file called product_faq.md inside the docs/ folder and paste this content:
# Product FAQ

## What is AcmeCRM?

AcmeCRM is a customer relationship management platform designed for small businesses with 5-50 employees. It includes contact management, deal tracking, and email integration.

## Pricing

AcmeCRM offers three tiers:
- Starter: $15/user/month (up to 1,000 contacts)
- Professional: $35/user/month (up to 10,000 contacts, includes reporting)
- Enterprise: $65/user/month (unlimited contacts, API access, dedicated support)

## Integrations

AcmeCRM integrates with Gmail, Outlook, Slack, Zapier, and QuickBooks. Custom integrations are available via the REST API on the Professional and Enterprise plans.

## Support

All plans include email support with 24-hour response time. Professional and Enterprise plans include live chat during business hours (9am - 6pm EST).
  • Save both files, then confirm they exist by running:
ls docs/

You should see company_policy.md and product_faq.md listed.

Why two documents with different topics?

Each file covers a distinct subject (remote work policy vs. product FAQ). When you ask a question about remote work, only the policy file should appear in the retrieved sources. This lets you verify that the vector search is actually matching on relevance, not just returning everything.

Build the RAG chain script

Now you will write the Python script that loads your documents, embeds them, and wires up the retrieval chain. You will build it function by function so each piece is testable on its own.

  • Create a new file called rag_chain.py in the root of your rag-service/ folder and paste these imports plus the helper function:
import time

from langchain_community.vectorstores import FAISS
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_ollama import ChatOllama
from langchain_community.document_loaders import DirectoryLoader, TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter


def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

What does this code do?

The imports pull in every component of the RAG pipeline: a vector store (FAISS), an output parser, a prompt template, an embedding model, the local LLM client, a document loader, and a text splitter.

The format_docs helper takes a list of retrieved document chunks and joins them into a single string separated by blank lines. The chain will use this to inject context into the prompt.

  • Below format_docs, add the build_index function that loads, splits, and embeds your documents:
def build_index(docs_path: str = "./docs"):
    # Load all markdown files from the docs folder
    loader = DirectoryLoader(docs_path, glob="**/*.md", loader_cls=TextLoader)
    documents = loader.load()

    # Split documents into overlapping chunks for better retrieval
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=500,
        chunk_overlap=50,
    )
    chunks = text_splitter.split_documents(documents)

    # Embed chunks and store in a FAISS vector index
    embeddings = HuggingFaceEmbeddings(
        model_name="sentence-transformers/all-MiniLM-L6-v2",
    )

    vector_store = FAISS.from_documents(chunks, embeddings)
    return vector_store, embeddings

What does build_index do?

DirectoryLoader finds every .md file in your docs/ folder and loads them as LangChain Document objects.

RecursiveCharacterTextSplitter breaks each document into 500-character chunks with a 50-character overlap. The overlap ensures that sentences split across boundaries still appear in at least one chunk.

HuggingFaceEmbeddings converts each chunk into a 384-dimensional vector using the all-MiniLM-L6-v2 model. FAISS stores these vectors in an in-memory index for fast similarity search.

  • Save rag_chain.py and verify the index builds correctly by running this quick test:
python -c "from rag_chain import build_index; vs, e = build_index(); print(f'Indexed {vs.index.ntotal} chunks')"

The first time you run this, the embedding model downloads (approximately 91 MB). After the download completes, you should see output like Indexed 5 chunks (the exact number depends on how your documents split).

Seeing a ModuleNotFoundError?

Make sure your virtual environment is active. Your terminal prompt should show (.venv) at the beginning. If not, activate it by running source .venv/bin/activate and try again.

help me fix a ModuleNotFoundError when importing from langchain

  • Return to rag_chain.py and add the build_chain function below build_index:
def build_chain(vector_store):
    # Create a retriever that returns the top 3 matching chunks
    retriever = vector_store.as_retriever(search_kwargs={"k": 3})

    # Connect to the local Ollama LLM
    llm = ChatOllama(model="llama3.2", temperature=0)

    # Define the prompt template with context and question placeholders
    prompt = ChatPromptTemplate.from_template(
        """Answer the question based only on the following context. If the context does not contain the answer, say "I don't have enough information to answer that."

Context:
{context}

Question: {question}

Answer:"""
    )

    # Build the LCEL chain: retrieve context, format it, prompt the LLM, parse output
    chain = (
        {"context": retriever | format_docs, "question": RunnablePassthrough()}
        | prompt
        | llm
        | StrOutputParser()
    )

    return chain, retriever

What does build_chain do?

The retriever wraps the FAISS index and returns the 3 most similar chunks for any query.

The prompt template instructs the LLM to answer ONLY from the provided context. This grounding constraint is what makes RAG different from asking the LLM directly.

The LCEL chain uses the pipe operator to connect: retriever feeds formatted context into the prompt, the prompt goes to the LLM, and StrOutputParser extracts the text response.

Test with a hardcoded question

The functions are built, but nothing runs yet. You need a main block that calls the functions and prints the result so you can see the full pipeline in action.

  • At the bottom of rag_chain.py, add this main block that asks a question and prints the answer with sources:
if __name__ == "__main__":
    print("Building index...")
    vector_store, embeddings = build_index()
    chain, retriever = build_chain(vector_store)

    question = "How many days per week can employees work remotely?"

    print(f"\nQuestion: {question}")
    print("Retrieving and generating...\n")

    # Time the retrieval separately to see how fast vector search is
    start = time.perf_counter()
    docs = retriever.invoke(question)
    retrieval_time = time.perf_counter() - start

    # Time the full chain (retrieval + generation)
    start = time.perf_counter()
    answer = chain.invoke(question)
    total_time = time.perf_counter() - start

    print(f"Answer: {answer}")
    print(f"\nSources ({len(docs)} chunks):")
    for doc in docs:
        print(f"  - {doc.page_content[:100]}...")
    print(f"\nRetrieval: {retrieval_time*1000:.0f}ms | Total: {total_time*1000:.0f}ms")

What does the main block do?

It builds the index once, constructs the chain, then asks a question whose answer lives in company_policy.md.

The timing separates vector retrieval from the full chain. You will see that retrieval takes milliseconds while LLM generation takes seconds. This makes it clear where latency lives in your pipeline.

  • Save rag_chain.py.

Before you run the script, predict: will the answer reference the remote work policy content, or will the LLM make something up?

  • Run the full RAG pipeline by executing:
python rag_chain.py

You should see output that includes an answer referencing "3 days per week" (from your policy document), a list of source chunks showing where the answer came from, and timing numbers showing retrieval in milliseconds vs. total time in seconds.

What should I see?

The answer should say something like "employees may work remotely up to 3 days per week." The exact wording varies because the LLM generates natural language, but the key fact (3 days) must come from the document.

The sources section should show chunks from company_policy.md, not from product_faq.md. That confirms the retriever matched on relevance.

Getting a connection error from Ollama?

If you see an error mentioning a refused connection to localhost:11434, Ollama is not running. Check your menu bar for the Ollama icon. If it is missing, open Ollama from your Applications folder and wait for the icon to appear before retrying.

help me fix an Ollama connection error when running my RAG chain

✔️ Awesome, I've got everything!

Great. Double check you have saved rag_chain.py and that your terminal shows a grounded answer with source chunks.

ⓧ I'd like to double check the full code

Your rag_chain.py should contain these sections in order. Scroll up and confirm each is present:

  • Imports (10 import lines) plus the format_docs helper function.
  • The build_index function that loads markdown files, splits them, embeds them, and returns the FAISS vector store.
  • The build_chain function that creates the retriever, LLM, prompt, and LCEL chain.
  • The if __name__ == "__main__": block that runs the question and prints the answer, sources, and timing.

Your RAG pipeline works end-to-end. Next up, you will wrap this chain in a FastAPI service with typed request and response models so other applications can call it over HTTP.

Wrap the Chain in a FastAPI Service

Your RAG chain works end-to-end as a Python script. Other applications cannot call a script directly.

In this step, you will wrap the chain in a FastAPI service with typed Pydantic request and response models. This turns your chain into an HTTP endpoint that any application can call.

In this step, get ready to:
  • Define Pydantic request and response models.
  • Create a FastAPI app with lifespan-managed startup.
  • Test the live API endpoints.
Define Pydantic models

Pydantic models define the exact shape of data going into and out of your API. FastAPI uses these models to validate incoming requests and serialize outgoing responses automatically.

  • Create a new file called models.py in your rag-service/ folder by pasting the following code:
from pydantic import BaseModel


class AskRequest(BaseModel):
    # The user's natural language question
    question: str


class AskResponse(BaseModel):
    # The generated answer from the RAG chain
    answer: str
    # Source chunks used to ground the answer
    sources: list[str]
    # Time spent on vector retrieval
    retrieval_ms: float
    # Time spent on LLM generation
    generation_ms: float
    # Total request processing time
    total_ms: float


class HealthResponse(BaseModel):
    status: str
    model: str
    # Number of vectors currently in the FAISS index
    index_size: int

What do these models do?

AskRequest defines the input contract: a single question string. Any POST body missing this field gets rejected with a 422 error before your chain runs.

AskResponse defines the output: the answer, source chunks for verification, and three latency measurements. HealthResponse lets callers confirm the service is ready before sending questions.

  • Save models.py.
  • Confirm the file exists by running:
ls models.py

You should see models.py in the output.

Not seeing models.py?

Make sure you are in the rag-service/ directory. Run pwd to check your current location.

help me find where I saved my models.py file

Create the FastAPI app with lifespan startup

Building the FAISS index takes several seconds. You only want to do it once when the server starts, not on every request. FastAPI's lifespan context manager runs setup code at startup and cleanup code at shutdown.

  • Create a new file called main.py and add the imports, module-level globals, and lifespan function:
import logging
import time
import uuid
from contextlib import asynccontextmanager

from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse

from models import AskRequest, AskResponse, HealthResponse
from rag_chain import build_index, build_chain, format_docs

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("rag-service")

vector_store = None
chain = None
retriever = None
embeddings = None

# Runs once at server startup, builds index and chain
@asynccontextmanager
async def lifespan(app: FastAPI):
    global vector_store, chain, retriever, embeddings
    logger.info("Loading embedding model and building FAISS index...")
    vector_store, embeddings = build_index()
    chain, retriever = build_chain(vector_store)
    logger.info(
        f"Index ready with {vector_store.index.ntotal} vectors"
    )
    yield
    logger.info("Shutting down RAG service")

What does this code do?

The module-level variables (vector_store, chain, retriever, embeddings) start as None. The lifespan function populates them once at startup by calling your existing build_index() and build_chain() functions.

Everything before yield runs at startup. Everything after yield runs at shutdown. This avoids rebuilding the index on every request.

  • Below the lifespan function in main.py, add the app instantiation and the /health endpoint:
app = FastAPI(
    title="RAG Microservice",
    description="Question answering over documents using RAG",
    version="0.1.0",
    lifespan=lifespan,
)


# Quick check that the service and model are ready
@app.get("/health", response_model=HealthResponse)
async def health():
    return HealthResponse(
        status="healthy",
        model="llama3.2",
        index_size=vector_store.index.ntotal,
    )

What does this code do?

The FastAPI() call creates the app and passes lifespan=lifespan so the index builds at startup. The title, description, and version appear in auto-generated API docs.

The /health endpoint returns the current index size so callers can confirm the service is loaded before sending questions.

  • Save main.py.
  • Start the development server by running:
fastapi dev main.py

What should I see?

The terminal shows the embedding model loading, then a log line saying the index is ready with several vectors. FastAPI prints the server URL (http://127.0.0.1:8000). The server stays running and watches for file changes.

  • Open a second terminal tab, activate your virtual environment, and test the health endpoint by running:
source .venv/bin/activate
curl http://localhost:8000/health

You should see a JSON response with "status": "healthy" and an index_size greater than zero.

Seeing connection refused?

Make sure the first terminal is still running fastapi dev main.py. If you see an import error, confirm your virtual environment is active in the first terminal (you should see (.venv) in the prompt).

help me debug why my FastAPI server won't start

Now add the main endpoint that answers questions. The server will auto-reload when you save.

  • Switch back to main.py and add the /ask endpoint below the /health function:
@app.post("/ask", response_model=AskResponse)
async def ask(request: AskRequest):
    total_start = time.perf_counter()

    # Measure vector retrieval separately
    retrieval_start = time.perf_counter()
    docs = retriever.invoke(request.question)
    retrieval_ms = (time.perf_counter() - retrieval_start) * 1000

    # Measure LLM generation separately
    generation_start = time.perf_counter()
    answer = chain.invoke(request.question)
    generation_ms = (time.perf_counter() - generation_start) * 1000

    total_ms = (time.perf_counter() - total_start) * 1000

    # Return first 200 chars of each source chunk
    sources = [doc.page_content[:200] for doc in docs]

    return AskResponse(
        answer=answer,
        sources=sources,
        retrieval_ms=round(retrieval_ms, 1),
        generation_ms=round(generation_ms, 1),
        total_ms=round(total_ms, 1),
    )

What does this code do?

The endpoint times retrieval and generation separately using time.perf_counter(). This shows you exactly how much time the vector search takes versus the LLM call.

The sources field returns the first 200 characters of each retrieved chunk. This lets callers verify the answer is grounded in real document content.

  • Save main.py.

Your first terminal should show the server detecting the change and reloading automatically.

✔️ Awesome, I've got everything!

Great. Double check you have saved both models.py and main.py.

ⓧ I'd like to double check the full code

Here is the complete models.py:

from pydantic import BaseModel


class AskRequest(BaseModel):
    question: str


class AskResponse(BaseModel):
    answer: str
    sources: list[str]
    retrieval_ms: float
    generation_ms: float
    total_ms: float


class HealthResponse(BaseModel):
    status: str
    model: str
    index_size: int

For main.py, confirm these sections appear in order by scrolling through your file:

  • Imports and module-level globals (vector_store, chain, retriever, embeddings all set to None).
  • The lifespan async context manager that calls build_index() and build_chain().
  • The app = FastAPI(...) instantiation with lifespan=lifespan.
  • The GET /health endpoint returning HealthResponse.
  • The POST /ask endpoint with separate retrieval and generation timing.
Test the running service

Before you send a question, think about what you expect to see. The endpoint should return a JSON object with five fields: the answer, source chunks, and three latency numbers.

  • In your second terminal, test the /ask endpoint by running:
curl -X POST http://localhost:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"question": "What is AcmeCRM?"}'

You should see a JSON response containing an answer field that references AcmeCRM being a customer relationship management platform, a sources array with retrieved chunks, and numeric values for retrieval_ms, generation_ms, and total_ms.

Why is generation_ms so much larger than retrieval_ms?

Vector retrieval with FAISS runs in milliseconds because it is a fast in-memory lookup. LLM generation with Ollama takes seconds because the model generates tokens one at a time. This latency breakdown shows you exactly where the bottleneck is.

  • Open the auto-generated API documentation by visiting http://localhost:8000/docs in your browser.

You should see an interactive Swagger UI listing both endpoints with their request and response schemas. FastAPI generated this from your Pydantic models automatically.

Getting an empty answer or error?

If the response has an empty answer, make sure Ollama is still running. Check with ollama list in a third terminal. If you see a 500 error, check the server terminal for the traceback.

help me debug my /ask endpoint response

Your RAG pipeline is now a live HTTP service. Next up, you will add structured logging middleware so you can see exactly what happens on every request.

Add Logging Middleware

Your /ask endpoint already returns timing breakdown in the response body. That helps callers understand performance.

But response bodies are for callers. Operators need structured logs that record every request automatically. Without them, you cannot search for slow requests, correlate errors across services, or pipe metrics to a log aggregator. In this step, you will add HTTP middleware that logs every single request as a structured dict with a unique ID.

In this step, get ready to:
  • Add HTTP middleware that logs every request as a structured JSON dict.
  • Verify structured log output appears in your terminal on each request.
Add the HTTP logging middleware

In FastAPI, middleware wraps every request. It runs code before the route handler processes the request and after the response is ready. This makes it the right place for cross-cutting concerns like logging and request tracking.

  • In main.py, find the closing parenthesis of the app = FastAPI(...) block.
  • Add the following middleware function below it, before the @app.get("/health") route:
@app.middleware("http")
async def logging_middleware(request: Request, call_next):
    # Generate a short unique ID for this request
    request_id = str(uuid.uuid4())[:8]
    request.state.request_id = request_id
    start = time.perf_counter()

    # Pass the request to the actual route handler
    response = await call_next(request)

    # Calculate how long the full request took
    duration_ms = (time.perf_counter() - start) * 1000
    log_data = {
        "request_id": request_id,
        "method": request.method,
        "path": request.url.path,
        "status": response.status_code,
        "duration_ms": round(duration_ms, 1),
    }
    logger.info(log_data)
    # Return the request ID so callers can reference it
    response.headers["X-Request-ID"] = request_id
    return response

What does this code do?

  • request_id = str(uuid.uuid4())[:8] generates a short unique ID for every incoming request.
  • request.state.request_id attaches the ID to the request object so route handlers can access it later.
  • await call_next(request) passes the request to the actual route handler and waits for the response to come back.
  • The log_data dict captures method, path, status code, and total duration in milliseconds as a structured log entry.
  • response.headers["X-Request-ID"] returns the ID to the caller so they can reference it in support requests or debugging.
  • Save main.py.
  • Start the dev server by running:
fastapi dev main.py
  • Open a second terminal tab (keep the server running in the first one).
  • Hit the health endpoint by running:
curl http://localhost:8000/health

Switch back to the terminal running your server. You should see a log line like:

INFO:rag-service:{'request_id': 'a1b2c3d4', 'method': 'GET', 'path': '/health', 'status': 200, 'duration_ms': 1.2}

Not seeing the log line?

  • Make sure you saved main.py before starting the server. The dev server auto-reloads on save, but only if it was already running.
  • Check that the middleware function sits between app = FastAPI(...) and your route definitions. If it is below the routes, it may not intercept requests correctly.
  • Confirm you are looking at the server terminal, not the curl terminal.

Why isn't my middleware logging anything?

Test observability with a real question

The /health endpoint returned in about 1 millisecond. Before you test /ask, think about it. How long do you think a request that runs vector retrieval AND LLM generation will take?

  • Make a request to the /ask endpoint by running:
curl -X POST http://localhost:8000/ask -H "Content-Type: application/json" -d '{"question": "How many days per week can employees work remotely?"}'

Switch back to the server terminal. You should see a new log line with a much larger duration_ms value compared to the health check.

What should I see?

The server terminal shows a structured dict with method: POST, path: /ask, and a duration_ms that is likely in the thousands (several seconds for LLM generation on a local model).

The JSON response body from curl includes retrieval_ms and generation_ms so you can see exactly where time is spent. Retrieval is typically fast (under 100ms). Generation dominates the total.

  • Check the curl output for the X-Request-ID header by running:
curl -v -X POST http://localhost:8000/ask -H "Content-Type: application/json" -d '{"question": "What is AcmeCRM?"}'

In the verbose output, you should see < x-request-id: followed by an 8-character ID that matches the server log line.

Why return the request ID to the caller?

In production, if a caller reports a problem, they can include the X-Request-ID from the response. You then search your logs for that exact ID to find everything that happened during that request. This is how teams correlate client-side errors with server-side logs.

✔️ Awesome, I've got everything!

Great. Confirm you have saved main.py and that your server terminal is producing structured log lines on every request.

ⓧ I'd like to double check the full code

Your main.py should contain these sections in order:

  • Imports (logging, time, uuid, asynccontextmanager, FastAPI, Request, JSONResponse, and your model/chain imports).
  • Logger setup with logging.basicConfig(level=logging.INFO) and logger = logging.getLogger("rag-service").
  • Global variables (vector_store, chain, retriever, embeddings).
  • The lifespan async context manager that builds the FAISS index at startup.
  • The app = FastAPI(...) declaration.
  • The @app.middleware("http") logging middleware you just added.
  • The GET /health endpoint.
  • The POST /ask endpoint with retrieval_ms and generation_ms timing.

Scroll up and confirm each section is present in this order. The middleware must sit between the app = FastAPI(...) block and the route definitions.

Your RAG microservice now has structured observability. Every request is logged with a unique ID, method, path, status, and timing. You can search these logs, pipe them to an aggregator, and correlate caller issues with server-side events. The service is complete and ready for its next challenge.

Secret mission

Add a Live Document Ingestion Endpoint

Your RAG service can only answer questions about documents that existed when the server started. What if you could feed it new knowledge on the fly? Build a /ingest endpoint that accepts raw text, chunks and embeds it, and merges it into the live FAISS index without a restart.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Ollama llama3.2 model files stored in ~/.ollama/models (approximately 2 GB).
  • HuggingFace all-MiniLM-L6-v2 embedding model cached in ~/.cache/huggingface/ (approximately 90 MB).
  • Project directory rag-service/ on your Desktop (includes virtual environment and source files).

Keep everything running

No action needed. Choose this if you plan to continue into Phase 2 of this project series.

  • Ollama continues running as a background service via the menu bar icon.
  • The llama3.2 model stays pulled and ready for instant use.
  • Your rag-service/ project folder stays on your Desktop with the virtual environment intact.

Pause - I'll come back to this later

Shut down running processes to free up memory. Your files and models stay on disk so you can resume later.

  • Stop the FastAPI server by pressing Ctrl+C in the terminal where it is running.
  • Quit the Ollama app by clicking the Ollama icon in the menu bar and selecting Quit Ollama. The model files remain on disk.
  • When you return, relaunch Ollama from Applications and reactivate your virtual environment with source .venv/bin/activate inside the rag-service/ folder.

Delete - I don't want to use this again

Remove all project resources and start fresh. This frees approximately 2 GB of disk space.

  • Remove the llama3.2 model from Ollama by running:
  • Remove the project directory by running:
  • Optionally, remove the cached HuggingFace embedding model by running:
ollama rm llama3.2

Next, delete the project folder.

rm -rf ~/Desktop/rag-service

On Windows, use:

rmdir /s /q "$env:USERPROFILE\Desktop\rag-service"

Finally, remove the cached embedding model if you no longer need it for other projects.

rm -rf ~/.cache/huggingface/hub/models--sentence-transformers--all-MiniLM-L6-v2

What about Ollama itself?

These steps remove the model and project files but leave the Ollama application installed. If you want to use Ollama for other projects in the future, keep it installed. To fully remove it, drag Ollama from your Applications folder to the Trash.

Nice Work!

Nice Work!

You just built a fully local RAG-powered question-answering API from scratch. Your service loads documents, chunks and embeds them into a vector store, retrieves the most relevant context for any question, and generates grounded answers with source citations and latency metrics.

You've learned how to:

  • Build a complete RAG pipeline connecting document loading, chunking, embedding, FAISS vector search, and local LLM generation into a single composable chain.
  • Create a FastAPI microservice with typed Pydantic request/response models, auto-generated OpenAPI docs, and a lifespan context manager for one-time startup initialization.
  • Add structured JSON logging middleware with per-request latency breakdown showing retrieval time vs. generation time, plus request IDs for correlation.
  • Secret Mission: Added a live /ingest endpoint that accepts new documents on the fly and merges them into the FAISS index under a threading lock, without restarting the server.

Ready to quiz yourself?