Upgrade Your RAG API with Hybrid Search

Introduction

30 Second Summary

When you search for something online, the best results often combine different search strategies behind the scenes. Some matches come from exact keywords, while others come from understanding the meaning of your question.

In this project, you will upgrade your existing Document Q&A RAG API with a production-grade retrieval pipeline that combines semantic search with keyword search, then uses a reranking model to precisely order results by relevance.

What You'll Build

You will add a hybrid search and cross-encoder reranking layer to your existing FastAPI + ChromaDB + Ollama RAG API, then build a comparison endpoint that proves the quality difference.

By the end of this project, you'll have:

• A hybrid retriever module that runs BM25 keyword search alongside vector similarity search and merges them using Reciprocal Rank Fusion.
• A cross-encoder reranking layer that scores every retrieved chunk against the query for precise relevance ordering.
• An upgraded /ask endpoint and a new /compare endpoint that shows side-by-side results from vector-only vs. hybrid+reranked retrieval.
• Secret Mission: Add retrieval strategy benchmarking with timing metrics and a /benchmark endpoint.

Are there any prerequisites?

This project builds on top of an existing RAG API project with FastAPI, Ollama, and ChromaDB. You should have that project working locally before starting.

You will also need Python 3.10+ installed and familiarity with Python virtual environments and REST APIs.

Install New Dependencies and Verify Setup

Your existing RAG API retrieves context using vector similarity search. This works for broad semantic queries, but it struggles with exact keyword matches like technical terms, error codes, and proper names.

In this step, you'll install two new packages that add BM25 keyword search and cross-encoder reranking to your retrieval pipeline. You'll also confirm your existing project is ready for the upgrade.

In this step, get ready to:

• Install rank_bm25 and sentence-transformers into your virtual environment.
• Download the cross-encoder reranking model from Hugging Face.
• Confirm your existing FastAPI server and endpoints still work.

Install rank_bm25 and sentence-transformers

• Install both packages into your virtual environment by running this command:

pip install rank_bm25==0.2.2 sentence-transformers==5.4.1

What do these packages do?

rank_bm25 provides the BM25Okapi algorithm for keyword-based document ranking. It scores how relevant a document is based on exact word matches.

sentence-transformers includes the CrossEncoder class for reranking. Cross-encoders score a query-passage pair together for more precise relevance judgments than comparing pre-computed embeddings.

Large download ahead

The sentence-transformers package may download additional dependencies like PyTorch. This can take a few minutes depending on your connection speed.

• Verify both packages installed correctly by running this command:

python -c "from rank_bm25 import BM25Okapi; from sentence_transformers import CrossEncoder; print('All imports OK')"

You should see All imports OK printed in your terminal.

Seeing an ImportError?

• Make sure your virtual environment is activated. You should see your environment name in parentheses at the start of your terminal prompt.
• Run pip list to check if rank-bm25 and sentence-transformers appear in the list.
• Help me debug this ImportError.

Download the cross-encoder model

The cross-encoder model needs to be downloaded from Hugging Face before you can use it. Pre-downloading it now avoids a surprise delay when you test your retrieval pipeline later.

• Download and cache the model locally by running this command:

python -c "from sentence_transformers import CrossEncoder; CrossEncoder('cross-encoder/ms-marco-MiniLM-L6-v2'); print('Model downloaded')"

What is this model?

cross-encoder/ms-marco-MiniLM-L6-v2 is a 22.7 million parameter model trained on the MS MARCO passage ranking dataset. It takes a query and a passage as input, then outputs a single relevance score.

This makes it ideal for reranking a small set of retrieval candidates. It is the go-to reranker for RAG pipelines, balancing accuracy with speed.

The download is about 80MB. Once it finishes, you should see Model downloaded in your terminal.

Download stalled or failed?

• Check your internet connection. The model downloads from Hugging Face's servers.
• If you see a timeout error, try running the command again. Hugging Face caches partial downloads so it will resume where it left off.
• Help me troubleshoot this model download.

Confirm your existing project runs

Before building new features, confirm your existing API still works. You'll upload a test document so you have data in ChromaDB to test against throughout this project.

• Check that Ollama is running by opening a new terminal window and running this command:

ollama list

You should see a list of your downloaded models, including the model your project uses.

Seeing a connection error?

This means Ollama is not running yet.

• On macOS, open the Ollama app from your Applications folder or use Spotlight search.
• On Linux, start the Ollama service by running ollama serve in a separate terminal window.
• Help me get Ollama running.

• Start your FastAPI server from your project directory by running this command:

fastapi dev main.py

You should see output indicating the server is running on http://127.0.0.1:8000.

• Open a new terminal window and upload a test PDF so you have documents to query throughout this project:

curl -X POST "http://127.0.0.1:8000/upload?file_path=[[PDF_PATH=\"path/to/your/test.pdf\"]]"

You should see a JSON response showing how many chunks were processed and stored.

• Test that retrieval still works by querying the /ask endpoint:

curl -X POST http://127.0.0.1:8000/ask -H "Content-Type: application/json" -d '{"question": "What is this document about?"}'

You should see a JSON response containing an answer generated from your uploaded document.

Server not starting or endpoints failing?

• Make sure you are in your project directory and your virtual environment is activated before running fastapi dev main.py.
• If you see a port conflict error, another process is already using port 8000. Stop the other process or add --port 8001 to the command.
• If the /ask endpoint returns an error, make sure Ollama is running and you have uploaded at least one document first.
• Help me debug my FastAPI server.

Your environment is set up with all the packages you need. Next up, you'll add new methods to your vector store that expose raw chunks for keyword search.

Expose Raw Chunks from Your Vector Store

Your dependencies are installed and the cross-encoder model is cached locally. Now it's time to prepare your vector store for hybrid search.

The hybrid retriever you'll build in the next step needs two things from your vector store that it doesn't currently expose. It needs all raw text chunks so BM25 can build a keyword index. It also needs distance scores alongside query results for the comparison endpoint.

In this step, get ready to:

• Add a get_all_chunks() method that returns all stored chunks for BM25 indexing.
• Add a query_vectors_with_scores() method that returns query results with distance scores.

Add a get_all_chunks() method to vector_store.py

BM25 builds its keyword index from every document in your collection. Your current vector store only returns query results, with no way to hand over all the raw chunks at once.

• In vector_store.py, add this method to your VectorStore class below your existing methods:

def get_all_chunks(self):
    """Return all stored chunks with their IDs for BM25 indexing."""
    results = self.collection.get()
    return {
        "ids": results["ids"],
        "documents": results["documents"],
    }

What does this code do?

• self.collection.get() retrieves every document stored in your ChromaDB collection. No query is needed.
• The method returns only the ids and documents keys. BM25 will use the text to build its keyword index, and the IDs to track which chunks score highest.

You won't see a change yet. The next method completes the vector store updates needed for hybrid search.

Add a query_vectors_with_scores() method to vector_store.py

The comparison endpoint you'll build later needs to show how close each chunk is to the query. ChromaDB can return distance scores alongside results, but your existing query method doesn't request them.

• In vector_store.py, add this method to your VectorStore class below the get_all_chunks() method you just added:

def query_vectors_with_scores(self, query_text, top_k=10):
    """Query ChromaDB and return chunks with distance scores."""
    results = self.collection.query(
        query_texts=[query_text],
        n_results=top_k,
        include=["documents", "distances"],
    )
    return {
        "ids": results["ids"][0],
        "documents": results["documents"][0],
        "distances": results["distances"][0],
    }

What does this code do?

• include=["documents", "distances"] tells ChromaDB to return L2 distance scores alongside the matched text. Lower distances mean higher similarity.
• ChromaDB nests query results in lists (one list per query). Since you're sending a single query, [0] unpacks the outer list to give you flat lists of IDs, documents, and distances.

• Save vector_store.py.
• Check the terminal where your FastAPI server is running. If the server reloads without any error messages, your new methods are syntactically correct.

Seeing an error when you save?

• Make sure both methods are indented inside your VectorStore class. They should be at the same indentation level as your existing add_chunks() and query() methods.
• Check that each method has self as its first parameter.
• If you see a SyntaxError, check for mismatched quotes or brackets in the return dictionaries.

My vector_store.py is showing an error after adding the new methods

✔️ Awesome, I've got everything!

Double check you've saved vector_store.py and your server is running without errors.

ⓧ I'd like to double check the full code

Here are the two new methods you should have added to your VectorStore class. The rest of your file stays the same.

# Add these methods to your existing VectorStore class

def get_all_chunks(self):
    """Return all stored chunks with their IDs for BM25 indexing."""
    results = self.collection.get()
    return {
        "ids": results["ids"],
        "documents": results["documents"],
    }

def query_vectors_with_scores(self, query_text, top_k=10):
    """Query ChromaDB and return chunks with distance scores."""
    results = self.collection.query(
        query_texts=[query_text],
        n_results=top_k,
        include=["documents", "distances"],
    )
    return {
        "ids": results["ids"][0],
        "documents": results["documents"][0],
        "distances": results["distances"][0],
    }

Your vector store now exposes everything the hybrid retriever needs. Next up, you'll build the hybrid retriever module that combines BM25 keyword search with vector search and reranks the results.

Build the Hybrid Retriever Module

Your vector store now exposes raw chunks and distance scores. The foundation for a smarter retrieval pipeline is set.

In this step, you will build the hybrid retriever module. This single file implements the entire multi-stage pipeline: keyword search with BM25, result merging with Reciprocal Rank Fusion, and precise relevance scoring with a cross-encoder reranker.

In this step, get ready to:

• Build BM25 keyword search and vector search methods.
• Implement Reciprocal Rank Fusion to merge results from both strategies.
• Add cross-encoder reranking to precisely score query-document relevance.

Create hybrid_retriever.py with BM25 keyword search

Vector search finds semantically similar chunks, but it can miss exact keyword matches. Technical terms, error codes, and proper names often get blurred by embedding models.

BM25 is a keyword-ranking algorithm that scores documents based on exact term frequency. By adding BM25 alongside vector search, your retriever covers both angles.

• Create a new file called hybrid_retriever.py in the same folder as your vector_store.py.
• Add the imports, class definition, and BM25 search method by pasting this code into the file:

import numpy as np
from rank_bm25 import BM25Okapi
from sentence_transformers import CrossEncoder


class HybridRetriever:
    def __init__(self, vector_store):
        # Store reference to existing vector store for search queries
        self.vector_store = vector_store
        # Load cross-encoder model once (reused across all requests)
        self.reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")

    def _bm25_search(self, query, top_k=10):
        # Fetch all stored chunks from ChromaDB for keyword indexing
        all_chunks = self.vector_store.get_all_chunks()
        if not all_chunks["documents"]:
            return []

        documents = all_chunks["documents"]
        ids = all_chunks["ids"]

        # Tokenize each chunk: lowercase and split on whitespace
        tokenized_corpus = [doc.lower().split() for doc in documents]
        bm25 = BM25Okapi(tokenized_corpus)

        # Score every chunk against the query using BM25
        tokenized_query = query.lower().split()
        scores = bm25.get_scores(tokenized_query)

        # Pair each chunk with its score and sort by relevance
        scored_docs = list(zip(ids, documents, scores))
        scored_docs.sort(key=lambda x: x[2], reverse=True)

        return [
            {"id": doc_id, "text": text, "bm25_score": float(score)}
            for doc_id, text, score in scored_docs[:top_k]
        ]

What does this code do?

The HybridRetriever class wraps your existing vector store. It does not replace it. This is the composable pipeline pattern used in production retrieval systems.

• __init__ stores a reference to your vector store and loads the cross-encoder model once. Loading it here means it stays in memory across requests instead of reloading each time.
• _bm25_search() pulls every chunk from ChromaDB, tokenizes them into lowercase words, builds a fresh BM25 index, and scores all chunks against your query. The top-k results come back sorted by BM25 score.

Why rebuild the BM25 index every time?

Rebuilding the index on each query keeps the code simple and guarantees freshness. If you upload new documents between queries, the index automatically includes them. For a production system with thousands of documents, you would cache the index and invalidate it when documents change.

• Save hybrid_retriever.py.
• Confirm the file exists by running:

ls hybrid_retriever.py

• You should see hybrid_retriever.py listed in the output.

Don't see the file?

Make sure you created the file in the same directory as vector_store.py. Run pwd to check your current directory.

Help me find where my hybrid_retriever.py file was created

Now add the vector search wrapper method. This gives the retriever a consistent interface for both search strategies.

• In hybrid_retriever.py, add this method inside the HybridRetriever class, below the _bm25_search() method:

    def _vector_search(self, query, top_k=10):
        # Use ChromaDB's embedding-based search with distance scores
        results = self.vector_store.query_vectors_with_scores(query, top_k=top_k)
        return [
            {"id": doc_id, "text": text, "distance": float(dist)}
            for doc_id, text, dist in zip(
                results["ids"], results["documents"], results["distances"]
            )
        ]

What does this method do?

This is a thin wrapper around your vector store's query_vectors_with_scores() method. It reformats the results into the same dict structure that _bm25_search() uses. Both methods return a list of dicts with an id and text key. This consistent shape is what makes the fusion step straightforward.

• Save hybrid_retriever.py.
• Verify the module loads without errors by running:

python -c "from hybrid_retriever import HybridRetriever; print('Module loads OK')"

• You should see Module loads OK printed in your terminal. This confirms all imports resolve and there are no syntax errors.

Seeing an ImportError?

If you see ModuleNotFoundError: No module named 'rank_bm25', make sure your virtual environment is activated. Run pip list and confirm rank_bm25 and sentence-transformers appear in the output.

Help me fix this import error in hybrid_retriever.py

Implement Reciprocal Rank Fusion

You now have two ranked lists of chunks for any query. One from vector search. One from BM25. Reciprocal Rank Fusion merges them into a single list by assigning each result a score of 1 / (k + rank).

Chunks appearing in both lists get their scores summed, which naturally floats them to the top. The constant k=60 is the standard default from the original RRF paper, also used by Elasticsearch and Weaviate.

• In hybrid_retriever.py, add this method inside the HybridRetriever class, below the _vector_search() method:

    def _reciprocal_rank_fusion(self, vector_results, bm25_results, k=60):
        rrf_scores = {}
        chunk_text = {}

        # Score each vector result: 1/(k + rank)
        for rank, result in enumerate(vector_results, start=1):
            doc_id = result["id"]
            rrf_scores[doc_id] = rrf_scores.get(doc_id, 0) + 1 / (k + rank)
            chunk_text[doc_id] = result["text"]

        # Score each BM25 result the same way (duplicates get summed)
        for rank, result in enumerate(bm25_results, start=1):
            doc_id = result["id"]
            rrf_scores[doc_id] = rrf_scores.get(doc_id, 0) + 1 / (k + rank)
            chunk_text[doc_id] = result["text"]

        # Build the merged list sorted by combined RRF score
        fused = [
            {"id": doc_id, "text": chunk_text[doc_id], "rrf_score": score}
            for doc_id, score in rrf_scores.items()
        ]
        fused.sort(key=lambda x: x["rrf_score"], reverse=True)
        return fused

What does this code do?

• rrf_scores is a dictionary keyed by chunk ID. Each chunk accumulates a score based on its position in the vector and BM25 result lists.
• A chunk ranked #1 in a list gets a score of 1/(60+1) = 0.0164. A chunk ranked #10 gets 1/(60+10) = 0.0143. The high k value dampens the gap between ranks so lower-ranked results still contribute.
• If a chunk appears in both lists, its scores are summed. This is the key advantage of RRF. Chunks that both strategies agree on naturally rise to the top.

• Save hybrid_retriever.py.

You will not see a change yet. The next substep adds the reranking layer that uses these fused results.

Add cross-encoder reranking and the retrieve pipeline

The fused list has good coverage, but the ordering is still approximate. A cross-encoder model takes each (query, chunk) pair and produces a precise relevance score.

This is more accurate than comparing pre-computed embeddings because it processes the query and passage together. The tradeoff is speed, but reranking a small candidate set of 10-20 chunks is fast enough for real-time use.

• In hybrid_retriever.py, add the rerank method below _reciprocal_rank_fusion():

    def _rerank(self, query, chunks, top_k=5):
        if not chunks:
            return []

        # Create (query, chunk_text) pairs for the cross-encoder
        pairs = [(query, chunk["text"]) for chunk in chunks]
        # Score each pair for relevance (higher = more relevant)
        scores = self.reranker.predict(pairs)

        for chunk, score in zip(chunks, scores):
            chunk["reranker_score"] = float(score)

        # Sort by reranker score and return the top results
        chunks.sort(key=lambda x: x["reranker_score"], reverse=True)
        return chunks[:top_k]

What does this code do?

• self.reranker.predict(pairs) sends every (query, chunk) pair through the ms-marco-MiniLM-L6-v2 cross-encoder. It returns a numpy array of relevance scores.
• Higher scores mean more relevant. The method attaches each score as a reranker_score field on the chunk dict, sorts descending, and returns the top-k results.

• Save hybrid_retriever.py.

You will not see a change yet. The next chunk ties all four stages together into one public method.

The final piece is a public retrieve() method that orchestrates the full pipeline, plus a vector_only() convenience method for baseline comparisons.

• In hybrid_retriever.py, add these two methods below _rerank():

    def retrieve(self, query, top_k=3):
        # Stage 1 & 2: Run both search strategies
        vector_results = self._vector_search(query, top_k=10)
        bm25_results = self._bm25_search(query, top_k=10)
        # Stage 3: Merge results using Reciprocal Rank Fusion
        fused = self._reciprocal_rank_fusion(vector_results, bm25_results)
        # Stage 4: Rerank the fused results with the cross-encoder
        reranked = self._rerank(query, fused, top_k=top_k)
        return reranked

    def vector_only(self, query, top_k=5):
        # Convenience method for baseline comparison (no fusion or reranking)
        return self._vector_search(query, top_k=top_k)

What does this code do?

• retrieve() is the method your API will call. It runs all four stages in sequence: vector search, BM25 search, RRF fusion, and cross-encoder reranking. Each stage narrows and refines the results.
• vector_only() skips fusion and reranking entirely. You will use this in the comparison endpoint to show the difference between basic and hybrid retrieval.
• Both search methods fetch 10 candidates each. RRF merges them into up to 20 unique chunks. The reranker scores all of them and returns the top 3. This broad-then-narrow pattern is how production retrieval systems balance recall and precision.

• Save hybrid_retriever.py.
• Verify the complete module loads and the cross-encoder initializes by running:

python -c "from hybrid_retriever import HybridRetriever; print('Full module OK')"

• You should see Full module OK in your terminal. This confirms every method is syntactically valid and the cross-encoder model loads from cache.

Seeing a SyntaxError or IndentationError?

Every method in this file must be indented one level inside the HybridRetriever class (4 spaces). Check that def _rerank, def retrieve, and def vector_only are at the same indentation level as def __init__.

Use the double-check tab below to compare your full file against the reference.

Help me fix this syntax error in hybrid_retriever.py

✔️ Awesome, I've got everything!

Double check you have saved hybrid_retriever.py and the import test printed Full module OK.

ⓧ I'd like to double check the full code

Here is the complete hybrid_retriever.py file. Compare it against yours and fix any differences.

import numpy as np
from rank_bm25 import BM25Okapi
from sentence_transformers import CrossEncoder


class HybridRetriever:
    def __init__(self, vector_store):
        self.vector_store = vector_store
        self.reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")

    def _bm25_search(self, query, top_k=10):
        all_chunks = self.vector_store.get_all_chunks()
        if not all_chunks["documents"]:
            return []

        documents = all_chunks["documents"]
        ids = all_chunks["ids"]

        tokenized_corpus = [doc.lower().split() for doc in documents]
        bm25 = BM25Okapi(tokenized_corpus)

        tokenized_query = query.lower().split()
        scores = bm25.get_scores(tokenized_query)

        scored_docs = list(zip(ids, documents, scores))
        scored_docs.sort(key=lambda x: x[2], reverse=True)

        return [
            {"id": doc_id, "text": text, "bm25_score": float(score)}
            for doc_id, text, score in scored_docs[:top_k]
        ]

    def _vector_search(self, query, top_k=10):
        results = self.vector_store.query_vectors_with_scores(query, top_k=top_k)
        return [
            {"id": doc_id, "text": text, "distance": float(dist)}
            for doc_id, text, dist in zip(
                results["ids"], results["documents"], results["distances"]
            )
        ]

    def _reciprocal_rank_fusion(self, vector_results, bm25_results, k=60):
        rrf_scores = {}
        chunk_text = {}

        for rank, result in enumerate(vector_results, start=1):
            doc_id = result["id"]
            rrf_scores[doc_id] = rrf_scores.get(doc_id, 0) + 1 / (k + rank)
            chunk_text[doc_id] = result["text"]

        for rank, result in enumerate(bm25_results, start=1):
            doc_id = result["id"]
            rrf_scores[doc_id] = rrf_scores.get(doc_id, 0) + 1 / (k + rank)
            chunk_text[doc_id] = result["text"]

        fused = [
            {"id": doc_id, "text": chunk_text[doc_id], "rrf_score": score}
            for doc_id, score in rrf_scores.items()
        ]
        fused.sort(key=lambda x: x["rrf_score"], reverse=True)
        return fused

    def _rerank(self, query, chunks, top_k=5):
        if not chunks:
            return []

        pairs = [(query, chunk["text"]) for chunk in chunks]
        scores = self.reranker.predict(pairs)

        for chunk, score in zip(chunks, scores):
            chunk["reranker_score"] = float(score)

        chunks.sort(key=lambda x: x["reranker_score"], reverse=True)
        return chunks[:top_k]

    def retrieve(self, query, top_k=3):
        vector_results = self._vector_search(query, top_k=10)
        bm25_results = self._bm25_search(query, top_k=10)
        fused = self._reciprocal_rank_fusion(vector_results, bm25_results)
        reranked = self._rerank(query, fused, top_k=top_k)
        return reranked

    def vector_only(self, query, top_k=5):
        return self._vector_search(query, top_k=top_k)

Your hybrid retriever module is complete. It handles every stage of the pipeline, from broad candidate search to precise relevance scoring. Next up, you will wire it into your API and see how it outperforms basic vector search on real queries.

Wire the Hybrid Retriever into Your API

Your hybrid retriever module is built and ready to go. It can run vector search, BM25 keyword search, fuse the results with Reciprocal Rank Fusion, and rerank them with a cross-encoder. The problem is that none of your API endpoints know it exists yet.

In this step, you will wire the HybridRetriever into your FastAPI server so every question flows through the full retrieval pipeline. You will also add a new /compare endpoint that shows side-by-side results from vector-only vs. hybrid+reranked retrieval.

In this step, get ready to:

• Update your /ask endpoint to use the hybrid retriever instead of direct vector search.
• Add a /compare endpoint that runs both retrieval strategies and shows the difference.
• Test with targeted queries that expose when hybrid search outperforms vector-only.

Update /ask to use the hybrid retriever

Right now, your /ask endpoint queries ChromaDB directly for vector matches. You need to swap that out so every question goes through the full hybrid+rerank pipeline instead.

• Open main.py and find your existing import lines at the top of the file.
• Add this import below your other local imports:

from hybrid_retriever import HybridRetriever

• Find the line where you create your FastAPI app and update the title:

app = FastAPI(title="Document Q&A API with Hybrid Search")

• Find the lines where you initialize your existing modules (document_processor, vector_store, ai_generator). Add the hybrid retriever initialization directly below them:

# Initialize the hybrid retriever with access to the vector store
hybrid_retriever = HybridRetriever(vector_store)

Why pass vector_store to HybridRetriever?

The hybrid retriever wraps your existing vector store rather than replacing it. This is the composable pipeline pattern used in production. Each retrieval module handles one job and can be swapped independently.

Now you need to replace the body of your /ask endpoint so it calls the hybrid retriever instead of querying the vector store directly.

• Find your existing /ask endpoint function in main.py and replace the entire function with this:

@app.post("/ask")
async def ask_question(request: QueryRequest):
    # Retrieve chunks using the full hybrid+rerank pipeline
    results = hybrid_retriever.retrieve(request.question, top_k=3)

    if not results:
        return {"answer": "No relevant documents found.", "sources": []}

    # Extract text from reranked results for the LLM
    context_chunks = [r["text"] for r in results]
    answer = ai_generator.generate(request.question, context_chunks)

    # Return the answer with source previews and reranker scores
    return {
        "answer": answer,
        "sources": [
            {"text": r["text"][:200], "reranker_score": r.get("reranker_score")}
            for r in results
        ],
    }

What does this code do?

The only line that changed is the retrieval call. Instead of querying ChromaDB directly, the endpoint calls hybrid_retriever.retrieve() which runs all four stages: vector search, BM25 search, RRF fusion, and cross-encoder reranking.

The response now includes a reranker_score for each source chunk. This score tells you how relevant the cross-encoder thinks each chunk is to the question.

• Save main.py.

You will not see a change yet. The next substep adds the /compare endpoint, and you will test everything together after that.

Seeing an import error?

Make sure hybrid_retriever.py is in the same folder as main.py. Python will not find it if it is in a subfolder or a different directory.

Also confirm the file is named exactly hybrid_retriever.py (with an underscore, not a hyphen).

Why can't Python find my hybrid_retriever import?

Add the /compare endpoint

The /compare endpoint is what makes the quality improvement visible. It runs both vector-only and hybrid+reranked retrieval on the same query and returns both result sets side by side.

• In main.py, find your existing QueryRequest class. Add a new request model directly below it:

# Request model for the comparison endpoint
class CompareRequest(BaseModel):
    question: str

• Find your /status endpoint. Add the /compare endpoint above it by copying this code:

@app.post("/compare")
async def compare_retrieval(request: CompareRequest):
    # Run vector-only retrieval
    vector_results = hybrid_retriever.vector_only(request.question, top_k=5)
    # Run the full hybrid+reranked pipeline
    hybrid_results = hybrid_retriever.retrieve(request.question, top_k=5)

    # Return both result sets for side-by-side comparison
    return {
        "query": request.question,
        "vector_only": [
            {"text": r["text"][:200], "distance": r["distance"]}
            for r in vector_results
        ],
        "hybrid_reranked": [
            {
                "text": r["text"][:200],
                "reranker_score": r.get("reranker_score"),
            }
            for r in hybrid_results
        ],
    }

What does this code do?

The endpoint runs two retrieval strategies on the same question. vector_only() queries ChromaDB directly and returns chunks ranked by embedding distance. retrieve() runs the full pipeline: vector search, BM25, RRF fusion, and cross-encoder reranking.

Each result includes a preview of the chunk text (first 200 characters) and its score. For vector results you see the raw distance. For hybrid results you see the reranker score. Comparing these scores shows you exactly where the reranker reordered the results.

One last small change. Your /status endpoint should reflect that the API now uses hybrid retrieval.

• Find your existing /status endpoint. It currently looks like this:

@app.get("/status")
async def status():
    return {"status": "running"}

• Replace it with this updated version that includes the retrieval method:

@app.get("/status")
async def status():
    return {"status": "running", "retrieval": "hybrid_reranked"}

• Save main.py.

Not sure where to place the /compare endpoint?

The endpoint order in FastAPI does not affect routing. Place /compare anywhere after the /ask endpoint and before /status to keep your file organized.

Use the full code check tab below to verify your file structure matches.

Help me figure out where to place my /compare endpoint in main.py.

✔️ Awesome, I've got everything!

Great. Double check that you have saved main.py before moving on.

ⓧ I'd like to double check the full code

Here is the complete main.py file. Compare your version against this:

from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel

from document_processor import DocumentProcessor
from vector_store import VectorStore
from ai_generator import AIGenerator
from hybrid_retriever import HybridRetriever

app = FastAPI(title="Document Q&A API with Hybrid Search")

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_methods=["*"],
    allow_headers=["*"],
)

document_processor = DocumentProcessor()
vector_store = VectorStore()
ai_generator = AIGenerator()
hybrid_retriever = HybridRetriever(vector_store)


class QueryRequest(BaseModel):
    question: str


class CompareRequest(BaseModel):
    question: str


@app.post("/upload")
async def upload_document(file_path: str):
    chunks = document_processor.process(file_path)
    vector_store.add_chunks(chunks)
    return {"message": f"Processed and stored {len(chunks)} chunks"}


@app.post("/ask")
async def ask_question(request: QueryRequest):
    results = hybrid_retriever.retrieve(request.question, top_k=3)

    if not results:
        return {"answer": "No relevant documents found.", "sources": []}

    context_chunks = [r["text"] for r in results]
    answer = ai_generator.generate(request.question, context_chunks)

    return {
        "answer": answer,
        "sources": [
            {"text": r["text"][:200], "reranker_score": r.get("reranker_score")}
            for r in results
        ],
    }


@app.post("/compare")
async def compare_retrieval(request: CompareRequest):
    vector_results = hybrid_retriever.vector_only(request.question, top_k=5)
    hybrid_results = hybrid_retriever.retrieve(request.question, top_k=5)

    return {
        "query": request.question,
        "vector_only": [
            {"text": r["text"][:200], "distance": r["distance"]}
            for r in vector_results
        ],
        "hybrid_reranked": [
            {
                "text": r["text"][:200],
                "reranker_score": r.get("reranker_score"),
            }
            for r in hybrid_results
        ],
    }


@app.get("/status")
async def status():
    return {"status": "running", "retrieval": "hybrid_reranked"}


@app.post("/reset")
async def reset():
    vector_store.reset()
    return {"message": "All data cleared"}

Test with targeted queries

The best way to see hybrid search in action is to use queries that expose the weaknesses of vector-only retrieval. Specific technical terms, proper names, and acronyms are where BM25 keyword search shines.

• Restart your FastAPI server by running:

fastapi dev main.py

What should I see?

The server should start without errors. You will see a message like Uvicorn running on http://127.0.0.1:8000. The first request that uses the hybrid retriever may take a few seconds as the cross-encoder model loads into memory.

Server won't start?

Check that Ollama is running. Your ai_generator.py needs it for the /ask endpoint.

If you see an ImportError for hybrid_retriever, confirm the file is in the same directory as main.py and that your virtual environment is active.

Help me debug why my FastAPI server won't start after adding the hybrid retriever.

Make sure you have a test PDF uploaded from earlier. If you reset your data, upload one now before testing.

• Open a new terminal tab and test the /compare endpoint with a query that contains a specific term from your uploaded document. Run a command like this (replace the question with something relevant to your PDF):

curl -X POST http://localhost:8000/compare \
  -H "Content-Type: application/json" \
  -d '{"question": "What is the main conclusion?"}'

What should I see?

The response is a JSON object with three keys. query echoes your question. vector_only shows chunks ranked by embedding distance (lower distance means more similar). hybrid_reranked shows chunks ranked by the cross-encoder reranker score (higher score means more relevant).

Compare the two lists. You will often see different chunks in each list, or the same chunks in a different order. The hybrid+reranked list should put the most genuinely relevant chunks first.

• Try a second query with a very specific term, name, or acronym from your PDF. BM25 keyword search excels at matching exact terms that vector search blurs together.
• Try a third query that is broad and conceptual. Vector search handles these well. Compare how the hybrid results combine strengths from both strategies.

Good test query patterns

Queries with specific technical terms, error codes, or proper names expose the biggest difference between vector-only and hybrid retrieval. BM25 catches exact keyword matches that embeddings miss.

Broad conceptual questions ("What is the overall approach?") tend to perform similarly across both strategies, because vector search already handles semantic similarity well.

• Verify the /ask endpoint still works by running:

curl -X POST http://localhost:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"question": "What is the main conclusion?"}'

The response should include an answer field with the LLM's generated response and a sources array where each source now includes a reranker_score.

Getting empty results or errors?

If vector_only returns results but hybrid_reranked is empty, check that get_all_chunks() in vector_store.py returns documents. BM25 search depends on this method to build its index.

If you see a timeout or slow response on the first request, that is normal. The cross-encoder model loads into memory on the first call and subsequent requests will be faster.

Help me debug why my /compare endpoint is returning empty or unexpected results.

Secret mission

Add Retrieval Benchmarking

Your hybrid retriever is smarter, but is it slower? Build a benchmarking endpoint that measures speed and quality across all three retrieval strategies. Then add a retrieval method parameter to your /ask endpoint so you can A/B test strategies without changing code.

Before you go

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing cloud costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

• Python packages: rank_bm25 and sentence-transformers installed in your virtual environment.
• Cross-encoder model: cross-encoder/ms-marco-MiniLM-L6-v2 downloaded from Hugging Face (~80MB cached locally).
• New project file: hybrid_retriever.py.
• Updated project files: vector_store.py and main.py.
• ChromaDB data stored on disk (test documents you uploaded).
• Ollama running locally with your Llama 3.2 model.

Keep everything running

No action needed. Choose this if you are still actively building on your RAG API or plan to continue experimenting with hybrid search queries.

• Your FastAPI server will keep running until you stop it or close the terminal.
• Ollama will keep running in the background and serve model requests.
• Your ChromaDB data persists on disk between server restarts.

Pause - I'll come back to this later

Shut down running processes to free up memory. Your project files and data stay on disk so you can pick up where you left off.

• Stop your FastAPI server by pressing Ctrl+C in the terminal where it is running.
• Stop Ollama by quitting the app from your menu bar (macOS) or system tray (Windows).
• Your project files, virtual environment, ChromaDB data, and downloaded models all stay on disk.
• To resume, start Ollama, activate your virtual environment, and run fastapi dev main.py.

Delete - I don't want to use this again

Remove all project resources and start fresh. This frees up disk space used by the cross-encoder model and ChromaDB data.

• Stop your FastAPI server by pressing Ctrl+C in the terminal.
• Stop Ollama by quitting the app from your menu bar (macOS) or system tray (Windows).
• Delete your project folder from your Desktop. This removes all project files, the virtual environment, and ChromaDB data.
• The cross-encoder model is cached in your home directory under .cache/huggingface/. Delete that folder if you want to reclaim the ~80MB of disk space.
• If you no longer need Ollama for other projects, uninstall it from your system.

Mission accomplished

That's a Wrap!

Nice work! You just upgraded a basic RAG API into a production-grade retrieval system.

This is the same multi-stage retrieval architecture used by production AI products like Notion AI and Perplexity. The pattern of retrieving broadly, fusing results, then reranking precisely is the standard approach in the industry.

You've learned how to:

• Build a hybrid retrieval pipeline that combines BM25 keyword search with vector similarity search using Reciprocal Rank Fusion.
• Add a cross-encoder reranker (ms-marco-MiniLM-L6-v2) that precisely scores every retrieved chunk against the query for accurate relevance ordering.
• Create a /compare endpoint that proves the quality difference between vector-only and hybrid+reranked retrieval with real queries.
• Secret Mission: Build a benchmarking endpoint to measure speed vs. quality tradeoffs across retrieval strategies.

Ready to quiz yourself?