Hybrid Retrieval for Vision RAG

Add BM25 text retrieval and RRF fusion to a vision RAG pipeline.

Introduction

30 Second Summary

Some answers live in the words on a page, where a keyword search finds them instantly. Others hide inside charts and diagrams with no searchable text at all.

In this project, you will build a text retrieval path alongside your existing vision retriever and fuse both rankings with Reciprocal Rank Fusion. You will run controlled experiments through your 83-question eval harness to prove exactly where vision wins, where text wins, and where the hybrid beats both.

What You'll Build

Picture switching a single environment variable between three modes, running your full eval, and opening a comparison table that shows you exactly which question types each retriever dominates.

By the end of this project, you'll have:

  • Three retrieval modes (vision-only, text-only, hybrid) controlled by a single env var, each producing a full eval report over your 83-question corpus.
  • A question-level diff between any two reports, showing exactly which answers improved, regressed, or stayed the same across retrieval strategies.
  • A tag-sliced comparison table in your experiment report showing recall metrics for each retrieval mode across chart, table, figure, text, and cross-doc question types.
  • Secret Mission: An optional challenge to push your skills further.

Are there any prerequisites?

This project extends the colpali-vision-rag project. You need a working clone with the eval corpus already fetched, Docker running for Qdrant, and your Gemini API key configured.

Before We Start

Before writing any code, take a moment to lock in what you're about to extend and why it matters. You have a working vision RAG pipeline that retrieves pages as images using ColQwen2. This project adds a parallel text retrieval path so you can run a controlled experiment where the only variable is the retrieval modality.

Install Text Extraction and BM25 Dependencies

Your existing pipeline treats every PDF page as an image. ColQwen2 embeds pixel data and Qdrant searches over those embeddings. This works brilliantly for charts, diagrams, and pixel-only documents. But for pages full of readable text, there is a faster, simpler retrieval method you have not tested yet.

To build a fair comparison arm, you need two new libraries. PyMuPDF extracts the embedded text layer from PDF pages (no OCR needed). BM25 ranks pages by keyword relevance using term frequency and inverse document frequency. In this step, you will install both libraries and verify they work against your corpus.

In this step, get ready to:
  • Install PyMuPDF and rank-bm25 via uv.
  • Verify PyMuPDF extracts text from arXiv papers and returns nothing from pixel-only PDFs.
  • Verify BM25Okapi returns scored results from test data.
Add PyMuPDF and rank-bm25 to your project

PyMuPDF reads the text layer embedded inside PDF files directly. It does not use OCR. rank-bm25 provides the BM25Okapi algorithm, a classic keyword-scoring method that ranks documents by how well they match a search query.

  • Add both libraries to your project dependencies by running this command from your project root:
uv add pymupdf rank-bm25

What does this command do?

uv add adds both packages to your pyproject.toml, updates the lockfile, and installs them into your project environment in one step. You will see output showing the resolved versions for both packages.

  • Confirm both packages appear in the output. You should see version lines for pymupdf and rank-bm25 in the resolved dependency list.

Seeing a resolution error?

Make sure you are running the command from your project root directory (where pyproject.toml lives). uv add searches the current directory and parent directories for a project file.

Help me debug a uv add resolution error for pymupdf and rank-bm25.

Verify PyMuPDF extracts text from PDFs

Not all PDFs contain a text layer. Your arXiv papers have embedded text that PyMuPDF can read directly. Your sales_report.pdf is deliberately pixel-only, with no text layer at all. This difference is the fundamental reason your vision pipeline exists.

  • Create a quick verification script by saving this code as test_pymupdf.py in your project root:
import pymupdf
from pathlib import Path

# Find an arXiv paper (any PDF that is not the pixel-only sales report)
pdfs = sorted(Path("pdfs").glob("*.pdf"))
arxiv_pdf = next(p for p in pdfs if p.name != "sales_report.pdf")

# Extract text from page 1 of the arXiv paper
doc = pymupdf.open(str(arxiv_pdf))
text = doc[0].get_text(sort=True)
print(f"{arxiv_pdf.name} page 1: {len(text)} characters")
doc.close()

# Now try sales_report.pdf (pixel-only, no text layer)
doc = pymupdf.open("pdfs/sales_report.pdf")
text = doc[0].get_text(sort=True)
print(f"sales_report.pdf page 1: {len(text)} characters")
doc.close()

What does this code do?

  • pymupdf.open() opens a PDF and gives you access to its pages.
  • page.get_text(sort=True) extracts the embedded text from a page, sorted from top-left to bottom-right. If a page has no text layer, it returns an empty string.
  • The script compares an arXiv paper (which has real text) against sales_report.pdf (which is pixel-only). The character counts should make the difference obvious.

Before you run this, predict: will both PDFs return text, or will one return nothing?

  • Run the script by executing:
uv run python test_pymupdf.py

The arXiv paper should report several hundred characters. sales_report.pdf should report 0 characters. That zero is the gap your vision pipeline fills. Text retrieval will never find anything on that page.

Seeing 0 characters for both PDFs?

Check that your pdfs/ directory contains the arXiv papers from the eval corpus. If you have not fetched them yet, run scripts/fetch_eval_corpus.py first.

Help me figure out why PyMuPDF returns 0 characters from my arXiv PDFs.

Verify BM25Okapi returns scored results

BM25 scores documents by how well their terms match a query. Higher scores mean stronger keyword matches. You need to confirm the library works before building a full retriever on top of it.

  • Create a second verification script by saving this as test_bm25.py in your project root:
from rank_bm25 import BM25Okapi

# A small test corpus (each doc is a list of tokens)
corpus = [
    "the cat sat on the mat",
    "the dog played in the park",
    "attention mechanisms transform sequences",
]
tokenized = [doc.lower().split() for doc in corpus]

# Build the BM25 index over the tokenized corpus
bm25 = BM25Okapi(tokenized)

# Query the index and print scores
query = "attention mechanism"
scores = bm25.get_scores(query.lower().split())
for doc, score in zip(corpus, scores):
    print(f"  {score:.3f}  {doc}")

What does this code do?

  • BM25Okapi(tokenized) builds a keyword index from a list of tokenized documents.
  • bm25.get_scores() returns a score for every document in the corpus. The document about attention mechanisms should score highest for the query "attention mechanism". The other two should score close to zero.
  • Run the BM25 test by executing:
uv run python test_bm25.py

You should see the "attention mechanisms transform sequences" document scoring well above zero, with the other two near zero. That confirms BM25 is ranking by keyword relevance.

Seeing all zeros?

Make sure you are calling .lower().split() on both the corpus documents and the query. BM25 matching is case-sensitive, so "Attention" will not match "attention" unless both are lowercased.

Help me debug why my BM25Okapi test returns all zero scores.

  • Clean up the test files now that you have confirmed both libraries work:
rm test_pymupdf.py test_bm25.py

You now have the two building blocks for a text retrieval path: PyMuPDF to pull text from PDF pages, and BM25 to rank those pages by query relevance. Next up, you will wire these into a full retriever that mirrors your vision pipeline's interface.

Build the Text Retriever

In Step 1 you installed PyMuPDF and rank-bm25 and proved they work on individual PDFs and test strings. Now it's time to wire them into proper modules that your eval harness can use.

Your vision retriever returns ranked pages for a query using ColQwen2 image embeddings. For a controlled comparison, you need a text retriever with the exact same interface: give it a query, get back ranked pages. The only variable between the two paths should be the retrieval modality itself.

In this step, get ready to:
  • Create text_extractor.py to extract text from every page of a PDF.
  • Create text_retriever.py to build a BM25 index over all extracted page text and rank pages by keyword relevance.
  • Test the text retriever on a text-heavy query and a chart query to see where it succeeds and where it fails.
Create the text extractor module

The text extractor is a thin wrapper around PyMuPDF that opens a PDF and returns a dictionary mapping each 1-based page number to its extracted text. Pages with no text layer (like sales_report.pdf) return empty strings.

  • Create a new file src/text_extractor.py and paste the following code:
"""Extract text from PDF pages using PyMuPDF."""

import logging
from pathlib import Path

import pymupdf

log = logging.getLogger(__name__)


def extract_pages_text(pdf_path: str | Path) -> dict[int, str]:
    """Extract text from each page of a PDF.

    Returns a dict mapping 1-based page numbers to their extracted text.
    Pages with no text layer (e.g. scanned images, pixel-only renders)
    return empty strings.
    """
    pdf_path = Path(pdf_path)
    pages_text: dict[int, str] = {}
    with pymupdf.open(str(pdf_path)) as doc:
        for page_num, page in enumerate(doc, start=1):
            pages_text[page_num] = page.get_text(sort=True)
    empty = sum(1 for t in pages_text.values() if not t.strip())
    log.info(
        "%s: %d pages, %d with text, %d empty",
        pdf_path.name,
        len(pages_text),
        len(pages_text) - empty,
        empty,
    )
    return pages_text

What does this code do?

extract_pages_text() opens a PDF with pymupdf.open(), iterates over every page, and calls page.get_text(sort=True) to extract the text layer sorted top-left to bottom-right.

The function returns a dictionary keyed by 1-based page numbers. It also logs how many pages have text vs. how many are empty. That log line is your first signal for how much of the corpus the text retriever can actually see.

  • Save src/text_extractor.py.
  • Run the extractor over your full corpus to see the per-PDF breakdown by running this command:
PYTHONPATH=. uv run python -c "
import logging; logging.basicConfig(level=logging.INFO)
from pathlib import Path
from src.text_extractor import extract_pages_text
for pdf in sorted(Path('pdfs').glob('*.pdf')):
    extract_pages_text(pdf)
"

You should see a log line for each PDF in your pdfs/ directory. The arXiv papers will report hundreds of pages with text. sales_report.pdf will report all pages as empty.

Seeing a ModuleNotFoundError?

  • Make sure you're running from the project root directory (the one containing the src/ folder). The PYTHONPATH=. prefix makes imports resolve relative to that directory.
  • Confirm that pymupdf is installed in your project environment. Run uv pip list | grep -i pymupdf to check.

Why can't my text extractor find the pymupdf module?

Create the text retriever module

The TextRetriever class is the text-side counterpart to your ColQwen2 vision retriever. It scans every PDF in pdfs/, extracts text per page, builds a BM25Okapi index over all pages, and exposes a search(query, k) method returning ranked results.

  • Create a new file src/text_retriever.py and start by adding the imports, logger, and tokenizer function:
"""BM25-based text retrieval over extracted PDF page text."""

import logging
from pathlib import Path

from rank_bm25 import BM25Okapi

from src.config import PDFS_DIR
from src.text_extractor import extract_pages_text

log = logging.getLogger(__name__)


def _tokenize(text: str) -> list[str]:
    """Lowercase whitespace tokenizer."""
    return text.lower().split()

What does this code do?

  • The imports pull in BM25Okapi from rank-bm25 and the PDFS_DIR path from your existing config. They also import extract_pages_text from the module you just created.
  • _tokenize() is a simple lowercase + whitespace split. BM25 needs tokenized input, and this keeps the retriever self-contained without pulling in a heavier NLP tokenizer.
  • Below the _tokenize function, add the TextRetriever class with its constructor and index-building method:
class TextRetriever:
    """BM25 retriever over per-page extracted text."""

    def __init__(self, pdfs_dir: Path | str = PDFS_DIR) -> None:
        pdfs_dir = Path(pdfs_dir)
        self.page_ids: list[tuple[str, int]] = []
        self.page_texts: list[str] = []
        self._build_index(pdfs_dir)

    def _build_index(self, pdfs_dir: Path) -> None:
        """Extract text from all PDFs and build BM25 index."""
        for pdf_path in sorted(pdfs_dir.glob("*.pdf")):
            pages = extract_pages_text(pdf_path)
            for page_num, text in sorted(pages.items()):
                self.page_ids.append((pdf_path.name, page_num))
                self.page_texts.append(text)

        tokenized = [_tokenize(t) for t in self.page_texts]
        self.bm25 = BM25Okapi(tokenized)

        n_pdfs = len(set(pid[0] for pid in self.page_ids))
        n_with_text = sum(
            1 for t in self.page_texts if t.strip()
        )
        log.info(
            "BM25 index: %d pages from %d PDFs (%d with text)",
            len(self.page_ids),
            n_pdfs,
            n_with_text,
        )

What does this code do?

  • _build_index() scans every PDF in pdfs/, calls your extract_pages_text() on each, and flattens the results into two parallel lists: page_ids (tuples of PDF name and page number) and page_texts (the raw extracted text).
  • It tokenizes every page's text and feeds the entire corpus to BM25Okapi, which builds the inverted index and computes IDF weights. The log line at the end tells you how many pages the index contains and how many actually have text.
  • Save src/text_retriever.py and verify the index builds successfully by running:
PYTHONPATH=. uv run python -c "
import logging; logging.basicConfig(level=logging.INFO)
from src.text_retriever import TextRetriever
tr = TextRetriever()
"

You should see the per-PDF extraction logs from your text extractor, followed by a summary line like BM25 index: N pages from M PDFs (X with text). The count of pages with text will be lower than the total, because sales_report.pdf contributes only empty pages.

Index builds but shows 0 pages with text?

  • Check that the arXiv papers are present in your pdfs/ directory. If you only have sales_report.pdf, every page will be empty. Make sure you ran scripts/fetch_eval_corpus.py in the prerequisite project.
  • Confirm your PDFS_DIR in src/config.py points to the correct directory containing your PDFs.

Why is my BM25 index showing zero pages with text?

  • Now add the search method below the _build_index method inside the TextRetriever class:
    def search(
        self, query: str, k: int = 12
    ) -> list[tuple[str, int, float]]:
        """Return top-k pages ranked by BM25 score.

        Returns list of (pdf_name, page_number, score) tuples,
        best first.
        """
        tokenized_query = _tokenize(query)
        scores = self.bm25.get_scores(tokenized_query)

        ranked = sorted(
            zip(self.page_ids, scores),
            key=lambda x: x[1],
            reverse=True,
        )
        return [
            (pdf_name, page_num, score)
            for (pdf_name, page_num), score in ranked[:k]
        ]

What does this code do?

  • search() tokenizes the query with the same _tokenize() function that was used to build the index, then calls bm25.get_scores() to compute a relevance score for every page in the corpus.
  • It returns the top-k pages as (pdf_name, page_number, score) tuples, which matches the format your vision retriever uses. This shared shape is what lets the eval harness treat both retrievers interchangeably.
  • Save src/text_retriever.py.

✔️ Awesome, I've got everything!

Make sure you've saved src/text_retriever.py with the imports, _tokenize function, and the full TextRetriever class.

ⓧ I'd like to double check the full code

Your src/text_retriever.py should contain three sections in this order. Scroll up and confirm each one is present:

  • Module docstring, imports (BM25Okapi, PDFS_DIR, extract_pages_text), logger, and the _tokenize() function.
  • TextRetriever class with __init__ and _build_index methods.
  • search() method inside the TextRetriever class, returning (pdf_name, page_number, score) tuples.
Test the text retriever with contrasting queries

You have a working text retriever. Now you need to see where it shines and where it falls flat. Two queries will reveal the pattern: one that plays to text retrieval's strength, and one that exposes its blind spot.

Before you run these queries, think about what you expect. The first query asks about the attention mechanism, a concept explained in dense prose across the arXiv papers. The second asks about Q4 revenue, which lives in a pixel-only chart with no text layer. What do you think the text retriever will return for each?

  • Test the text retriever with both queries by running:
PYTHONPATH=. uv run python -c "
from src.text_retriever import TextRetriever
tr = TextRetriever()

print('=== Text query: attention mechanism ===')
for pdf, page, score in tr.search('What is the attention mechanism?', k=5):
    print(f'  {pdf} p{page}: {score:.4f}')

print()
print('=== Chart query: Q4 revenue ===')
for pdf, page, score in tr.search('Q4 revenue chart', k=5):
    print(f'  {pdf} p{page}: {score:.4f}')
"

The attention mechanism query should return arXiv pages ranked at the top with meaningful BM25 scores. The Q4 revenue query should return either irrelevant pages with near-zero scores or pages that happen to contain the word "revenue" in a completely unrelated context.

Why does the chart query fail?

sales_report.pdf is deliberately pixel-only. There is no text layer for PyMuPDF to read. BM25 can only rank pages it has tokens for, so a page with no text is invisible to the text retriever.

This is the designed limitation. Your vision retriever (ColQwen2) handles that chart query by treating the page as an image and matching visual patterns. The text retriever handles the attention mechanism query by matching keywords in the extracted prose. Neither retriever covers both. That gap is exactly what the hybrid fusion in Step 4 will close.

Both queries returning near-zero scores?

  • If the attention mechanism query also returns near-zero scores, your BM25 index may not contain the arXiv paper text. Re-run the extractor test from the previous substep to confirm the arXiv papers report pages with text.
  • Check that the search() method is indented inside the TextRetriever class. If it's at the module level, it won't have access to self.bm25.

Why are both of my text retriever queries returning zero-score results?

You now have a complete text retrieval path that mirrors your vision pipeline's interface. You've seen it succeed on text-heavy questions and fail on visual ones. In the next step, you'll run this text retriever through the full 83-question eval harness to get real numbers instead of anecdotes.

Run Text-Only Retrieval Eval

In Step 2, two anecdotal queries showed your text retriever succeeding on a text-heavy question and failing on a chart question. Those two data points gave you intuition, but they can't tell you the full story across all 83 evaluation questions.

In this step, you'll add a RETRIEVAL_MODE switch to your pipeline config, run the full eval harness in text-only mode, and compare the results against your vision baseline. The numbers will prove exactly where and by how much text retrieval falls short.

In this step, get ready to:
  • Wire a RETRIEVAL_MODE switch into your config and retrieval module.
  • Run a text-only retrieval eval and save the report.
  • Compare text-only recall numbers against the vision baseline by tag.
Add a retrieval mode switch to your pipeline

Your eval harness already calls retrieve() from src/retrieval.py for every question. By adding a single env-overridable config option, you can swap the retrieval backend without touching any eval code.

  • Open src/config.py and find the RETRIEVE_K line. Add this config option directly above it:
# Retrieval mode: vision (ColQwen2+Qdrant), text (BM25), or hybrid
RETRIEVAL_MODE = os.environ.get("RETRIEVAL_MODE", "vision")

What does this config option do?

It defaults to "vision", so your existing pipeline works unchanged. Setting RETRIEVAL_MODE=text via the environment swaps retrieval to your BM25 text retriever. This follows the same env-overridable pattern your project already uses for RETRIEVE_K, RERANK_K, and every other tuning knob.

Now you need to wire the mode into the retrieval function itself. Your retrieve() function currently embeds the question with ColQwen2 and searches Qdrant. You'll add a conditional branch that uses TextRetriever when the mode is "text".

  • In src/retrieval.py, add these two imports alongside the existing imports at the top of the file:
from src.config import RETRIEVAL_MODE, RETRIEVE_K
from src.text_retriever import TextRetriever
  • Find the retrieve() function. Add this conditional branch at the very top of the function body, before the existing queries = decompose(question) line:
def retrieve(question: str) -> list[dict]:
    # Text-only mode: BM25 retrieval, no vision embeddings
    if RETRIEVAL_MODE == "text":
        text_retriever = TextRetriever()
        results = text_retriever.search(question, k=RETRIEVE_K)
        return [
            {"pdf": pdf, "page_number": page, "score": score,
             "image_path": ""}
            for pdf, page, score in results
        ]

    # Existing vision retrieval continues below
    queries = decompose(question) if QUERY_DECOMPOSE else [question]

What does this wiring do?

When RETRIEVAL_MODE is "text", the function returns early with BM25 results, skipping ColQwen2 embedding and Qdrant search entirely. The list comprehension converts TextRetriever's (pdf_name, page_number, score) tuples into the same dict format the rest of the pipeline expects.

The image_path is set to an empty string because text retrieval doesn't use page images. This is fine for retrieval-only eval, which only checks whether the gold page appears in the ranked list.

  • Save both src/config.py and src/retrieval.py.

✔️ Awesome, I've got everything!

Make sure both files are saved. The config change adds RETRIEVAL_MODE above RETRIEVE_K, and the retrieval change adds the text branch at the top of retrieve().

ⓧ I'd like to double check the full code

Your changes are small additions to two existing files. Confirm each one is present:

  • In src/config.py: the RETRIEVAL_MODE line sits directly above RETRIEVE_K. It reads os.environ.get("RETRIEVAL_MODE", "vision").
  • In src/retrieval.py: two new imports at the top (RETRIEVAL_MODE, RETRIEVE_K, and TextRetriever), and the if RETRIEVAL_MODE == "text": branch as the first thing inside retrieve().
Run the text-only retrieval eval

Before you run this, think about what you expect. Your vision baseline achieves recall@12 of 1.000. That means the gold page appears somewhere in the top 12 results for every single answerable question. Do you think text-only retrieval will match that?

  • Run the retrieval-only eval with text mode by running this command:
RETRIEVAL_MODE=text GEMINI_API_KEY= PYTHONPATH=. uv run python eval/run_eval.py --retrieval-only --output eval/reports/text_only.json

What does this command do?

  • RETRIEVAL_MODE=text activates your new BM25 retrieval path.
  • GEMINI_API_KEY= is intentionally empty. Retrieval-only mode doesn't call Gemini, so no API key is needed and no tokens are spent.
  • --retrieval-only measures recall@k only, without running rerank, answer, or judge stages.
  • --output eval/reports/text_only.json saves the report to a fixed path so you can reference it in later steps.

The eval runs all 83 questions through your BM25 text retriever and prints a results table when it finishes.

Seeing an import error?

  • Make sure you saved both src/config.py and src/retrieval.py.
  • Check that the import in src/retrieval.py spells RETRIEVAL_MODE exactly as it appears in src/config.py.
  • Confirm your PYTHONPATH=. prefix is present so Python can find the src package.

Still stuck? Help me debug my text-only eval import error.

Compare text-only recall against the vision baseline

Your report is saved. Now compare the text-only numbers against the pinned vision baseline to see exactly where text retrieval falls short.

  • Open eval/reports/text_only.json and find the summary section. Note the recall@1, recall@3, and recall@12 values.
  • Compare them against your vision baseline:
  • Vision recall@1: 0.671. Vision recall@3: 0.863. Vision recall@12: 1.000.
  • Text-only recall@12 is well below 1.000. Pages with no text layer (like your pixel-only sales_report.pdf) are invisible to BM25, so they can never appear in the results.
  • Now scroll to the per_tag section in the report. Look at the tag-level breakdown and compare recall across tags like text, chart, and figure.

What should the per-tag breakdown show?

Text retrieval may actually match or beat vision on the text tag, where questions target text-heavy pages that BM25 handles well. On the chart and figure tags, text retrieval scores near zero because those pages have no extractable text layer.

This per-tag pattern is the whole point. Overall recall hides the story. Tag-level recall reveals that each retriever has a strength the other lacks. That complementarity is what Reciprocal Rank Fusion will exploit in the next step.

Your text-only numbers prove the gap. Vision saturates recall@12 at 1.000 while text misses entire pages. But the per-tag breakdown hints at something useful. In the next step, you'll combine both retrievers with Reciprocal Rank Fusion so each covers the other's blind spots.

Implement RRF Fusion and Run Hybrid Eval

Your vision retriever scores 0.671 recall@1, meaning about a third of questions have the wrong page ranked first. Your text retriever fails entirely on charts and figures. But for text-heavy questions, the text retriever might rank the gold page higher than vision does.

Reciprocal Rank Fusion combines both ranked lists so each retriever's strengths cover the other's blind spots. In this step, you will implement the RRF formula, wire it into a hybrid retrieval mode, and run a full eval to test whether the fusion lifts recall.

In this step, get ready to:
  • Implement the RRF fusion formula in a new module.
  • Wire a hybrid retrieval mode into the pipeline.
  • Run a full hybrid eval and compare against the vision baseline.
Create the RRF fusion module

The RRF formula scores each page based on where it appears in each retriever's ranking. A page ranked #1 gets a higher contribution than one ranked #12. Pages that appear in both lists accumulate scores from both, pushing them to the top of the fused ranking.

  • In src/config.py, find your existing config options (where RETRIEVAL_MODE is defined).
  • Add the following three RRF config options below your existing options:
# RRF fusion parameters (env-overridable)
RRF_K: int = int(os.environ.get("RRF_K", "60"))
RRF_VISION_WEIGHT: float = float(os.environ.get("RRF_VISION_WEIGHT", "1.0"))
RRF_TEXT_WEIGHT: float = float(os.environ.get("RRF_TEXT_WEIGHT", "1.0"))

What are these config values?

RRF_K is the constant k in the RRF formula. The value 60 is the industry-standard default established by Cormack, Clarke, and Buettcher (SIGIR 2009). It is robust within the range [40, 80].

RRF_VISION_WEIGHT and RRF_TEXT_WEIGHT default to 1.0 (equal weighting). You will use them in the Secret Mission to sweep for a better balance.

  • Save src/config.py.
  • Now create a new file src/fusion.py and paste the following code:
from src.config import RRF_K, RRF_TEXT_WEIGHT, RRF_VISION_WEIGHT

def reciprocal_rank_fusion(
    vision_ranked: list[tuple[str, int, float]],
    text_ranked: list[tuple[str, int, float]],
    k: int = RRF_K,
    vision_weight: float = RRF_VISION_WEIGHT,
    text_weight: float = RRF_TEXT_WEIGHT,
) -> list[tuple[str, int, float]]:
    """Fuse vision and text ranked lists using RRF.

    Each input is a list of (pdf_name, page_number, score) tuples,
    best first. The score field is ignored; only rank position matters.

    Returns fused list of (pdf_name, page_number, rrf_score), best first.
    """
    scores: dict[tuple[str, int], float] = {}

    for rank, (pdf, page, _) in enumerate(vision_ranked, start=1):
        key = (pdf, page)
        scores[key] = scores.get(key, 0.0) + vision_weight / (k + rank)

    for rank, (pdf, page, _) in enumerate(text_ranked, start=1):
        key = (pdf, page)
        scores[key] = scores.get(key, 0.0) + text_weight / (k + rank)

    fused = sorted(scores.items(), key=lambda x: x[1], reverse=True)
    return [(pdf, page, score) for (pdf, page), score in fused]

What does this code do?

The formula RRF_score(d) = sum_r weight_r / (k + rank_r(d)) gives each page a score based on its rank position in each retriever's list. A page ranked #1 in vision contributes 1/(60+1). A page ranked #12 contributes 1/(60+12).

Pages that appear in both lists get scores from both retrievers added together, which pushes them higher in the fused ranking. Pages that only one retriever found still appear, just with a lower combined score.

  • Save src/fusion.py.
  • Verify the fusion function works by running this test in your terminal:
PYTHONPATH=. uv run python -c "
from src.fusion import reciprocal_rank_fusion

vision = [('paper_a.pdf', 1, 0.9), ('paper_b.pdf', 3, 0.7), ('paper_c.pdf', 5, 0.5)]
text   = [('paper_b.pdf', 3, 8.1), ('paper_d.pdf', 2, 6.3), ('paper_a.pdf', 1, 4.0)]

fused = reciprocal_rank_fusion(vision, text)
for pdf, page, score in fused:
    print(f'{pdf} p{page}: {score:.6f}')
"

paper_b.pdf page 3 should appear at the top. It was ranked #2 by vision and #1 by text, so it accumulates RRF score from both lists. paper_a.pdf page 1 should come second (ranked #1 by vision, #3 by text).

Seeing an import error?

  • Make sure you saved both src/config.py and src/fusion.py before running the command.
  • Check that RRF_K, RRF_VISION_WEIGHT, and RRF_TEXT_WEIGHT are defined in src/config.py. The import in fusion.py expects all three names to exist.
  • help me debug an import error in src/fusion.py

✔️ Awesome, I've got everything!

Great. Make sure you have saved src/fusion.py.

ⓧ I'd like to double check the full code

Scroll up and confirm your src/fusion.py has these two parts in order:

  • The import line: from src.config import RRF_K, RRF_TEXT_WEIGHT, RRF_VISION_WEIGHT.
  • The reciprocal_rank_fusion() function with its full docstring and both for loops (one for vision, one for text).
Wire hybrid mode into the retrieval pipeline

Your src/retrieval.py already routes to TextRetriever when RETRIEVAL_MODE is text. Now you need a third branch for hybrid that runs both retrievers and fuses their results.

  • Open src/retrieval.py.
  • Add the following imports at the top of the file:
from src.fusion import reciprocal_rank_fusion
  • Find the section where RETRIEVAL_MODE is checked. You should see branches for vision and text.
  • Add a hybrid branch that does three things in sequence:
  • Run the existing ColQwen2 + Qdrant vision retrieval to get RETRIEVE_K (12) candidate results.
  • Run TextRetriever().search(query, k=RETRIEVE_K) to get 12 BM25 candidate results.
  • Pass both result lists to reciprocal_rank_fusion(vision_results, text_results) and use the fused ranking as the retrieval output.

Why run both retrievers in sequence?

RRF only needs rank positions, not raw scores, so the two retrievers can run independently. The vision retriever returns 12 pages ranked by ColQwen2 MaxSim scores. The text retriever returns 12 pages ranked by BM25 scores. RRF does not care about the score magnitudes. It only uses where each page landed in its retriever's ranking.

  • Save src/retrieval.py.
  • Verify the hybrid mode wiring works by running a retrieval-only eval (no Gemini needed):
RETRIEVAL_MODE=hybrid GEMINI_API_KEY= PYTHONPATH=. uv run python eval/run_eval.py --retrieval-only

You should see the eval harness run through all 83 questions using the hybrid retriever. The recall numbers at the end confirm the fusion is working. If the run completes without errors, your wiring is correct.

Getting an error about the retrieval mode?

  • Make sure your hybrid branch in src/retrieval.py matches the string "hybrid" exactly (lowercase, no extra spaces).
  • Check that reciprocal_rank_fusion is imported from src.fusion at the top of retrieval.py.
  • Confirm the hybrid branch returns results in the same (pdf_name, page_number, score) tuple format that the rest of the pipeline expects.
  • help me debug the hybrid retrieval mode in src/retrieval.py
Run the full hybrid eval and compare results

The retrieval-only run confirmed that the wiring works. Now you need the full pipeline eval, which includes Gemini reranking, answer generation, and judge scoring. This produces the complete report you will compare against the vision baseline.

Before you run it, think about what you expect. Your vision baseline has recall@1 of 0.671. The text retriever failed on charts and figures but succeeded on text-heavy questions. Do you think the hybrid recall@1 will be higher, lower, or the same as the vision-only baseline?

  • Run the full hybrid eval by running this command:
RETRIEVAL_MODE=hybrid PYTHONPATH=. uv run python eval/run_eval.py --judge

What does --judge do that --retrieval-only doesn't?

The --judge flag runs all 83 questions through Gemini for reranking and answer generation, then uses a Gemini judge to score correctness. This costs a few cents of API usage. The --retrieval-only mode only measures recall metrics from the retrieval step with no Gemini calls.

  • When the eval finishes, save the report as eval/reports/hybrid.json by copying or renaming the output report file:
cp eval/reports/$(ls -t eval/reports/ | head -1) eval/reports/hybrid.json
  • Open eval/reports/hybrid.json and find the recall metrics. Compare against the vision baseline:
  • Vision baseline recall@1: 0.671.
  • Vision baseline recall@3: 0.863.
  • Vision baseline recall@12: 1.000.

Your hybrid recall@1 should be equal to or higher than 0.671. The hypothesis is that text retrieval rescues some of the 33% of questions where vision's top-1 pick was wrong. When both retrievers agree on a page, RRF pushes it to the top of the fused ranking.

Why might recall@12 not change much?

The vision baseline already has recall@12 of 1.000, meaning every gold page appears somewhere in the top 12 results. RRF can improve where the gold page ranks within those 12 (lifting recall@1 and recall@3), but it cannot push recall@12 above 1.0. The win is at the top of the ranking, not the bottom.

Hybrid recall@1 lower than expected?

  • Make sure both retrievers return RETRIEVE_K (12) results each. If the text retriever returns fewer candidates, RRF has less signal to work with.
  • Confirm that the vision results and text results are passed to reciprocal_rank_fusion() in the correct order (vision first, text second).
  • Check that the fused results are used as the final retrieval output. If the code falls through to the vision-only branch, you are not actually testing hybrid retrieval.
  • help me understand why my hybrid recall@1 is lower than vision-only

You now have three retrieval modes (vision, text, hybrid) producing three JSON reports. Next up, you will diff these reports, slice the results by question tag, and write up a proper experiment report that explains exactly where each retriever wins and why.

Analyze Results and Write the Experiment Report

You now have three eval reports sitting in eval/reports/: the pinned vision baseline, text-only retrieval, and hybrid RRF fusion.

JSON files full of numbers are not an experiment. In this step, you will diff the reports question-by-question to find which questions flipped, build a tag-level comparison table that shows where each retrieval mode wins, and write up the full experiment in EXPERIMENTS.md with reproducible commands.

In this step, get ready to:
  • Diff three report pairs to identify which questions flipped between retrievers.
  • Build a tag-level comparison table showing where vision, text, and hybrid each excel.
  • Document the experiment in EXPERIMENTS.md with hypothesis, results, interpretation, and reproduction commands.
Diff the three report pairs

Your eval harness produced reports with recall metrics for every question. The eval/diff_reports.py tool joins two reports on question ID and prints which questions improved, regressed, or stayed the same.

Before you run the first diff, think about this: if you swap your vision retriever for text-only BM25, do you expect more questions to improve or regress? Why?

  • Run the first diff comparing your vision baseline against text-only retrieval:
PYTHONPATH=. uv run python eval/diff_reports.py eval/reports/baseline_relabeled.json eval/reports/text_only.json

You should see a large number of regressions. Text-only retrieval cannot find pages that have no text layer (charts, figures), so those questions all drop.

  • Run the second diff comparing your vision baseline against hybrid retrieval:
PYTHONPATH=. uv run python eval/diff_reports.py eval/reports/baseline_relabeled.json eval/reports/hybrid.json

This diff tells the real story. Look at the improved count. These are questions where adding text retrieval rescued the gold page that vision alone ranked too low.

  • Run the third diff comparing text-only against hybrid:
PYTHONPATH=. uv run python eval/diff_reports.py eval/reports/text_only.json eval/reports/hybrid.json

What patterns should I look for?

Scan the flipped questions for tag patterns. Do the improvements in the hybrid diff cluster around text or cross-doc tags? Do any regressions appear on chart or figure tags?

If the hybrid shows improvements with zero or near-zero regressions compared to vision, that confirms the fusion is adding signal rather than noise.

Getting a file not found error?

Make sure you are running the command from the project root directory. The paths eval/reports/baseline_relabeled.json, eval/reports/text_only.json, and eval/reports/hybrid.json are relative to the project root.

If your baseline report has a different filename, check your eval/reports/ directory with ls eval/reports/ and use the correct filename.

Help me debug my diff_reports.py command.

Build the tag-level comparison table

The diffs show you which questions flipped. Now you need the aggregate story: how does each retrieval mode perform on each question type?

  • Open eval/reports/baseline_relabeled.json in your editor.
  • Find the per-tag breakdown section.
  • For each tag (chart, table, figure, text, cross-doc, unanswerable), note the recall@1 and recall@3 values.
  • Repeat for eval/reports/text_only.json and eval/reports/hybrid.json.

What should the tag-level patterns look like?

Text retrieval should dominate on the text tag, where pages have rich embedded text that BM25 matches directly. It should score near zero on chart and figure tags, where pages are pixel-only.

Vision retrieval should dominate on visual tags. The hybrid should match or beat both everywhere, because Reciprocal Rank Fusion only adds signal from a retriever when it ranks a page highly. A retriever that returns nothing useful contributes zero to the fused score.

Write the experiment section in EXPERIMENTS.md

A table of numbers is the raw data. The experiment report is where you turn that data into an argument: what you hypothesized, how you tested it, what you found, and what it means.

  • Open docs/EXPERIMENTS.md in your editor.
  • Add the following experiment heading, hypothesis, and methodology section:
## Hybrid Retrieval Experiment

### Hypothesis

Adding BM25 text retrieval alongside ColQwen2 vision retrieval
and fusing with Reciprocal Rank Fusion (k=60) will improve
recall@1 by rescuing text-heavy questions where vision's
top-1 pick was wrong.

### Methodology

- **Corpus**: 363 pages across 19 PDFs (3 gold + 16 distractors)
- **Eval set**: 83 labeled questions (73 answerable + 10 unanswerable)
- **Retrieval modes**: vision-only (ColQwen2 + Qdrant), text-only (PyMuPDF + BM25), hybrid (RRF k=60)
- **Metrics**: recall@1, recall@3, recall@12

What does this section establish?

The hypothesis states your prediction up front so the results can confirm or reject it. The methodology ensures reproducibility. Anyone reading this can see you used the same 83 questions and the same corpus across all three modes, so the only variable is the retrieval strategy.

  • Below the methodology, add the results table. Fill in every cell with the per-tag recall values you extracted from your three reports:
### Results

| Tag | V R@1 | V R@3 | T R@1 | T R@3 | H R@1 | H R@3 |
|-----|-------|-------|-------|-------|-------|-------|
| chart |  |  |  |  |  |  |
| table |  |  |  |  |  |  |
| figure |  |  |  |  |  |  |
| text |  |  |  |  |  |  |
| cross-doc |  |  |  |  |  |  |
| unanswerable |  |  |  |  |  |  |
| **Overall** | 0.671 | 0.863 |  |  |  |  |

What do V, T, and H mean?

V is vision-only (ColQwen2), T is text-only (BM25), and H is hybrid (RRF fusion). R@1 and R@3 are recall at rank 1 and 3 respectively. Fill in every cell from your report JSON files. The overall vision values (0.671, 0.863) are already pre-filled from your pinned baseline.

  • Below the table, add your interpretation and the reproduction commands:
### Interpretation

(Write your analysis here: which tags does vision win on?
Where does text beat vision? Where does hybrid beat both?
What explains the tradeoffs you observed?)

### Reproduction Commands

Vision-only baseline:
RETRIEVAL_MODE=vision PYTHONPATH=. uv run python eval/run_eval.py --judge

Text-only (retrieval only, no Gemini needed):
RETRIEVAL_MODE=text GEMINI_API_KEY= PYTHONPATH=. uv run python eval/run_eval.py --retrieval-only

Hybrid (full pipeline):
RETRIEVAL_MODE=hybrid PYTHONPATH=. uv run python eval/run_eval.py --judge

What should the interpretation cover?

Name the specific tags where each retriever wins and explain why. For example, text retrieval fails on chart because those PDFs have no text layer. Vision retrieval may underperform on text because BM25 keyword matching is a more direct signal for text-heavy questions.

The key insight is whether hybrid recall@1 exceeds the vision baseline of 0.671. If it does, the fusion proved that combining modalities lifts the top-1 result without sacrificing coverage.

  • Replace the interpretation placeholder with your actual analysis based on the patterns you observed in the diffs and the tag-level table.
  • Save docs/EXPERIMENTS.md.

Not sure what to write in the interpretation?

Go back to your diff outputs. Count how many of the improved questions in the vision-vs-hybrid diff carry the text or cross-doc tag. That pattern is your argument.

Help me write my experiment interpretation for the hybrid retrieval results.

Before you verify, do you think the three reproduction commands in your EXPERIMENTS.md would produce the same table if someone else ran them on the same corpus? What would be different?

  • Confirm your experiment report is complete by running:
cat docs/EXPERIMENTS.md

You should see a complete experiment section with a hypothesis, methodology listing three retrieval modes, a filled-in results table showing recall@1 and recall@3 for each tag across all three modes, a paragraph of interpretation, and three reproduction commands.

Missing sections in your EXPERIMENTS.md?

Check that your file has all five sections: Hypothesis, Methodology, Results (with the full tag-level table), Interpretation, and Reproduction Commands. Each section should have its own ### heading.

Help me fix the structure of my EXPERIMENTS.md.

You now have a rigorous, reproducible experiment report showing exactly where vision, text, and hybrid retrieval each win. This is the kind of measured, evidence-backed analysis that distinguishes an ML engineer from someone who just assembled APIs. Next up, there is an optional Secret Mission to push the fusion even further with weighted RRF tuning.

Secret mission

Weighted RRF with Per-Source Tuning

Your equal-weight RRF fusion treats both retrievers as equally trustworthy. But your corpus is heavily visual, so vision probably deserves more influence. In this mission, you will write a parameter sweep script, test 16 weight combinations against your eval harness, and find the optimal balance for your corpus.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your new hybrid retrieval files, pause your Docker containers, or remove everything you added in this project.

This project runs entirely locally. The only external cost was Gemini API usage during full --judge eval runs, which is already consumed. There are no ongoing charges.

Resources you used:

  • New source files: src/text_extractor.py, src/text_retriever.py, src/fusion.py.
  • New eval reports: eval/reports/text_only.json and eval/reports/hybrid.json.
  • New sweep script: scripts/sweep_rrf_weights.py.
  • Updated files: src/config.py, src/retrieval.py, and docs/EXPERIMENTS.md.
  • Installed Python packages: pymupdf and rank-bm25 (managed by uv).

Keep everything running

No action needed. Choose this if you plan to keep experimenting with retrieval modes or tuning RRF weights.

  • Your Qdrant Docker container and indexed corpus persist as before.
  • The BM25 text index is built in-memory each time you run the text or hybrid retriever. No additional persistent storage is used.
  • Setting RETRIEVAL_MODE=vision (the default) restores the original vision-only pipeline behavior at any time.

Pause - I'll come back to this later

Stop running processes to free up memory. Your files and reports stay on disk.

  • Stop the Qdrant container by running:
docker compose stop qdrant
  • Everything else is stateless. The text retriever, fusion module, and eval reports are all files on disk that need no running process.

Delete - I don't want to use this again

Remove all files and packages added during this project. This restores your repository to the original vision-only pipeline.

  • Remove the new source files and sweep script by running:
rm src/text_extractor.py src/text_retriever.py src/fusion.py scripts/sweep_rrf_weights.py
  • Remove the eval reports generated during this project by running:
rm eval/reports/text_only.json eval/reports/hybrid.json
  • Remove the packages you installed by running:
uv remove pymupdf rank-bm25
  • Revert the changes you made to src/config.py and src/retrieval.py using git:
git checkout src/config.py src/retrieval.py
  • Your docs/EXPERIMENTS.md contains your experiment writeup. Keep it if you want to reference your analysis later, or revert it with git checkout docs/EXPERIMENTS.md.
  • After these steps, RETRIEVAL_MODE=vision is the only mode and your pipeline behaves exactly as it did before this project.

That's a Wrap!

That's a Wrap!

Nice work! You just extended a vision-only RAG pipeline into a rigorous hybrid retrieval system, then proved exactly where each modality wins with measured, reproducible evidence.

You've learned how to:

  • Build a BM25 text retrieval path using PyMuPDF and rank-bm25 alongside your existing ColQwen2 vision retriever, then fuse both rankings with Reciprocal Rank Fusion into a single hybrid slate controlled by one env var.
  • Run three controlled retrieval experiments (vision-only, text-only, hybrid) through your 83-question eval harness and produce diffable JSON reports for each.
  • Slice results by question tag to show where vision beats text, where text beats vision, and where the hybrid beats both. Then document every finding in EXPERIMENTS.md with reproducible commands.
  • Secret Mission: Completed an optional challenge to push your hybrid retrieval skills further.

Ready to quiz yourself?