Research Paper Scout with Hybrid RAG

Build an AI agent that fetches and queries research papers using hybrid retrieval.

Introduction

30 Second Summary

Staying on top of the latest research in any field feels like drinking from a firehose. Hundreds of papers are published every week, and finding the ones that actually matter to your work takes hours of manual searching.

In this project, you will build a Research Paper Scout agent that fetches the latest academic papers on any topic and uses advanced retrieval techniques to answer your questions about them.

What You'll Build

A command-line Python agent that combines BM25 keyword search, dense vector embeddings, and neural reranking to build a production-quality RAG pipeline over real academic papers from Semantic Scholar.

By the end of this project, you'll have:

  • A paper fetching system that pulls the 20 most recent papers from Semantic Scholar on any topic you choose.
  • A hybrid retrieval index combining dense vector embeddings and sparse BM25 search with Reciprocal Rank Fusion for precise document matching.
  • An interactive query agent that synthesizes insights from papers and generates project ideas grounded in cutting-edge research.
  • Secret Mission: Add a citation graph explorer that follows reference chains to discover influential papers.

Are there any prerequisites?

You should be comfortable with basic Python (writing functions, using pip, and working with virtual environments). No prior experience with RAG, embeddings, or AI APIs is needed.

Set Up Your Environment and API Access

Before your Research Paper Scout can fetch and analyze papers, it needs a foundation. You need the right Python packages for retrieval, embeddings, and LLM synthesis, plus an API key so your agent can talk to NVIDIA's models.

In this step, you will create the project directory, install all dependencies, obtain a free NVIDIA API key, and confirm everything connects properly.

In this step, get ready to:
  • Create the project directory, virtual environment, and install all required packages.
  • Get your free NVIDIA API key from build.nvidia.com.
  • Verify that embeddings, LLM, and ChromaDB all work end-to-end.
Create your project and install dependencies

Your agent needs a dedicated folder and an isolated Python environment so its packages don't collide with anything else on your system.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal and press Enter to open it.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir research-scout
cd research-scout
  • Create a Python virtual environment by running:
python3 -m venv venv

What is a virtual environment?

A virtual environment is an isolated space where Python packages are installed separately from your system Python. This prevents version conflicts between different projects on your machine.

  • Activate the virtual environment so packages install into it:

macOS

source venv/bin/activate

Windows

.\venv\Scripts\Activate.ps1

You should see (venv) appear at the start of your terminal prompt. This confirms you are working inside the virtual environment.

Don't see (venv) in your prompt?

On Windows PowerShell, you may need to allow script execution first. Run Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned and then try the activation command again.

On macOS, make sure you are in the research-scout folder (run pwd to check). The venv folder must exist in your current directory.

Help me activate my virtual environment.

  • Install all required packages by running this command:
pip install llama-index-core llama-index-llms-nvidia llama-index-embeddings-nvidia llama-index-postprocessor-nvidia-rerank llama-index-retrievers-bm25 llama-index-vector-stores-chroma chromadb requests nest-asyncio

This installs the LlamaIndex framework (core + NVIDIA integrations), ChromaDB for vector storage, and supporting libraries. The install may take a minute or two.

Seeing installation errors?

If you see errors mentioning pip not found, make sure your virtual environment is activated (you should see (venv) in your prompt). If packages fail to build, ensure you are running Python 3.10 or higher by running python3 --version.

Help me fix my pip install errors.

  • Create a requirements.txt file to record your project's dependencies. Open VS Code in this folder by running:
code .
  • In VS Code, click File in the top menu bar.
  • Click New File.
  • Name the file requirements.txt and press Enter.
  • Paste the following package list into requirements.txt:
llama-index-core
llama-index-llms-nvidia
llama-index-embeddings-nvidia
llama-index-postprocessor-nvidia-rerank
llama-index-retrievers-bm25
llama-index-vector-stores-chroma
chromadb
requests
nest-asyncio

What is requirements.txt for?

This file documents every package your project depends on. Anyone (including future you) can recreate the environment by running pip install -r requirements.txt. It is a standard convention for Python projects.

  • Save requirements.txt with Cmd+S (macOS) or Ctrl+S (Windows).
  • Confirm the file exists by running this in your terminal:
ls

You should see requirements.txt and venv in the output.

Get your free NVIDIA API key

Your agent uses NVIDIA NIM for embeddings, reranking, and LLM synthesis. All three services are accessible through a single free API key from the NVIDIA API Catalog.

  • Open your browser and go to build.nvidia.com.
  • Create a free account (or sign in if you already have an NVIDIA account).
  • Click your profile icon in the top right corner, then click API Keys.
  • Click Generate API Key.
  • Click Generate Key.
  • Copy the key that appears. It starts with nvapi-.

Save your key somewhere safe

You will only see the full key once. Copy it now and paste it into the .env file in the next instruction. If you lose it, you can generate a new one from the same page.

  • In VS Code, create a new file called .env in your research-scout folder.
  • Paste the following into .env, replacing nvapi-your-key-here with the key you just copied:
NVIDIA_API_KEY=nvapi-your-key-here
  • Save .env with Cmd+S (macOS) or Ctrl+S (Windows).

Why a .env file?

Storing secrets in a .env file keeps them out of your source code. You should never commit API keys to version control. The .env file is a reference so you always know where your key lives.

Verify the setup works

Before building the full agent, you should confirm that the NVIDIA API key works, embeddings generate correctly, and ChromaDB initializes without issues.

  • In your terminal, export your NVIDIA API key so Python can access it. Replace nvapi-your-key-here with your actual key:

macOS

export NVIDIA_API_KEY=nvapi-your-key-here

Windows

$env:NVIDIA_API_KEY="nvapi-your-key-here"
  • Create a file called test_setup.py in VS Code and paste the following verification script:
import os
import chromadb
from llama_index.embeddings.nvidia import NVIDIAEmbedding
from llama_index.llms.nvidia import NVIDIA

# Check 1: ChromaDB initializes
db = chromadb.PersistentClient(path="./test_chroma")
print("ChromaDB initialized successfully.")

# Check 2: NVIDIA embedding generates a vector
embed_model = NVIDIAEmbedding(model="NV-Embed-QA", truncate="END")
result = embed_model.get_text_embedding("test query")
print(f"Embedding generated: {len(result)} dimensions")

# Check 3: NVIDIA LLM responds to a message
llm = NVIDIA(model="meta/llama-3.1-8b-instruct")
response = llm.complete("Say hello in one word.")
print(f"LLM response: {response}")

print("\nAll checks passed!")

What does this script do?

This script runs three independent checks to verify your environment:

  • chromadb.PersistentClient creates a local vector database directory. If this works, ChromaDB is installed correctly.
  • NVIDIAEmbedding calls the NVIDIA API to convert text into a vector. If this works, your API key and the embedding service are connected.
  • NVIDIA LLM sends a prompt and receives a generated response. If this works, the language model service is accessible.
  • Save test_setup.py and run it in your terminal:
python3 test_setup.py

You should see three success messages followed by "All checks passed!" at the end.

Seeing an API error or timeout?

If you see an authentication error, double check that your NVIDIA_API_KEY environment variable is set correctly. Run echo $NVIDIA_API_KEY (macOS) or echo $env:NVIDIA_API_KEY (Windows) to confirm it starts with nvapi-.

If you see a timeout, the NVIDIA API may be momentarily busy. Wait 30 seconds and try again.

Help me troubleshoot my NVIDIA API connection.

  • Clean up the test database folder by running:
rm -rf test_chroma

You can also delete test_setup.py since it was only needed for verification.

✔️ Awesome, I've got everything!

Great. Make sure you have saved both requirements.txt and .env in your research-scout folder.

ⓧ I'd like to double check the full code

Here is what each file should look like at this point:

NVIDIA_API_KEY=nvapi-your-key-here
llama-index-core
llama-index-llms-nvidia
llama-index-embeddings-nvidia
llama-index-postprocessor-nvidia-rerank
llama-index-retrievers-bm25
llama-index-vector-stores-chroma
chromadb
requests
nest-asyncio

Your environment is locked and loaded. Next up, you will connect to Semantic Scholar and fetch real research papers for your agent to analyze.

Fetch Papers from Semantic Scholar

Your environment is set up with all the packages you need and your NVIDIA API key is stored safely.

Now it is time to write the core of your research agent: a paper fetcher that pulls the latest academic papers from Semantic Scholar. You will use their free Academic Graph API to search by topic and retrieve paper metadata like titles, abstracts, and citation counts.

In this step, get ready to:
  • Build a paper fetcher that queries the Semantic Scholar bulk search API.
  • Convert raw paper data into LlamaIndex Document objects for indexing.
Build the paper fetcher module

The Semantic Scholar Academic Graph API provides a bulk search endpoint that returns up to 1,000 papers per call. It is free and requires no authentication for basic usage.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type VS Code and press Enter to open it.
  • Click File in the top menu bar.
  • Click Open Folder.
  • Select the research-scout folder on your Desktop, then click Open.
  • Create a new file called scout.py by clicking the new file icon in the Explorer sidebar.

First, you need to set up the imports and API key validation. This ensures the script can access all the libraries and confirms your NVIDIA key is ready before doing any work.

  • Add the following imports and configuration to scout.py:
import os
import requests
import chromadb
import nest_asyncio

from llama_index.core import (
    VectorStoreIndex,
    StorageContext,
    Settings,
    Document,
)
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.storage.docstore import SimpleDocumentStore
from llama_index.core.retrievers import QueryFusionRetriever
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.embeddings.nvidia import NVIDIAEmbedding
from llama_index.llms.nvidia import NVIDIA
from llama_index.postprocessor.nvidia_rerank import NVIDIARerank
from llama_index.retrievers.bm25 import BM25Retriever
from llama_index.vector_stores.chroma import ChromaVectorStore

nest_asyncio.apply()

# --- Configuration ---
NVIDIA_API_KEY = os.environ.get("NVIDIA_API_KEY", "")
if not NVIDIA_API_KEY.startswith("nvapi-"):
    print("Error: Set NVIDIA_API_KEY environment variable (starts with nvapi-)")
    print("Get your free key at https://build.nvidia.com")
    exit(1)

os.environ["NVIDIA_API_KEY"] = NVIDIA_API_KEY

What does this code do?

The import block pulls in every library you will need across the full project. You import them all now so you do not need to revisit this section later.

The nest_asyncio.apply() call patches Python's event loop so LlamaIndex's async retriever can run inside a regular script. Without it, you would get an event loop conflict error.

The configuration block reads your NVIDIA_API_KEY from the environment and exits early with a helpful message if it is missing or malformed.

  • Save scout.py.

Seeing an import error?

Make sure your virtual environment is activated. You should see (venv) at the start of your terminal prompt. If not, run source venv/bin/activate (macOS/Linux) or venv\Scripts\activate (Windows) inside the research-scout folder.

If you see a specific package not found, re-run the pip install command from Step 1 to install any missing dependencies.

Help me debug an import error in scout.py

Now add the function that actually calls the Semantic Scholar API. This function accepts a research topic, queries the bulk search endpoint, and returns the most recent papers that have abstracts available.

  • Add the fetch_papers function below the configuration block in scout.py:
# --- Paper Fetching ---
def fetch_papers(topic, count=20):
    """Fetch recent papers from Semantic Scholar on a given topic."""
    url = "https://api.semanticscholar.org/graph/v1/paper/search/bulk"
    params = {
        "query": topic,
        "fields": "title,abstract,year,citationCount,authors,publicationDate,paperId",
        "sort": "publicationDate:desc",
        "publicationDateOrYear": "2024-01-01:",
    }

    print(f"\nSearching Semantic Scholar for: '{topic}'...")
    response = requests.get(url, params=params, timeout=30)
    response.raise_for_status()

    data = response.json()
    papers = data.get("data", [])

    # Filter papers with abstracts and limit to count
    valid_papers = [p for p in papers if p.get("abstract")][:count]
    print(f"Found {len(valid_papers)} papers with abstracts.")
    return valid_papers

What does this function do?

  • The url points to Semantic Scholar's paper bulk search endpoint, which is designed for retrieving many papers efficiently.
  • The fields parameter tells the API exactly which data to return for each paper. Without it, you would only get paper IDs and titles.
  • The sort parameter orders results by publication date (newest first), so you always get the latest research.
  • The publicationDateOrYear filter restricts results to papers published from 2024 onwards, keeping your research current.
  • The final filter removes papers without abstracts (some papers in the database lack them) and caps the result at the requested count.
  • Save scout.py.

Why no API key for Semantic Scholar?

Semantic Scholar's endpoints are publicly available without authentication. Unauthenticated requests share a pool of 1,000 requests per second across all users. For single-user development work like this project, you will never hit that limit.

Convert papers to LlamaIndex Documents

Raw API responses are just dictionaries. To use them with LlamaIndex's retrieval pipeline, you need to convert each paper into a Document object where the abstract becomes the searchable text and the metadata (title, authors, year) travels alongside it.

  • Add the papers_to_documents function below fetch_papers in scout.py:
def papers_to_documents(papers):
    """Convert Semantic Scholar papers to LlamaIndex Documents."""
    documents = []
    for paper in papers:
        authors = ", ".join(
            [a.get("name", "Unknown") for a in paper.get("authors", [])[:5]]
        )
        doc = Document(
            text=paper["abstract"],
            metadata={
                "title": paper.get("title", "Untitled"),
                "authors": authors,
                "year": paper.get("year", "Unknown"),
                "citation_count": paper.get("citationCount", 0),
                "publication_date": paper.get("publicationDate", "Unknown"),
                "paper_id": paper.get("paperId", ""),
            },
        )
        documents.append(doc)
    return documents

What does this function do?

  • Each paper's abstract becomes the text field of the Document. This is the content that will be embedded and searched later.
  • Metadata like title, authors, year, and citation count travels with the document. This lets you display source information when the agent answers questions.
  • Author names are capped at 5 per paper and joined into a single comma-separated string for clean display.
  • The paper_id is preserved so you can look up the original paper on Semantic Scholar later.
  • Save scout.py.
  • Open the integrated terminal in VS Code by pressing Ctrl+` (backtick key).
  • Make sure your virtual environment is active (you should see (venv) in the prompt).
  • Export your NVIDIA API key so the script can access it by running this command (replace the placeholder with your actual key from the .env file):
export NVIDIA_API_KEY=nvapi-your-actual-key-here
  • Test the fetcher by running this command:
python3 -c "from scout import fetch_papers, papers_to_documents; papers = fetch_papers('transformer architectures'); docs = papers_to_documents(papers); print('Converted', len(docs), 'papers to documents')"

You should see output showing that Semantic Scholar was searched and approximately 20 papers were converted to documents.

Getting a connection error or empty results?

If you see a connection timeout, check your internet connection. The Semantic Scholar API is publicly accessible but requires network access.

If you get 0 papers, try a different topic string. Some very niche or misspelled topics may not return results. A broad topic like transformer architectures or machine learning should always return papers.

If the script exits immediately with the NVIDIA key error, make sure you ran the export command in the same terminal session before running the test.

Help me fix a connection or empty results issue with my paper fetcher

✔️ Awesome, I've got everything!

Great. Double check you have saved scout.py before moving on.

ⓧ I'd like to double check the full code

Here is the complete scout.py so far:

import os
import requests
import chromadb
import nest_asyncio

from llama_index.core import (
    VectorStoreIndex,
    StorageContext,
    Settings,
    Document,
)
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.storage.docstore import SimpleDocumentStore
from llama_index.core.retrievers import QueryFusionRetriever
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.embeddings.nvidia import NVIDIAEmbedding
from llama_index.llms.nvidia import NVIDIA
from llama_index.postprocessor.nvidia_rerank import NVIDIARerank
from llama_index.retrievers.bm25 import BM25Retriever
from llama_index.vector_stores.chroma import ChromaVectorStore

nest_asyncio.apply()

# --- Configuration ---
NVIDIA_API_KEY = os.environ.get("NVIDIA_API_KEY", "")
if not NVIDIA_API_KEY.startswith("nvapi-"):
    print("Error: Set NVIDIA_API_KEY environment variable (starts with nvapi-)")
    print("Get your free key at https://build.nvidia.com")
    exit(1)

os.environ["NVIDIA_API_KEY"] = NVIDIA_API_KEY

# --- Paper Fetching ---
def fetch_papers(topic, count=20):
    """Fetch recent papers from Semantic Scholar on a given topic."""
    url = "https://api.semanticscholar.org/graph/v1/paper/search/bulk"
    params = {
        "query": topic,
        "fields": "title,abstract,year,citationCount,authors,publicationDate,paperId",
        "sort": "publicationDate:desc",
        "publicationDateOrYear": "2024-01-01:",
    }

    print(f"\nSearching Semantic Scholar for: '{topic}'...")
    response = requests.get(url, params=params, timeout=30)
    response.raise_for_status()

    data = response.json()
    papers = data.get("data", [])

    # Filter papers with abstracts and limit to count
    valid_papers = [p for p in papers if p.get("abstract")][:count]
    print(f"Found {len(valid_papers)} papers with abstracts.")
    return valid_papers


def papers_to_documents(papers):
    """Convert Semantic Scholar papers to LlamaIndex Documents."""
    documents = []
    for paper in papers:
        authors = ", ".join(
            [a.get("name", "Unknown") for a in paper.get("authors", [])[:5]]
        )
        doc = Document(
            text=paper["abstract"],
            metadata={
                "title": paper.get("title", "Untitled"),
                "authors": authors,
                "year": paper.get("year", "Unknown"),
                "citation_count": paper.get("citationCount", 0),
                "publication_date": paper.get("publicationDate", "Unknown"),
                "paper_id": paper.get("paperId", ""),
            },
        )
        documents.append(doc)
    return documents

Your paper fetcher is working. Next up, you will build a hybrid retrieval index that combines dense vector embeddings with BM25 keyword search for precise, multi-strategy retrieval.

Build the Hybrid Retrieval Index

Your paper fetcher grabs documents from Semantic Scholar, but to answer questions about them you need a search system that can find the right papers fast.

A single search strategy always has blind spots. Vector embeddings capture meaning but miss exact keywords like author names or acronyms. Keyword search catches those but misses semantic similarity. In this step, you'll build a hybrid index that uses both strategies together for significantly better retrieval accuracy.

In this step, get ready to:
  • Configure NVIDIA embeddings and chunk documents into searchable nodes.
  • Create a dual-store index with ChromaDB for vectors and a document store for BM25 keyword search.
  • Test hybrid retrieval to confirm both search paths return scored results.
Configure NVIDIA embeddings and chunk the documents

Before you can search paper abstracts, you need to convert them into a format optimized for retrieval. This means splitting each abstract into chunks and generating a dense vector embedding for each chunk.

  • In scout.py, scroll to the bottom of your file (below the papers_to_documents function).
  • Add the first part of a new build_hybrid_index function by copying this code:
# --- Index Building ---
def build_hybrid_index(documents):
    """Build a hybrid index with ChromaDB vectors + BM25 docstore."""
    print("\nBuilding hybrid index...")

    # Configure embedding model
    Settings.embed_model = NVIDIAEmbedding(model="NV-Embed-QA", truncate="END")
    Settings.text_splitter = SentenceSplitter(chunk_size=500)

    # Parse documents into nodes
    splitter = SentenceSplitter(chunk_size=500)
    nodes = splitter.get_nodes_from_documents(documents)
    print(f"Created {len(nodes)} nodes from {len(documents)} papers.")

What does this code do?

Settings.embed_model tells LlamaIndex to use NVIDIA's NV-Embed-QA model for generating vector embeddings. The truncate="END" parameter ensures that any text exceeding the model's 512-token limit is trimmed from the end rather than throwing an error.

The SentenceSplitter(chunk_size=500) breaks each paper's abstract into chunks of up to 500 tokens. This keeps chunks within the embedding model's 512-token window while maximizing the amount of context in each chunk.

You won't see a change yet. The next chunk completes the function with the storage setup.

Create the dual-store index

Now you need two separate stores: a vector store for semantic search and a document store for keyword-based BM25 retrieval. Together they form the foundation of hybrid search.

  • Still inside the build_hybrid_index function (directly below the code you just added), paste the following:
    # Set up ChromaDB vector store
    db = chromadb.PersistentClient(path="./chroma_db")
    chroma_collection = db.get_or_create_collection("research_papers")
    vector_store = ChromaVectorStore(chroma_collection=chroma_collection)

    # Set up docstore for BM25
    docstore = SimpleDocumentStore()
    docstore.add_documents(nodes)

    # Create storage context with both stores
    storage_context = StorageContext.from_defaults(
        docstore=docstore, vector_store=vector_store
    )

    # Build the vector index
    index = VectorStoreIndex(nodes=nodes, storage_context=storage_context)
    print("Hybrid index built successfully.")
    return index

What does this code do?

  • chromadb.PersistentClient(path="./chroma_db") creates a local ChromaDB database that persists to disk. Your embeddings survive between script runs.
  • ChromaVectorStore wraps the Chroma collection so LlamaIndex can use it as a vector store for dense semantic retrieval.
  • SimpleDocumentStore holds the raw text nodes for BM25 keyword matching. BM25 scores documents by term frequency, catching exact matches that vector search might miss.
  • StorageContext.from_defaults bundles both stores together so a single VectorStoreIndex can access both retrieval strategies.

Why two stores instead of one?

Vector embeddings are great at finding semantically similar content ("neural network architectures" matches "deep learning models"). But they often miss exact keyword matches like paper titles, author names, and technical acronyms ("GNN", "BERT").

BM25 catches those exact matches. By running both and fusing the results, you get the best of both worlds.

  • Save scout.py.

Seeing an indentation error?

Make sure the second code block is indented inside the build_hybrid_index function. Every line should be indented with 4 spaces relative to the def line.

help me fix an indentation error in my build_hybrid_index function

✔️ Awesome, I've got everything!

Great. Double check you've saved scout.py before moving on.

ⓧ I'd like to double check the full code

import os
import requests
import chromadb
import nest_asyncio

from llama_index.core import (
    VectorStoreIndex,
    StorageContext,
    Settings,
    Document,
)
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.storage.docstore import SimpleDocumentStore
from llama_index.core.retrievers import QueryFusionRetriever
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.embeddings.nvidia import NVIDIAEmbedding
from llama_index.llms.nvidia import NVIDIA
from llama_index.postprocessor.nvidia_rerank import NVIDIARerank
from llama_index.retrievers.bm25 import BM25Retriever
from llama_index.vector_stores.chroma import ChromaVectorStore

nest_asyncio.apply()

# --- Configuration ---
NVIDIA_API_KEY = os.environ.get("NVIDIA_API_KEY", "")
if not NVIDIA_API_KEY.startswith("nvapi-"):
    print("Error: Set NVIDIA_API_KEY environment variable (starts with nvapi-)")
    print("Get your free key at https://build.nvidia.com")
    exit(1)

os.environ["NVIDIA_API_KEY"] = NVIDIA_API_KEY

# --- Paper Fetching ---
def fetch_papers(topic, count=20):
    """Fetch recent papers from Semantic Scholar on a given topic."""
    url = "https://api.semanticscholar.org/graph/v1/paper/search/bulk"
    params = {
        "query": topic,
        "fields": "title,abstract,year,citationCount,authors,publicationDate,paperId",
        "sort": "publicationDate:desc",
        "publicationDateOrYear": "2024-01-01:",
    }

    print(f"\nSearching Semantic Scholar for: '{topic}'...")
    response = requests.get(url, params=params, timeout=30)
    response.raise_for_status()

    data = response.json()
    papers = data.get("data", [])

    valid_papers = [p for p in papers if p.get("abstract")][:count]
    print(f"Found {len(valid_papers)} papers with abstracts.")
    return valid_papers


def papers_to_documents(papers):
    """Convert Semantic Scholar papers to LlamaIndex Documents."""
    documents = []
    for paper in papers:
        authors = ", ".join(
            [a.get("name", "Unknown") for a in paper.get("authors", [])[:5]]
        )
        doc = Document(
            text=paper["abstract"],
            metadata={
                "title": paper.get("title", "Untitled"),
                "authors": authors,
                "year": paper.get("year", "Unknown"),
                "citation_count": paper.get("citationCount", 0),
                "publication_date": paper.get("publicationDate", "Unknown"),
                "paper_id": paper.get("paperId", ""),
            },
        )
        documents.append(doc)
    return documents


# --- Index Building ---
def build_hybrid_index(documents):
    """Build a hybrid index with ChromaDB vectors + BM25 docstore."""
    print("\nBuilding hybrid index...")

    Settings.embed_model = NVIDIAEmbedding(model="NV-Embed-QA", truncate="END")
    Settings.text_splitter = SentenceSplitter(chunk_size=500)

    splitter = SentenceSplitter(chunk_size=500)
    nodes = splitter.get_nodes_from_documents(documents)
    print(f"Created {len(nodes)} nodes from {len(documents)} papers.")

    db = chromadb.PersistentClient(path="./chroma_db")
    chroma_collection = db.get_or_create_collection("research_papers")
    vector_store = ChromaVectorStore(chroma_collection=chroma_collection)

    docstore = SimpleDocumentStore()
    docstore.add_documents(nodes)

    storage_context = StorageContext.from_defaults(
        docstore=docstore, vector_store=vector_store
    )

    index = VectorStoreIndex(nodes=nodes, storage_context=storage_context)
    print("Hybrid index built successfully.")
    return index
Wire up the QueryFusionRetriever for hybrid search

The index is built, but you haven't tested retrieval yet. You need to confirm that both the dense vector path and the sparse BM25 path return scored results when you query them together.

  • Add this temporary test block at the very bottom of scout.py (outside any function) to run a quick verification:
# --- Temporary test (remove after verification) ---
if __name__ == "__main__":
    # Fetch a small batch of test papers
    papers = fetch_papers("transformer architectures", count=5)
    documents = papers_to_documents(papers)

    # Build the hybrid index
    index = build_hybrid_index(documents)

    # Create both retrievers
    vector_retriever = index.as_retriever(similarity_top_k=5)
    bm25_retriever = BM25Retriever.from_defaults(
        docstore=index.docstore, similarity_top_k=5
    )

    # Fuse them with QueryFusionRetriever
    hybrid_retriever = QueryFusionRetriever(
        [vector_retriever, bm25_retriever],
        retriever_weights=[0.6, 0.4],
        similarity_top_k=5,
        num_queries=1,
        mode="relative_score",
        use_async=True,
    )

    # Run a test query
    nodes_with_scores = hybrid_retriever.retrieve("attention mechanisms")
    print("\n--- Hybrid Retrieval Results ---")
    for node in nodes_with_scores:
        print(f"Score: {node.score:.3f} | {node.metadata.get('title', 'Unknown')}")

What does this test do?

This temporary block fetches 5 papers, builds your hybrid index, then creates a QueryFusionRetriever that combines both retrieval strategies. The fusion mode relative_score applies a MinMax scaler to each result set and computes a weighted sum (60% vector, 40% BM25).

Setting num_queries=1 disables query generation (no extra LLM calls). The retriever returns nodes with fused scores from both paths.

  • Save scout.py.
  • Run the test by executing this command in your terminal:
python scout.py
  • You should see output showing papers fetched from Semantic Scholar, the index being built, and then a list of scored results under "Hybrid Retrieval Results". Each result shows a score (between 0 and 1) and a paper title.

Seeing a connection error or empty results?

  • If you see a timeout or connection error, check your internet connection. The Semantic Scholar API requires network access.
  • If you see "Found 0 papers with abstracts", try a different search term. Some niche topics may have fewer results with available abstracts.
  • If you see an NVIDIA API error, confirm your NVIDIA_API_KEY environment variable is set correctly (starts with nvapi-).

help me debug my hybrid retrieval test in scout.py

  • Once you've confirmed the test works, remove the entire temporary test block (everything from # --- Temporary test down to the last print line) from scout.py.
  • Save scout.py.

Why remove the test code?

This test block was just to verify your index works. In the next step, you'll build the permanent query engine with reranking and LLM synthesis. The retriever configuration you just tested (weights, mode, top-k) will be part of that permanent function.

Your hybrid index is working. Next up, you'll add neural reranking and LLM synthesis to turn raw retrieval into intelligent answers about the research landscape.

Add Neural Reranking and LLM Synthesis

Your hybrid index is built and ready to search. Now you need a pipeline that actually uses that index to answer questions intelligently.

A raw retrieval result is just a ranked list of text chunks. To turn those chunks into a coherent, cited answer, you need three more layers: a fusion retriever that merges dense and sparse results, a neural reranker that picks the best passages, and an LLM that synthesizes everything into a human-readable response.

In this step, get ready to:
  • Build a retrieval pipeline combining hybrid search with neural reranking.
  • Configure the LLM for answer synthesis.
  • Create an interactive query loop with special commands for research exploration.
Create the retrieval pipeline with neural reranking

Your index stores data in two forms: dense vectors in ChromaDB and sparse keyword representations in the BM25 docstore. The query engine needs to pull results from both, fuse the scores, rerank the top candidates with a neural model, and then pass the winners to an LLM for synthesis.

  • In scout.py, scroll to the bottom of the file (below the build_hybrid_index function).
  • Add the following create_query_engine function:
# --- Retrieval Pipeline ---
def create_query_engine(index):
    """Create the full hybrid retrieval + reranking query engine."""
    # Dense vector retriever
    vector_retriever = index.as_retriever(similarity_top_k=10)

    # BM25 sparse retriever
    bm25_retriever = BM25Retriever.from_defaults(
        docstore=index.docstore, similarity_top_k=10
    )

    # Hybrid fusion retriever (relative score fusion)
    hybrid_retriever = QueryFusionRetriever(
        [vector_retriever, bm25_retriever],
        retriever_weights=[0.6, 0.4],
        similarity_top_k=10,
        num_queries=1,
        mode="relative_score",
        use_async=True,
    )

    # Neural reranker narrows top-10 to the 5 most relevant
    reranker = NVIDIARerank(top_n=5)

    # LLM for synthesis
    Settings.llm = NVIDIA(model="meta/llama-3.1-8b-instruct")

    # Assemble query engine
    query_engine = RetrieverQueryEngine.from_args(
        retriever=hybrid_retriever,
        node_postprocessors=[reranker],
    )

    return query_engine

What does this code do?

This function builds the full retrieval pipeline in four stages:

  • vector_retriever fetches the top 10 semantically similar chunks from ChromaDB using dense embeddings.
  • bm25_retriever fetches the top 10 keyword-matched chunks from the docstore using BM25 sparse scoring.
  • QueryFusionRetriever merges both result sets using relative score fusion. The weights [0.6, 0.4] give dense embeddings slightly more influence while BM25 catches exact keyword matches like author names and technical acronyms.
  • NVIDIARerank(top_n=5) takes the fused top-10 results and uses a neural model to rerank them down to the 5 most relevant passages. This final reranking step dramatically improves precision.

The RetrieverQueryEngine ties everything together: retrieve, rerank, then pass the top results to the LLM for answer synthesis.

Why relative score fusion instead of reciprocal rank?

The relative_score mode normalizes scores from each retriever to a 0-1 range before combining them with explicit weights. This gives you predictable, tunable control over how much influence each retrieval strategy has.

Reciprocal rank fusion only looks at position in the ranked list, ignoring score magnitude. With relative score fusion, a highly confident vector match gets properly rewarded even if BM25 ranked it lower.

  • Save scout.py.

You won't see visible output yet. The next chunk builds the interactive loop that exercises this pipeline.

Seeing an import error?

If your editor underlines NVIDIARerank or BM25Retriever, confirm your virtual environment is activated. These imports were added at the top of the file in an earlier step.

help me debug an import error with NVIDIARerank or BM25Retriever in my scout.py file

Build the interactive query loop

The query engine can answer questions, but you need a way for the user to interact with it. You will build a command-line loop that accepts free-form questions plus special commands for generating project ideas and research summaries.

  • Below your create_query_engine function, add these two prompt helper functions:
# --- Interactive Loop ---
def generate_ideas_prompt(topic):
    """Build a prompt that asks the LLM for project ideas."""
    return (
        f"Based on the research papers indexed about '{topic}', "
        "suggest 3 concrete project ideas I could build. "
        "For each idea, provide: a title, a one-sentence description, "
        "the key papers it draws from, and a difficulty level (Easy/Medium/Hard). "
        "Ground each idea in specific findings from the papers."
    )


def generate_summary_prompt(topic):
    """Build a prompt that asks the LLM for a structured research overview."""
    return (
        f"Provide a structured research summary about '{topic}' based on the indexed papers. "
        "Include: 1) Key themes and trends (3-5 bullet points), "
        "2) Notable methodologies or approaches, "
        "3) Open questions or research gaps, "
        "4) Top 3 project ideas with difficulty levels."
    )

What do these helpers do?

These functions format structured prompts that guide the LLM to produce useful outputs from the indexed papers. The /ideas command asks for concrete project suggestions grounded in specific papers. The /summary command asks for a high-level overview of themes, methodologies, and research gaps.

Keeping prompts in helper functions makes them easy to tune later without touching the loop logic.

  • Below the prompt helpers, add the run_interactive_loop function:
def run_interactive_loop(query_engine, topic):
    """Run the interactive query loop."""
    print("\n" + "=" * 60)
    print("Research Paper Scout - Ready!")
    print("=" * 60)
    print(f"Topic: {topic}")
    print("\nCommands:")
    print("  /ideas    - Generate project ideas from the research")
    print("  /summary  - Get a structured research overview")
    print("  /quit     - Exit the agent")
    print("  Or ask any question about the research papers.")
    print("=" * 60)

    while True:
        # Wait for user input
        user_input = input("\nYou: ").strip()
        if not user_input:
            continue

        # Handle special commands
        if user_input.lower() == "/quit":
            print("Goodbye!")
            break
        elif user_input.lower() == "/ideas":
            query = generate_ideas_prompt(topic)
        elif user_input.lower() == "/summary":
            query = generate_summary_prompt(topic)
        else:
            query = user_input

        # Run the full pipeline: retrieve -> rerank -> synthesize
        print("\nSearching and synthesizing...")
        response = query_engine.query(query)
        print(f"\nScout: {response.response}")

        # Show source papers for transparency
        if response.source_nodes:
            print("\n--- Sources ---")
            for i, node in enumerate(response.source_nodes[:3], 1):
                title = node.metadata.get("title", "Unknown")
                year = node.metadata.get("year", "?")
                score = f"{node.score:.3f}" if node.score else "N/A"
                print(f"  {i}. [{year}] {title} (relevance: {score})")

What does this code do?

The loop prints a welcome banner with available commands, then waits for input. Each iteration:

  • /ideas and /summary swap the user's command for a structured prompt that guides the LLM's output format.
  • Free-form text passes directly to the query engine as-is.
  • query_engine.query() triggers the full pipeline: hybrid retrieval, score fusion, neural reranking, then LLM synthesis.
  • After each answer, the loop prints up to 3 source papers with their title, year, and relevance score so the user can verify the answer is grounded in real research.
  • Save scout.py.

You won't be able to run the full agent yet because there is no main() entry point. You will wire everything together in the next step.

Seeing a syntax error on the f-strings?

Make sure you are using Python 3.10+ in your virtual environment. The f-string formatting in the prompt helpers requires at least Python 3.6, but older system Python versions can shadow your virtual environment.

help me fix a syntax error in my interactive loop or prompt helper functions

✔️ Awesome, I've got everything!

Great. Double check you have saved scout.py with all three new functions: create_query_engine, generate_ideas_prompt/generate_summary_prompt, and run_interactive_loop.

ⓧ I'd like to double check the full code

Here is the complete scout.py as it should look at the end of this step:

import os
import requests
import chromadb
import nest_asyncio

from llama_index.core import (
    VectorStoreIndex,
    StorageContext,
    Settings,
    Document,
)
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.storage.docstore import SimpleDocumentStore
from llama_index.core.retrievers import QueryFusionRetriever
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.embeddings.nvidia import NVIDIAEmbedding
from llama_index.llms.nvidia import NVIDIA
from llama_index.postprocessor.nvidia_rerank import NVIDIARerank
from llama_index.retrievers.bm25 import BM25Retriever
from llama_index.vector_stores.chroma import ChromaVectorStore

nest_asyncio.apply()

# --- Configuration ---
NVIDIA_API_KEY = os.environ.get("NVIDIA_API_KEY", "")
if not NVIDIA_API_KEY.startswith("nvapi-"):
    print("Error: Set NVIDIA_API_KEY environment variable (starts with nvapi-)")
    print("Get your free key at https://build.nvidia.com")
    exit(1)

os.environ["NVIDIA_API_KEY"] = NVIDIA_API_KEY

# --- Paper Fetching ---
def fetch_papers(topic, count=20):
    url = "https://api.semanticscholar.org/graph/v1/paper/search/bulk"
    params = {
        "query": topic,
        "fields": "title,abstract,year,citationCount,authors,publicationDate,paperId",
        "sort": "publicationDate:desc",
        "publicationDateOrYear": "2024-01-01:",
    }
    print(f"\nSearching Semantic Scholar for: '{topic}'...")
    response = requests.get(url, params=params, timeout=30)
    response.raise_for_status()
    data = response.json()
    papers = data.get("data", [])
    valid_papers = [p for p in papers if p.get("abstract")][:count]
    print(f"Found {len(valid_papers)} papers with abstracts.")
    return valid_papers


def papers_to_documents(papers):
    documents = []
    for paper in papers:
        authors = ", ".join(
            [a.get("name", "Unknown") for a in paper.get("authors", [])[:5]]
        )
        doc = Document(
            text=paper["abstract"],
            metadata={
                "title": paper.get("title", "Untitled"),
                "authors": authors,
                "year": paper.get("year", "Unknown"),
                "citation_count": paper.get("citationCount", 0),
                "publication_date": paper.get("publicationDate", "Unknown"),
                "paper_id": paper.get("paperId", ""),
            },
        )
        documents.append(doc)
    return documents


# --- Index Building ---
def build_hybrid_index(documents):
    print("\nBuilding hybrid index...")
    Settings.embed_model = NVIDIAEmbedding(model="NV-Embed-QA", truncate="END")
    Settings.text_splitter = SentenceSplitter(chunk_size=500)
    splitter = SentenceSplitter(chunk_size=500)
    nodes = splitter.get_nodes_from_documents(documents)
    print(f"Created {len(nodes)} nodes from {len(documents)} papers.")
    db = chromadb.PersistentClient(path="./chroma_db")
    chroma_collection = db.get_or_create_collection("research_papers")
    vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
    docstore = SimpleDocumentStore()
    docstore.add_documents(nodes)
    storage_context = StorageContext.from_defaults(
        docstore=docstore, vector_store=vector_store
    )
    index = VectorStoreIndex(nodes=nodes, storage_context=storage_context)
    print("Hybrid index built successfully.")
    return index


# --- Retrieval Pipeline ---
def create_query_engine(index):
    vector_retriever = index.as_retriever(similarity_top_k=10)
    bm25_retriever = BM25Retriever.from_defaults(
        docstore=index.docstore, similarity_top_k=10
    )
    hybrid_retriever = QueryFusionRetriever(
        [vector_retriever, bm25_retriever],
        retriever_weights=[0.6, 0.4],
        similarity_top_k=10,
        num_queries=1,
        mode="relative_score",
        use_async=True,
    )
    reranker = NVIDIARerank(top_n=5)
    Settings.llm = NVIDIA(model="meta/llama-3.1-8b-instruct")
    query_engine = RetrieverQueryEngine.from_args(
        retriever=hybrid_retriever,
        node_postprocessors=[reranker],
    )
    return query_engine


# --- Interactive Loop ---
def generate_ideas_prompt(topic):
    return (
        f"Based on the research papers indexed about '{topic}', "
        "suggest 3 concrete project ideas I could build. "
        "For each idea, provide: a title, a one-sentence description, "
        "the key papers it draws from, and a difficulty level (Easy/Medium/Hard). "
        "Ground each idea in specific findings from the papers."
    )


def generate_summary_prompt(topic):
    return (
        f"Provide a structured research summary about '{topic}' based on the indexed papers. "
        "Include: 1) Key themes and trends (3-5 bullet points), "
        "2) Notable methodologies or approaches, "
        "3) Open questions or research gaps, "
        "4) Top 3 project ideas with difficulty levels."
    )


def run_interactive_loop(query_engine, topic):
    print("\n" + "=" * 60)
    print("Research Paper Scout - Ready!")
    print("=" * 60)
    print(f"Topic: {topic}")
    print("\nCommands:")
    print("  /ideas    - Generate project ideas from the research")
    print("  /summary  - Get a structured research overview")
    print("  /quit     - Exit the agent")
    print("  Or ask any question about the research papers.")
    print("=" * 60)

    while True:
        user_input = input("\nYou: ").strip()
        if not user_input:
            continue
        if user_input.lower() == "/quit":
            print("Goodbye!")
            break
        elif user_input.lower() == "/ideas":
            query = generate_ideas_prompt(topic)
        elif user_input.lower() == "/summary":
            query = generate_summary_prompt(topic)
        else:
            query = user_input

        print("\nSearching and synthesizing...")
        response = query_engine.query(query)
        print(f"\nScout: {response.response}")

        if response.source_nodes:
            print("\n--- Sources ---")
            for i, node in enumerate(response.source_nodes[:3], 1):
                title = node.metadata.get("title", "Unknown")
                year = node.metadata.get("year", "?")
                score = f"{node.score:.3f}" if node.score else "N/A"
                print(f"  {i}. [{year}] {title} (relevance: {score})")

Your retrieval pipeline and interactive loop are ready. In the next step, you will wire everything into a single entry point that fetches papers, builds the index, and launches the query loop as one seamless workflow.

Assemble the Complete Agent

Your hybrid retrieval pipeline is built, the neural reranker is wired up, and the interactive loop is ready to go. All that's left is an entry point that ties every piece together into a single runnable script.

In this step, you'll add a main() function that orchestrates the full workflow. It prompts for a topic, fetches papers, builds the index, and drops the user into the query loop.

In this step, get ready to:
  • Build the main entry point that orchestrates paper fetching, indexing, and querying.
  • Run the full agent end-to-end and verify the /summary and /ideas commands.
Build the main entry point

Every Python script needs a clear starting point. The main() function will call each piece you've already built in sequence, with error handling at critical points like network failures and empty search results.

  • In scout.py, scroll to the very bottom of the file (below the run_interactive_loop function) and add this code:
# --- Main ---
def main():
    print("=" * 60)
    print("  Research Paper Scout")
    print("  Advanced Hybrid RAG with BM25 + Reranking")
    print("=" * 60)

    topic = input("\nEnter a research topic: ").strip()
    if not topic:
        print("No topic provided. Exiting.")
        return

    # Fetch papers from Semantic Scholar
    papers = fetch_papers(topic)
    if not papers:
        print("No papers found. Try a different topic.")
        return

    # Convert raw paper data to LlamaIndex Documents
    documents = papers_to_documents(papers)

    # Build the hybrid index (ChromaDB + BM25)
    index = build_hybrid_index(documents)

    # Create the full retrieval + reranking query engine
    query_engine = create_query_engine(index)

    # Start the interactive query loop
    run_interactive_loop(query_engine, topic)


if __name__ == "__main__":
    main()

What does this code do?

The main() function is the orchestrator. It calls each function you built in the previous steps in the right order.

  • It prints a welcome banner, then prompts the user for a research topic.
  • Early returns handle error cases: if no topic is entered or no papers are found, the script exits gracefully instead of crashing.
  • if __name__ == "__main__": ensures main() only runs when you execute the script directly, not when it's imported.
  • Save scout.py.

Seeing a syntax error when you save?

  • Make sure the main() function is at the same indentation level (no indent) as your other function definitions.
  • Check that if __name__ == "__main__": is also at the top level (zero indentation), not inside another function.
  • help me fix a syntax error at the bottom of my scout.py where I added main()
Run the complete agent end-to-end

Your agent is now complete. Time to run the full pipeline and verify that every piece works together: paper fetching, hybrid indexing, and the interactive query commands.

  • Make sure your NVIDIA_API_KEY environment variable is set by running this in your terminal:
export NVIDIA_API_KEY=nvapi-your-key-here
  • Replace nvapi-your-key-here with your actual NVIDIA API key from the .env file.
  • Run the agent by executing this command:
python scout.py
  • When prompted, enter a research topic like retrieval augmented generation.
  • Wait for the agent to fetch papers and build the index. You should see progress messages like "Found 20 papers with abstracts" and "Hybrid index built successfully."
  • Once you see the "Research Paper Scout - Ready!" banner, type /summary and press Enter.

What does /summary do?

The /summary command triggers the generate_summary_prompt() function, which asks the LLM to produce a structured research overview. It covers key themes and trends, notable methodologies, open research gaps, and the top 3 project ideas with difficulty levels.

This gives you an instant high-level map of the research landscape without needing to ask individual questions first.

  • After the summary appears, type /ideas and press Enter to generate project suggestions grounded in the papers.
  • Try asking a natural question like "What are the main approaches in this field?" to test free-form queries.
  • Type /quit to exit when you're done exploring.

You should see coherent answers that cite specific papers by title and year in the "--- Sources ---" section below each response.

Agent crashes or returns empty results?

  • If you see "Error: Set NVIDIA_API_KEY environment variable", make sure you exported the key in your current terminal session. The .env file is not auto-loaded. You need the export command above.
  • If paper fetching fails with a network error, check your internet connection. The Semantic Scholar API is public but shared across users.
  • If you get zero papers, try a broader topic. Very niche queries may not return enough results with abstracts available.
  • If the LLM response is empty or errors out, your NVIDIA API key may have hit rate limits. Wait a minute and try again.
  • help me debug why my Research Paper Scout agent is failing

✔️ Awesome, I've got everything!

Great. Make sure you've saved scout.py and verified that the agent runs end-to-end with all three commands working.

ⓧ I'd like to double check the full code

Here is the complete scout.py file. Compare your code against this:

import os
import requests
import chromadb
import nest_asyncio

from llama_index.core import (
    VectorStoreIndex,
    StorageContext,
    Settings,
    Document,
)
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.storage.docstore import SimpleDocumentStore
from llama_index.core.retrievers import QueryFusionRetriever
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.embeddings.nvidia import NVIDIAEmbedding
from llama_index.llms.nvidia import NVIDIA
from llama_index.postprocessor.nvidia_rerank import NVIDIARerank
from llama_index.retrievers.bm25 import BM25Retriever
from llama_index.vector_stores.chroma import ChromaVectorStore

nest_asyncio.apply()

# --- Configuration ---
NVIDIA_API_KEY = os.environ.get("NVIDIA_API_KEY", "")
if not NVIDIA_API_KEY.startswith("nvapi-"):
    print("Error: Set NVIDIA_API_KEY environment variable (starts with nvapi-)")
    print("Get your free key at https://build.nvidia.com")
    exit(1)

os.environ["NVIDIA_API_KEY"] = NVIDIA_API_KEY

# --- Paper Fetching ---
def fetch_papers(topic, count=20):
    """Fetch recent papers from Semantic Scholar on a given topic."""
    url = "https://api.semanticscholar.org/graph/v1/paper/search/bulk"
    params = {
        "query": topic,
        "fields": "title,abstract,year,citationCount,authors,publicationDate,paperId",
        "sort": "publicationDate:desc",
        "publicationDateOrYear": "2024-01-01:",
    }

    print(f"\nSearching Semantic Scholar for: '{topic}'...")
    response = requests.get(url, params=params, timeout=30)
    response.raise_for_status()

    data = response.json()
    papers = data.get("data", [])

    valid_papers = [p for p in papers if p.get("abstract")][:count]
    print(f"Found {len(valid_papers)} papers with abstracts.")
    return valid_papers


def papers_to_documents(papers):
    """Convert Semantic Scholar papers to LlamaIndex Documents."""
    documents = []
    for paper in papers:
        authors = ", ".join(
            [a.get("name", "Unknown") for a in paper.get("authors", [])[:5]]
        )
        doc = Document(
            text=paper["abstract"],
            metadata={
                "title": paper.get("title", "Untitled"),
                "authors": authors,
                "year": paper.get("year", "Unknown"),
                "citation_count": paper.get("citationCount", 0),
                "publication_date": paper.get("publicationDate", "Unknown"),
                "paper_id": paper.get("paperId", ""),
            },
        )
        documents.append(doc)
    return documents


# --- Index Building ---
def build_hybrid_index(documents):
    """Build a hybrid index with ChromaDB vectors + BM25 docstore."""
    print("\nBuilding hybrid index...")

    Settings.embed_model = NVIDIAEmbedding(model="NV-Embed-QA", truncate="END")
    Settings.text_splitter = SentenceSplitter(chunk_size=500)

    splitter = SentenceSplitter(chunk_size=500)
    nodes = splitter.get_nodes_from_documents(documents)
    print(f"Created {len(nodes)} nodes from {len(documents)} papers.")

    db = chromadb.PersistentClient(path="./chroma_db")
    chroma_collection = db.get_or_create_collection("research_papers")
    vector_store = ChromaVectorStore(chroma_collection=chroma_collection)

    docstore = SimpleDocumentStore()
    docstore.add_documents(nodes)

    storage_context = StorageContext.from_defaults(
        docstore=docstore, vector_store=vector_store
    )

    index = VectorStoreIndex(nodes=nodes, storage_context=storage_context)
    print("Hybrid index built successfully.")
    return index


# --- Retrieval Pipeline ---
def create_query_engine(index):
    """Create the full hybrid retrieval + reranking query engine."""
    vector_retriever = index.as_retriever(similarity_top_k=10)

    bm25_retriever = BM25Retriever.from_defaults(
        docstore=index.docstore, similarity_top_k=10
    )

    hybrid_retriever = QueryFusionRetriever(
        [vector_retriever, bm25_retriever],
        retriever_weights=[0.6, 0.4],
        similarity_top_k=10,
        num_queries=1,
        mode="relative_score",
        use_async=True,
    )

    reranker = NVIDIARerank(top_n=5)

    Settings.llm = NVIDIA(model="meta/llama-3.1-8b-instruct")

    query_engine = RetrieverQueryEngine.from_args(
        retriever=hybrid_retriever,
        node_postprocessors=[reranker],
    )

    return query_engine


# --- Interactive Loop ---
def generate_ideas_prompt(topic):
    return (
        f"Based on the research papers indexed about '{topic}', "
        "suggest 3 concrete project ideas I could build. "
        "For each idea, provide: a title, a one-sentence description, "
        "the key papers it draws from, and a difficulty level (Easy/Medium/Hard). "
        "Ground each idea in specific findings from the papers."
    )


def generate_summary_prompt(topic):
    return (
        f"Provide a structured research summary about '{topic}' based on the indexed papers. "
        "Include: 1) Key themes and trends (3-5 bullet points), "
        "2) Notable methodologies or approaches, "
        "3) Open questions or research gaps, "
        "4) Top 3 project ideas with difficulty levels."
    )


def run_interactive_loop(query_engine, topic):
    """Run the interactive query loop."""
    print("\n" + "=" * 60)
    print("Research Paper Scout - Ready!")
    print("=" * 60)
    print(f"Topic: {topic}")
    print("\nCommands:")
    print("  /ideas    - Generate project ideas from the research")
    print("  /summary  - Get a structured research overview")
    print("  /quit     - Exit the agent")
    print("  Or ask any question about the research papers.")
    print("=" * 60)

    while True:
        user_input = input("\nYou: ").strip()
        if not user_input:
            continue

        if user_input.lower() == "/quit":
            print("Goodbye!")
            break
        elif user_input.lower() == "/ideas":
            query = generate_ideas_prompt(topic)
        elif user_input.lower() == "/summary":
            query = generate_summary_prompt(topic)
        else:
            query = user_input

        print("\nSearching and synthesizing...")
        response = query_engine.query(query)
        print(f"\nScout: {response.response}")

        if response.source_nodes:
            print("\n--- Sources ---")
            for i, node in enumerate(response.source_nodes[:3], 1):
                title = node.metadata.get("title", "Unknown")
                year = node.metadata.get("year", "?")
                score = f"{node.score:.3f}" if node.score else "N/A"
                print(f"  {i}. [{year}] {title} (relevance: {score})")


# --- Main ---
def main():
    print("=" * 60)
    print("  Research Paper Scout")
    print("  Advanced Hybrid RAG with BM25 + Reranking")
    print("=" * 60)

    topic = input("\nEnter a research topic: ").strip()
    if not topic:
        print("No topic provided. Exiting.")
        return

    papers = fetch_papers(topic)
    if not papers:
        print("No papers found. Try a different topic.")
        return

    documents = papers_to_documents(papers)

    index = build_hybrid_index(documents)

    query_engine = create_query_engine(index)

    run_interactive_loop(query_engine, topic)


if __name__ == "__main__":
    main()

Secret mission

Citation Graph Explorer

Your Research Paper Scout finds papers on a topic, but what about the papers THOSE papers reference? Academic research is a web of citations. In this secret mission, you'll add a /deep-dive command that follows citation chains to discover influential papers your initial search missed, then dynamically expands your index with what it finds.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally with free-tier API access, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • research-scout/ project directory (contains scout.py, requirements.txt, .env, virtual environment, and ChromaDB data).
  • chroma_db/ directory inside the project (persisted vector database on disk).
  • .env file with your NVIDIA API key.
  • NVIDIA API Catalog free-tier usage (no charges incurred).

Keep everything running

No action needed. Choose this if you plan to keep exploring research topics or want to rerun the agent on a different query.

  • Your chroma_db/ directory persists between runs, so re-launching scout.py with the same topic will reuse indexed data.
  • Your NVIDIA API key remains in .env and stays within the free tier.
  • The virtual environment stays on disk inside venv/.

Pause - I'll come back to this later

Shut down running processes and free up terminal resources while keeping all files intact.

  • If the agent is running, type /quit to exit gracefully.
  • Deactivate your virtual environment by running:
deactivate
  • When you return, reactivate the virtual environment by running:
cd ~/Desktop/research-scout
source venv/bin/activate
  • All your project files, indexed data, and dependencies remain in place.

Delete - I don't want to use this again

Remove all project resources and start fresh. This deletes all code, the virtual environment, and the ChromaDB vector database.

  • If you are still inside the virtual environment, deactivate it first by running:
deactivate
  • Delete the entire project directory by running:
rm -rf ~/Desktop/research-scout

On Windows, run this instead:

rmdir /s /q %USERPROFILE%\Desktop\research-scout
  • Your NVIDIA API Catalog account is free and does not require deletion. You can reuse the key for future projects.

That's a Wrap!

That's a Wrap!

Nice work! You just built a production-quality research intelligence agent that combines multiple advanced retrieval strategies into a single powerful pipeline.

You've learned how to:

  • Fetch and index academic papers from the Semantic Scholar Academic Graph API, then build a hybrid retrieval system combining dense vector embeddings (NVIDIA NIM) with BM25 sparse keyword search.
  • Merge results from multiple retrieval strategies using QueryFusionRetriever with Relative Score Fusion, then refine them with a neural reranker to surface the most relevant passages.
  • Create an interactive research agent that synthesizes insights from cutting-edge papers and generates grounded project ideas using LLM synthesis powered by NVIDIA NIM.
  • Secret Mission: Extended the agent with a citation graph explorer that follows reference and citation chains for deeper research discovery.

Ready to quiz yourself?