Multi-Modal RAG API with FastAPI

Build a RAG API that retrieves answers from text, tables, and images in PDFs.

Introduction

30 Second Summary

Most real-world documents contain more than just text. They have tables packed with numbers, charts showing trends, and images explaining processes that traditional text extraction completely ignores.

In this project, you will build a multi-modal RAG agent powered by LangGraph that extracts text, tables, and images from PDFs using custom tools, then answers questions across all content types. You'll wire a ChatNVIDIA vision model, local Ollama LLM, and ChromaDB vector store into a FastAPI service.

What You'll Build

You'll ask your agent a question about a revenue chart buried on page three of a PDF, watch it decide to search the vector store, retrieve the vision model's description of that chart, and generate a grounded, accurate answer.

By the end of this project, you'll have:

  • A LangGraph ReAct agent that decides which tools to call based on the user's question, retrieves multi-modal context, and generates an accurate answer using your local Ollama model.
  • Custom LangChain tools for PDF ingestion, table extraction, image description (via NVIDIA NIM), and vector store retrieval — all callable by the agent.
  • A ChromaDB vector store with content-type metadata that indexes text, tables, and image descriptions in a unified searchable collection.
  • Secret Mission: A /upload endpoint that accepts new PDF files and processes them through the full tool harness on demand.

Are there any prerequisites?

This is Project 3 in the series. You should have completed the CRAG with LangGraph project and the Multi-Agent system with Inngest project first.

You'll need Python 3, uv, Ollama, and an NVIDIA NIM API key already set up on your machine.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters for your RAG skills.

Set Up Your Project and Dependencies

You know what you're building and why multi-modal retrieval matters. Now you need a working project with all the right packages installed before you can build your agent.

In this step, you'll initialize a Python project with uv, install the core libraries for LangGraph agent orchestration, LLM integrations, and vector storage, and create a minimal FastAPI server you can build on in later steps.

In this step, get ready to:
  • Create the project directory and install all dependencies with uv.
  • Create a FastAPI app skeleton with health-check and query endpoints.
  • Pull llama3.2:3b and nomic-embed-text models in Ollama.
Create the project and install dependencies

You need a dedicated folder for this project with all Python packages managed by uv.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and initialize it with uv by running these commands:
mkdir multimodal-rag-api
cd multimodal-rag-api
uv init --no-package

What does uv init --no-package do?

The --no-package flag creates a flat project with a pyproject.toml for dependency management, without generating a src/ directory or build system. This keeps the project structure simple so you can create your own app/ package layout.

  • Install the project dependencies by running:
uv add langchain langchain-ollama langchain-nvidia-ai-endpoints langchain-chroma langgraph pymupdf4llm "fastapi[standard]" python-dotenv langchain-text-splitters

You'll see uv resolving and installing packages. This may take a minute.

What are these packages?

  • langchain is the core framework for building LLM-powered applications with chains, tools, and prompts.
  • langchain-ollama provides ChatOllama for chat completions and OllamaEmbeddings for vector embeddings, both talking to your local Ollama server.
  • langchain-nvidia-ai-endpoints provides ChatNVIDIA for calling NVIDIA NIM vision models with multimodal (text + image) inputs.
  • langchain-chroma integrates ChromaDB as a LangChain-compatible vector store for storing and querying document embeddings.
  • langgraph lets you build agents that reason about which tools to call, looping until the task is complete.
  • pymupdf4llm extracts text and structure from PDF files.
  • langchain-text-splitters provides smart text chunking that preserves paragraph and sentence boundaries.
  • fastapi[standard] is the web framework with uvicorn included for serving your API.
  • python-dotenv loads environment variables from a .env file.
  • Create the app package directory and the project files by running:
mkdir app
touch app/__init__.py
touch .env
  • Confirm your project structure by running:
ls

You should see app, .env, pyproject.toml, and other uv files in the output.

  • Now add your NVIDIA NIM API key to the .env file. Open .env in a text editor and paste the following, replacing your-api-key-here with your actual NVIDIA NIM API key:
NVIDIA_API_KEY=your-api-key-here

Don't have your NVIDIA NIM API key?

You generated this key in a previous project. Sign into the NVIDIA Developer Portal and find your existing key under your API credentials. The free tier covers all prototyping usage for this project.

help me find my NVIDIA NIM API key

Create the FastAPI app skeleton

Your API needs a starting point. You'll create a minimal FastAPI app with a health-check endpoint and a placeholder for the query endpoint you'll build out later.

  • Open VS Code by pressing Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Visual Studio Code and press Enter to open it.
  • Click File in the top menu bar.
  • Click Open Folder.
  • Select the multimodal-rag-api folder on your Desktop, then click Open.
  • Open app/main.py in the editor. Create this file if it does not exist yet by right-clicking the app folder in the Explorer sidebar and selecting New File.
  • Paste the following code into app/main.py:
from fastapi import FastAPI
from pydantic import BaseModel
from dotenv import load_dotenv

load_dotenv()

app = FastAPI(title="Multi-Modal RAG API")


class QueryRequest(BaseModel):
    question: str
    content_type: str | None = None
    top_k: int = 5


class QueryResponse(BaseModel):
    answer: str
    sources: list[dict]


@app.get("/health")
def health_check():
    return {"status": "healthy"}


@app.post("/query", response_model=QueryResponse)
def handle_query(request: QueryRequest):
    return {"answer": "Not implemented yet", "sources": []}

What does this code do?

  • load_dotenv() reads your .env file so the NVIDIA API key is available as an environment variable.
  • QueryRequest defines the shape of incoming queries: a question, an optional content type filter, and how many results to retrieve.
  • QueryResponse defines what the API returns: an answer string and a list of source metadata dictionaries.
  • /health is a simple endpoint to confirm the server is running.
  • /query is a placeholder that returns a dummy response for now. You'll wire it to real retrieval logic in Step 2.
  • Save app/main.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file.

ⓧ I'd like to double check the full code

Here is the complete app/main.py at this point:

from fastapi import FastAPI
from pydantic import BaseModel
from dotenv import load_dotenv

load_dotenv()

app = FastAPI(title="Multi-Modal RAG API")


class QueryRequest(BaseModel):
    question: str
    content_type: str | None = None
    top_k: int = 5


class QueryResponse(BaseModel):
    answer: str
    sources: list[dict]


@app.get("/health")
def health_check():
    return {"status": "healthy"}


@app.post("/query", response_model=QueryResponse)
def handle_query(request: QueryRequest):
    return {"answer": "Not implemented yet", "sources": []}
Pull the Ollama models and start the server

Your agent needs two models through Ollama: a language model with tool-calling support to power the agent's reasoning, and an embedding model to convert text into searchable vectors.

  • Pull the llama3.2:3b agent model and the nomic-embed-text embedding model by running these commands:
ollama pull llama3.2:3b
ollama pull nomic-embed-text

The agent model is about 2GB and the embedding model is about 274MB. llama3.2:3b supports tool calling natively, which is essential for the LangGraph agent to decide which tools to invoke.

Seeing 'Error: could not connect to ollama app'?

Make sure the Ollama application is running on your machine. Open it from your Applications folder or system tray. Ollama must be running in the background before you can pull or use models.

help me get Ollama running on my machine

Now start your FastAPI server. Before you run the next command, what do you think you'll see when you open the docs page?

  • Start the FastAPI development server by running:
uv run uvicorn app.main:app --reload

Why uv run?

The uv run prefix ensures the command executes inside your project's virtual environment. The --reload flag tells uvicorn to restart the server automatically whenever you save a file.

  • Open your browser and navigate to http://127.0.0.1:8000/docs.

You should see FastAPI's interactive documentation page listing two endpoints: /health and /query. This confirms your server is running and both routes are registered.

Page not loading?

Make sure the terminal shows a line like Uvicorn running on http://127.0.0.1:8000 with no errors above it. If you see an import error, double-check that you saved app/main.py and that app/__init__.py exists (even if empty).

help me debug why my FastAPI server won't start

Your development environment is ready and the server is running. Next up, you'll build the text-only RAG tools and a LangGraph agent that decides how to use them.

Build Text-Only RAG (The Baseline That Breaks)

Your API skeleton can respond to requests, but it has no intelligence behind it yet. The /query endpoint returns a hardcoded dummy response.

In this step, you'll build a text-only RAG pipeline as LangGraph tools: one to ingest a PDF into a vector store, and one to search it. You'll wire these tools into a ReAct agent that reasons about how to answer questions. This version works for plain text, but visibly breaks on table data or image content.

In this step, get ready to:
  • Create a sample PDF and build ingestion and retrieval tools with LangChain.
  • Build a LangGraph ReAct agent and wire it to your FastAPI endpoints.
  • Test a query about table data and observe the text-only limitation first-hand.
Create a sample PDF with mixed content

Your RAG pipeline needs a document to process. You need a PDF that contains all three content types: text paragraphs, a data table, and an embedded image.

  • Open any word processor (Google Docs, Microsoft Word, Apple Pages, or LibreOffice).
  • Type a short paragraph about a fictional company, such as: "Acme Corp reported strong growth in 2024 driven by expansion into new markets."
  • Add a data table with quarterly revenue figures. For example, a table with columns Q1, Q2, Q3, Q4 and values like $3.1M, $3.5M, $4.2M, $4.8M.
  • Insert any image below the table. A bar chart, a logo, or a photo will work.
  • Export or save the document as a PDF named sample.pdf.
  • Move sample.pdf into your project's root folder (the multimodal-rag-api directory on your Desktop).

Why this specific content?

You need all three content types so you can test what text-only extraction captures and what it misses. The table data and image will be invisible to the text-only pipeline, which is exactly the gap you will fix in the next step.

Build the PDF ingestion and retrieval tools

Instead of writing plain functions, you'll define LangChain tools that the agent can invoke. Each tool has a description that tells the agent when and how to use it.

  • Create a new file called app/tools.py and add the imports and shared configuration:
import os
import pymupdf4llm
from langchain.tools import tool
from langchain_ollama import OllamaEmbeddings
from langchain_chroma import Chroma
from langchain_text_splitters import RecursiveCharacterTextSplitter

# Shared vector store configuration
CHROMA_PATH = "./chroma_data"
COLLECTION_NAME = "multimodal_docs"

# Embedding model for converting text to vectors
embeddings = OllamaEmbeddings(model="nomic-embed-text")

What does this code do?

  • OllamaEmbeddings wraps Ollama's embedding API. Every time you store or search text, this model converts it into a 768-dimension vector.
  • Chroma from langchain-chroma gives you a LangChain-compatible vector store backed by ChromaDB.
  • CHROMA_PATH and COLLECTION_NAME are module-level constants so both tools share the same vector store.
  • Add the ingest_pdf tool below the configuration in app/tools.py:
@tool
def ingest_pdf(pdf_path: str) -> str:
    """Extract text from a PDF and store chunks in the vector database."""
    # Convert PDF to Markdown text
    md_text = pymupdf4llm.to_markdown(pdf_path)

    # Split into chunks preserving paragraph boundaries
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=1000,
        chunk_overlap=200,
        separators=["\n\n", "\n", " ", ""],
    )
    chunks = splitter.split_text(md_text)

    # Store chunks in ChromaDB with content-type metadata
    vectorstore = Chroma(
        persist_directory=CHROMA_PATH,
        collection_name=COLLECTION_NAME,
        embedding_function=embeddings,
    )
    vectorstore.add_texts(
        texts=chunks,
        metadatas=[{"content_type": "text", "source": os.path.basename(pdf_path)} for _ in chunks],
    )

    return f"Ingested {len(chunks)} text chunks from {os.path.basename(pdf_path)}"

What does this code do?

  • The @tool decorator turns this function into a LangChain tool the agent can call. The docstring becomes the tool's description.
  • pymupdf4llm.to_markdown() converts the PDF to a Markdown string. Tables render as pipe tables, but images are completely ignored.
  • RecursiveCharacterTextSplitter splits on double newlines first (paragraphs), then single newlines, then spaces. This keeps related content together.
  • vectorstore.add_texts() embeds each chunk using nomic-embed-text and stores it with metadata identifying it as text content.
  • Add the query_vectorstore tool below ingest_pdf in the same file:
@tool
def query_vectorstore(question: str) -> str:
    """Search the vector store for document chunks relevant to the question."""
    vectorstore = Chroma(
        persist_directory=CHROMA_PATH,
        collection_name=COLLECTION_NAME,
        embedding_function=embeddings,
    )

    # Find the 5 most similar chunks to the question
    results = vectorstore.similarity_search(question, k=5)

    if not results:
        return "No relevant documents found in the vector store."

    # Format results with content-type metadata
    formatted = []
    for doc in results:
        content_type = doc.metadata.get("content_type", "unknown")
        formatted.append(f"[{content_type}] {doc.page_content[:500]}")

    return "\n\n---\n\n".join(formatted)

What does this code do?

  • similarity_search() embeds the question with the same nomic-embed-text model, then finds the 5 closest stored chunks by cosine distance.
  • Each result is formatted with its content_type label so the agent knows what kind of content it retrieved.
  • Right now, every chunk is labelled text. In Step 4, you'll add table and image chunks too.
  • Save app/tools.py.
  • Test the ingestion tool by running:
uv run python -c "from dotenv import load_dotenv; load_dotenv(); from app.tools import ingest_pdf; print(ingest_pdf.invoke('sample.pdf'))"

You should see output like Ingested 3 text chunks from sample.pdf (the exact count depends on your PDF's length).

Seeing a ModuleNotFoundError?

  • Make sure you are running the command from your project root directory (the folder containing the app/ folder).
  • Check that sample.pdf is in the project root, not inside the app/ folder.
  • Confirm Ollama is running. The embedding model needs Ollama to be active.

help me debug this ModuleNotFoundError when running my tools

Build the LangGraph agent and connect your API

Your tools are ready. Now you need an agent that decides when to call query_vectorstore and uses the retrieved context to answer questions.

  • Create a new file called app/agent.py and add the agent code:
from langchain_ollama import ChatOllama
from langgraph.prebuilt import create_react_agent
from app.tools import query_vectorstore

# Agent LLM with tool-calling support
llm = ChatOllama(model="llama3.2:3b", temperature=0.3)

# Create a ReAct agent that reasons about which tools to call
agent = create_react_agent(llm, tools=[query_vectorstore])


def run_agent(question: str) -> dict:
    """Invoke the agent with a user question and return the response."""
    result = agent.invoke(
        {"messages": [{"role": "user", "content": question}]}
    )
    # Extract the final AI message
    final_message = result["messages"][-1]
    return {"answer": final_message.content, "sources": []}

What does this code do?

  • ChatOllama(model="llama3.2:3b") creates a LangChain chat model backed by your local Ollama server. This model supports native tool calling.
  • create_react_agent() builds a ReAct (Reason + Act) agent. It loops: the LLM decides whether to call a tool, the tool runs, and the result feeds back until the LLM produces a final answer.
  • run_agent() is a convenience wrapper that sends a question, runs the agent loop, and extracts the final response text.

💡 Why a ReAct agent instead of a simple chain?

A chain runs a fixed sequence: always embed, always search, always generate. A ReAct agent reasons first. It reads the question, decides whether it needs the vector store, calls the tool, reads the result, and only then writes an answer.

When you add more tools in later steps (vision, tables), the agent will choose the right one without you rewriting the orchestration logic.

  • Save app/agent.py.

Now connect the agent to your FastAPI endpoint.

  • In app/main.py, find the handle_query function at the bottom of the file:
@app.post("/query", response_model=QueryResponse)
def handle_query(request: QueryRequest):
    return {"answer": "Not implemented yet", "sources": []}
  • Replace it with this version that calls your agent:
@app.post("/query", response_model=QueryResponse)
def handle_query(request: QueryRequest):
    from app.agent import run_agent
    result = run_agent(request.question)
    return result
  • Save app/main.py.

Auto-reload picks up your changes

Because you started the server with --reload in Step 1, uvicorn automatically restarts when you save. If you stopped the server, restart it by running uv run uvicorn app.main:app --reload from your project root.

Test the agent and observe the text-only limitation
  • Test a text-based query to confirm the full agent pipeline works by running:
curl -X POST http://127.0.0.1:8000/query \
  -H "Content-Type: application/json" \
  -d '{"question": "What is Acme Corp?"}'

You should see a JSON response with an answer field containing a generated response about Acme Corp. The agent called query_vectorstore, retrieved the relevant text chunks, and used them to write an answer.

Getting a connection error or empty answer?

  • Make sure Ollama is running. Open a separate terminal and run ollama list to verify.
  • Check that you ran the ingestion command earlier. If chroma_data/ folder does not exist, re-run the ingest_pdf test command.
  • Confirm your question matches content in your PDF. Try a question about text you actually wrote in the document.

help me debug why my /query endpoint returns an error or empty answer

Your text-based query worked. But your PDF also has a table with quarterly revenue data and an embedded image. What happens when you ask a question that depends on that visual content?

Before you run the next command, predict: do you think the agent will find the revenue data from your table, or will it come up empty?

  • Ask a question about the table data by running:
curl -X POST http://127.0.0.1:8000/query \
  -H "Content-Type: application/json" \
  -d '{"question": "What was the revenue in Q3?"}'

Look at the response. The answer is either wrong, vague, or says it does not have enough information. The agent searched the vector store but only found text chunks. No table data was retrieved because the table structure was flattened during markdown conversion.

  • Try asking about the image content:
curl -X POST http://127.0.0.1:8000/query \
  -H "Content-Type: application/json" \
  -d '{"question": "What does the chart show?"}'

This time the response is completely empty or hallucinated. The image was never extracted or described, so it simply does not exist in the vector store.

Why does the agent fail here?

pymupdf4llm.to_markdown() converts tables into pipe-delimited text, which loses semantic structure and makes retrieval unreliable for data questions.

Images are completely ignored by default. No pixel data is extracted and no description is generated. From the vector store's perspective, those images do not exist. The agent can only work with what's in the store.

This is the exact gap you will close in the next step. By adding vision and table extraction tools, the agent will have the ability to process all content types, not just plain text.

You have proven that text-only tools hit a wall with visual content. Next up, you will add vision and table extraction tools that give the agent the ability to see what's inside images and structured tables.

Extract and Describe Visual Content with the Vision Model

In the last step, you asked your RAG agent a question about table data and got silence (or worse, a hallucination). That gap exists because ingest_pdf only extracts text paragraphs.

This step closes the gap by building two new tools: one that extracts tables as structured Markdown, and one that pulls out images and sends them to a cloud vision model for descriptions. Once every visual element has a text representation, your vector store can index and retrieve it just like any paragraph.

In this step, get ready to:
  • Build a table extraction tool using PyMuPDF.
  • Build an image description tool using ChatNVIDIA's vision model.
  • Test both tools independently to confirm they produce searchable text.
Install additional dependencies for table rendering

The table extraction tool converts table data into a pandas DataFrame, then renders it as a Markdown string. This requires two extra packages: pandas and tabulate.

  • Install them by running this command in your terminal:
uv add pandas tabulate
  • You should see both packages added to your pyproject.toml.

Seeing a resolver error?

Make sure you are running the command from inside your project folder (the one with pyproject.toml).

help me fix a uv resolver error when adding pandas and tabulate

Build the table extraction tool

PyMuPDF can detect tables drawn with lines and rectangles on a page. You'll build a tool that uses page.find_tables() to get structured data, converts it to Markdown, and stores it in the vector store with table metadata.

  • In app/tools.py, add these new imports at the top of the file, below the existing imports:
import base64
import pymupdf
from langchain_nvidia_ai_endpoints import ChatNVIDIA
from langchain_core.messages import HumanMessage
  • Below the imports, add the vision model configuration alongside your existing constants:
# Vision model for describing images
vision_llm = ChatNVIDIA(model="meta/llama-3.2-11b-vision-instruct")

What are these new imports?

  • pymupdf gives you low-level access to embedded images and table structures inside the PDF.
  • base64 encodes raw image bytes into a string the vision API can accept.
  • ChatNVIDIA wraps the NVIDIA NIM API as a LangChain chat model. It accepts multimodal inputs (text + images) using the standard HumanMessage format.
  • Add the extract_and_store_tables tool below your existing tools in app/tools.py:
@tool
def extract_and_store_tables(pdf_path: str) -> str:
    """Extract tables from a PDF and store them in the vector database."""
    doc = pymupdf.open(pdf_path)
    tables_data = []

    for page_num, page in enumerate(doc):
        # find_tables detects tables drawn with lines/rectangles
        tables = page.find_tables()
        for table in tables:
            # Convert to pandas DataFrame, then to Markdown
            df = table.to_pandas()
            markdown_table = df.to_markdown(index=False)
            tables_data.append(markdown_table)

    doc.close()

    if not tables_data:
        return "No tables found in the PDF."

    # Store table chunks in the vector store
    vectorstore = Chroma(
        persist_directory=CHROMA_PATH,
        collection_name=COLLECTION_NAME,
        embedding_function=embeddings,
    )
    vectorstore.add_texts(
        texts=tables_data,
        metadatas=[{"content_type": "table", "source": os.path.basename(pdf_path)} for _ in tables_data],
    )

    return f"Extracted and stored {len(tables_data)} table(s) from {os.path.basename(pdf_path)}"

What does this code do?

  • page.find_tables() scans the page for rectangular line patterns that form table cells.
  • table.to_pandas() turns the detected cell data into a DataFrame with rows and columns.
  • df.to_markdown(index=False) renders the DataFrame as a pipe-separated Markdown table, preserving row/column structure as plain text your vector store can embed.
  • Each table is stored with content_type: "table" metadata so the retriever can identify it later.
  • Save app/tools.py.
  • Test the table extraction tool by running:
uv run python -c "from dotenv import load_dotenv; load_dotenv(); from app.tools import extract_and_store_tables; print(extract_and_store_tables.invoke('sample.pdf'))"

You should see output like Extracted and stored 1 table(s) from sample.pdf.

Seeing 'No tables found'?

PyMuPDF detects tables drawn with vector graphics (lines and rectangles). If your sample PDF's table is plain text formatted to look like a table without real cell borders, it will not be detected. Use a PDF that has a proper bordered table.

help me troubleshoot why PyMuPDF find_tables returns no tables from my PDF

Build the image description tool

Tables are covered. Now you need a tool that pulls out embedded images, sends each to the NVIDIA NIM vision model for a text description, and stores those descriptions in the vector store.

  • Add the describe_and_store_images tool below extract_and_store_tables in app/tools.py:
@tool
def describe_and_store_images(pdf_path: str) -> str:
    """Extract images from a PDF, describe them with a vision model, and store descriptions."""
    doc = pymupdf.open(pdf_path)
    descriptions = []

    for page_num, page in enumerate(doc):
        image_list = page.get_images(full=True)
        for img_info in image_list:
            xref = img_info[0]
            # Extract the image pixels using the xref ID
            pix = pymupdf.Pixmap(doc, xref)

            # Convert CMYK/RGBA to RGB
            if pix.alpha:
                pix = pymupdf.Pixmap(pymupdf.csRGB, pix)

            # Encode as base64 PNG for the vision API
            img_bytes = pix.tobytes("png")
            img_b64 = base64.b64encode(img_bytes).decode("utf-8")

            # Send to NVIDIA vision model via ChatNVIDIA
            message = HumanMessage(content=[
                {"type": "text", "text": "Describe this image in detail. If it contains a chart or graph, describe the data points and trends."},
                {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{img_b64}"}},
            ])
            response = vision_llm.invoke([message])
            descriptions.append(response.content)

    doc.close()

What does this code do?

  • page.get_images(full=True) lists every embedded image object on the page, returning an xref (cross-reference ID) for each.
  • pymupdf.Pixmap(doc, xref) extracts the raw pixel data for that image.
  • The alpha check converts RGBA images to RGB so the PNG export works cleanly.
  • ChatNVIDIA accepts a HumanMessage with mixed text and image_url content, just like OpenAI's multimodal format. The base64 data URI goes directly in the message.
  • Immediately below the code you just added (still inside the same function), add the storage logic:
    if not descriptions:
        return "No images found in the PDF."

    # Store image descriptions in the vector store
    vectorstore = Chroma(
        persist_directory=CHROMA_PATH,
        collection_name=COLLECTION_NAME,
        embedding_function=embeddings,
    )
    vectorstore.add_texts(
        texts=descriptions,
        metadatas=[{"content_type": "image", "source": os.path.basename(pdf_path)} for _ in descriptions],
    )

    return f"Described and stored {len(descriptions)} image(s) from {os.path.basename(pdf_path)}"

What does this code do?

This second half stores the vision model's text descriptions in ChromaDB, tagged with content_type: "image". Now when the agent searches for image-related content, these descriptions will appear in the results.

  • Save app/tools.py.
Verify the vision model describes your images

Before you think about what the vision model will say, predict for yourself: based on the image in your sample PDF, what description would be useful for answering questions about it later?

  • Run this test to extract images and send them to the vision model:
uv run python -c "from dotenv import load_dotenv; load_dotenv(); from app.tools import describe_and_store_images; print(describe_and_store_images.invoke('sample.pdf'))"

You should see output like Described and stored 1 image(s) from sample.pdf. The vision model has now converted your image into searchable text.

Getting a 401 or 403 error?

This means your NVIDIA NIM API key is not being loaded correctly. Confirm your .env file has NVIDIA_API_KEY= followed by your actual key (no quotes around the value). The load_dotenv() call in the test script loads it into the environment.

help me fix NVIDIA NIM API authentication in my multi-modal RAG project

🙋‍♀️ Seeing 'No images found'?

Some PDFs embed charts as vector graphics (drawn with lines) rather than raster images. If your sample PDF's chart is vector-based, get_images() will not find it. Use a PDF that contains an actual embedded image (a PNG or JPEG pasted into the document).

help me find images in my PDF for the multi-modal RAG project

✔️ Awesome, I've got everything!

Great. Double check you have saved app/tools.py with all the new pieces: updated imports, vision_llm config, extract_and_store_tables, and describe_and_store_images.

ⓧ I'd like to double check the full code

Scroll through your app/tools.py and confirm these sections are present in order:

  • Imports: os, base64, pymupdf, pymupdf4llm, ChatNVIDIA, HumanMessage, OllamaEmbeddings, Chroma, RecursiveCharacterTextSplitter.
  • Constants: CHROMA_PATH, COLLECTION_NAME, embeddings, vision_llm.
  • Tools in order: ingest_pdf, query_vectorstore, extract_and_store_tables, describe_and_store_images.

You now have text representations of every visual element in your PDF. In the next step, you will give the agent access to all three tools, re-ingest your PDF with the full multi-modal pipeline, and prove that the same queries which previously failed now return accurate answers.

Build the Unified Multi-Modal Vector Store

In Step 3, you built tools that extract tables and describe images from your PDF. Those tools are defined but the agent doesn't know they exist yet.

In this step, you'll give the agent access to all your tools, add an ingestion endpoint to your API, re-ingest your sample PDF with the full multi-modal pipeline, and prove that the same queries which previously failed now return accurate answers.

In this step, get ready to:
  • Update the agent to use all four tools and add a /ingest endpoint.
  • Re-ingest your sample PDF with text, table, and image tools.
  • Test that the same query which previously failed now returns accurate answers.
Update the agent with all tools

Your agent currently only has query_vectorstore. You need to register all four tools so the agent can handle both ingestion and retrieval.

  • Open app/agent.py and find these lines:
from app.tools import query_vectorstore

# Agent LLM with tool-calling support
llm = ChatOllama(model="llama3.2:3b", temperature=0.3)

# Create a ReAct agent that reasons about which tools to call
agent = create_react_agent(llm, tools=[query_vectorstore])
  • Replace those lines so the agent has access to all four tools:
from app.tools import (
    ingest_pdf,
    query_vectorstore,
    extract_and_store_tables,
    describe_and_store_images,
)

# Agent LLM with tool-calling support
llm = ChatOllama(model="llama3.2:3b", temperature=0.3)

# Create a ReAct agent with all RAG tools
agent = create_react_agent(llm, tools=[
    ingest_pdf,
    query_vectorstore,
    extract_and_store_tables,
    describe_and_store_images,
])

What does this change?

  • The agent now sees four tool descriptions. When it receives a question, it reasons about which tools are relevant.
  • For a query like "What was Q3 revenue?", the agent will call query_vectorstore and find table chunks alongside text chunks.
  • For an ingestion request, the agent can call all three ingest tools in sequence to fully process a PDF.
  • Save app/agent.py.
Add the /ingest endpoint

You need an API endpoint that triggers full multi-modal ingestion. This endpoint will call all three ingestion tools directly (text, tables, images) to fully process a PDF.

  • In app/main.py, find the QueryRequest class. Add this new request model above it:
class IngestRequest(BaseModel):
    pdf_path: str
  • Add the /ingest endpoint below the /health endpoint in app/main.py:
@app.post("/ingest")
def handle_ingest(request: IngestRequest):
    from app.tools import (
        ingest_pdf,
        extract_and_store_tables,
        describe_and_store_images,
    )
    # Run all three ingestion tools
    text_result = ingest_pdf.invoke(request.pdf_path)
    table_result = extract_and_store_tables.invoke(request.pdf_path)
    image_result = describe_and_store_images.invoke(request.pdf_path)

    return {
        "text": text_result,
        "tables": table_result,
        "images": image_result,
    }

Why call tools directly instead of through the agent?

  • The /ingest endpoint calls each tool directly with .invoke(). This is intentional: ingestion is a deterministic workflow, not a reasoning task.
  • The agent is reserved for the /query endpoint where it needs to reason about what to search and how to answer.
  • Each tool returns a confirmation string, which the endpoint collects into a response showing what was processed.
  • Save app/main.py.
Re-ingest your PDF with all content types

Your old ChromaDB data only contains text chunks from Step 2. You need to clear it and re-run ingestion so all three modalities get stored.

  • If your FastAPI server is still running, stop it with Ctrl+C.
  • Delete the old ChromaDB data by running:
rm -rf ./chroma_data
  • Restart your FastAPI server by running:
uv run uvicorn app.main:app --reload
  • Trigger full multi-modal ingestion by running:
curl -X POST http://127.0.0.1:8000/ingest \
  -H "Content-Type: application/json" \
  -d '{"pdf_path": "sample.pdf"}'

You should see a JSON response with three fields showing what was processed:

{
  "text": "Ingested 3 text chunks from sample.pdf",
  "tables": "Extracted and stored 1 table(s) from sample.pdf",
  "images": "Described and stored 1 image(s) from sample.pdf"
}

Your exact numbers will differ based on your PDF's content, but all three fields should show successful processing.

Seeing 0 for tables or images?

  • If tables shows "No tables found", your sample PDF may not have tables drawn with lines or borders. PyMuPDF's find_tables() detects tables using vector graphics. Try a PDF with a clearly formatted table.
  • If images shows "No images found", your sample PDF may not have embedded images. Text rendered as an image inside a scanned PDF does not count. Try a PDF with a chart or photo embedded.
  • If you see an NVIDIA API error, confirm your NVIDIA_API_KEY is correct in .env.

Why is my /ingest returning 0 for some content types?

Test multi-modal retrieval with the agent

The moment of truth. In Step 2, you asked a question about table data and the agent could not find relevant content. Now all three modalities are indexed.

Before you send the request, think about what you expect to happen. In Step 2, this same question failed because tables were not indexed. Do you think it will work now?

  • Ask the same table question that failed before by running:
curl -X POST http://127.0.0.1:8000/query \
  -H "Content-Type: application/json" \
  -d '{"question": "What was the revenue in Q3?"}'

This time, the agent retrieves table chunks alongside text chunks and produces an accurate answer grounded in the actual data from your PDF.

  • Now test an image-related question:
curl -X POST http://127.0.0.1:8000/query \
  -H "Content-Type: application/json" \
  -d '{"question": "What does the chart show?"}'

The agent now retrieves the vision model's description of your chart and uses it to answer. The image that was completely invisible in Step 2 is now searchable because it has a text representation in the vector store.

Why does content-type metadata matter?

In production systems, users often know what type of content they are looking for. A finance analyst might want only table data. A designer might want only image descriptions.

Because each chunk is tagged with content_type metadata, you could extend query_vectorstore to accept a filter parameter. The agent would then decide whether to filter based on the user's question.

🙋‍♀️ Still getting irrelevant answers?

  • Make sure you deleted ./chroma_data before re-running ingestion. Old text-only chunks may be polluting results.
  • Confirm your /ingest response showed successful processing for all three content types.
  • Check that your question matches content that actually exists in your sample PDF.

Why is my agent still returning poor answers after re-ingestion?

✔️ Awesome, I've got everything!

Great. Double check you have saved both app/agent.py and app/main.py.

ⓧ I'd like to double check the full code

Confirm your files have these key sections:

app/agent.py:

  • Imports all four tools from app.tools.
  • create_react_agent with all four tools in the list.
  • run_agent() wrapper function.

app/main.py:

  • IngestRequest model with pdf_path field.
  • /ingest endpoint calling all three ingestion tools.
  • /query endpoint calling run_agent().

Your multi-modal RAG agent is fully operational. Text, tables, and images all live in one unified vector store, and the agent reasons about how to search and answer. Next up, you'll tackle the Secret Mission and extend the agent with a production-style upload capability.

Secret mission

Add a /upload Endpoint

Your multi-modal RAG agent can answer questions across text, tables, and images. But the only way to ingest a document is to pass a file path. What if users could upload any PDF on demand and immediately query its content through the agent?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Project folder on your Desktop (contains app/ code, .env, chroma_data/, and virtual environment).
  • Ollama llama3.2:3b and nomic-embed-text models stored in Ollama's local model cache.

Keep everything running

No action needed. Choose this if you're still actively building or plan to extend this project further.

  • Your FastAPI server and ChromaDB data persist locally with no ongoing costs.
  • The NVIDIA NIM free tier does not charge for stored state. Your API key remains in your .env file.
  • The Ollama models stay downloaded on your machine (approximately 2.3 GB total disk space for both models).

Pause - I'll come back to this later

Shut down running processes to free up memory. Your files and data stay on disk so you can resume later.

  • Stop the FastAPI server by pressing Ctrl+C in the terminal where it is running.
  • Your ChromaDB data persists in ./chroma_data inside your project folder.
  • When you return, navigate back into your project folder and run uvicorn app.main:app --reload to start the server again.

Delete - I don't want to use this again

Remove all project resources and reclaim disk space. This is permanent.

  • Stop the FastAPI server by pressing Ctrl+C in the terminal where it is running.
  • Delete the Ollama model by running this command in your terminal:
ollama rm llama3.2:3b
ollama rm nomic-embed-text
  • Delete your project folder. On macOS/Linux, navigate to your Desktop and remove it:
cd ~/Desktop
rm -rf multimodal-rag-api

On Windows, use this command instead:

cd ~\Desktop
rmdir /s /q multimodal-rag-api
  • If your project folder has a different name, replace multimodal-rag-api with whatever you named it when you created it.

Nice Work!

Nice Work!

Nice work! You just built a multi-modal RAG agent that uses LangGraph to reason about which tools to call, processes text, tables, and images from PDFs, and answers questions across all content types.

You've learned how to:

  • Build a LangGraph ReAct agent with custom tools that autonomously decides how to ingest documents and answer questions across text, tables, and images.
  • Use ChatNVIDIA to send images to a cloud vision model and convert visual content into searchable text descriptions, keeping your local RAM free for the agent model.
  • Experience first-hand how text-only tools fail on visual content and how adding vision and table tools to the agent's toolkit bridges that gap.
  • Secret Mission: Added a production-style /upload endpoint for on-demand document ingestion through the full tool harness.

Ready to quiz yourself?