Hybrid C++/Python Document Agent

Build a local RAG agent with a C++ preprocessing engine and LLM tool calling.

Introduction

30 Second Summary

When documents get large, AI tools slow to a crawl. The bottleneck is often text preprocessing. Cleaning, splitting, and chunking thousands of paragraphs in pure Python takes seconds that add up fast.

In this project, you will build a fully local document intelligence system that combines a C++ preprocessing engine with Python AI orchestration. You will feed it text documents, have C++ process them at native speed, index the chunks as vector embeddings, and query them through an AI agent that reasons autonomously over the content.

What You'll Build

You hand the system any text document, ask a natural language question, and watch an AI agent autonomously search, retrieve, and synthesize an answer with cited sources.

By the end of this project, you'll have:

A working RAG pipeline where you ask natural language questions about any document and get grounded answers with cited sources from the AI agent.
An autonomous tool calling loop where the agent decides when to search, what to search for, and when it has enough context to answer.
A C++ engine compiled into Python via pybind11 that processes text significantly faster than equivalent pure Python code.
Secret Mission: Add a benchmarking tool that lets the agent compare its own C++ preprocessing speed against a pure Python equivalent in real-time.

Are there any prerequisites?

This project assumes you have C++ compilation experience and Python fundamentals (virtual environments, pip, basic OOP). You also need Visual Studio with the C++ workload installed, LM Studio installed with the Meta Llama 3.1 8B Instruct model already downloaded, and VS Code for editing.

Before We Start

Before diving in, let's clarify what you're building and why the C++ layer matters for document processing at scale.

Set Up the Project Environment

To build a hybrid C++/Python system, you need both languages talking to each other cleanly. That starts with an isolated Python 3.12 environment and the pybind11 bridge that lets Python import compiled C++ code as a native module.

In this step, you will create the project scaffold, configure the C++ build pipeline, and compile a minimal extension module to confirm the toolchain works end to end.

In this step, get ready to:

Create the project directory, virtual environment, and install dependencies.
Write the build configuration files and initial C++ source.
Compile the C++ extension and verify it imports into Python.

Create the project directory and virtual environment

You need Python 3.12 for this project. Check which version you have by running the version command in your terminal.

Press the Windows key to open your search bar.
Type cmd and press Enter to open Command Prompt.
Check your Python version by running:

python --version

✔️ I see version 3.12 or higher

You are ready to continue.

ⓧ I see an older version

You need Python 3.12 or higher for this project.

Download the Python 3.12 installer from python.org/downloads.
Run the installer and check Add python.exe to PATH during installation.
Close and reopen Command Prompt, then run python --version again to confirm 3.12 appears.

What if my old version still shows?

Windows may still resolve the older Python first. After installing 3.12, try running py -3.12 --version instead. If that shows 3.12, use py -3.12 in place of python for all commands in this project.

ⓧ Command not found

Python is not installed on your machine yet.

Download the Python 3.12 installer from python.org/downloads.
Run the installer and check Add python.exe to PATH during installation.
Close and reopen Command Prompt, then run python --version again to confirm 3.12 appears.

Move to your Desktop by running this command:

cd %USERPROFILE%\Desktop

Create your project folder with its subdirectories and move into it by running these commands:

mkdir doc-intel-agent
cd doc-intel-agent
mkdir cpp scripts documents

Confirm the folders exist by running:

dir

You should see cpp, scripts, and documents listed as directories.

Create a Python virtual environment by running:

python -m venv .venv

Activate the virtual environment by running:

.venv\Scripts\activate

Your terminal prompt now shows (.venv) at the beginning, confirming the virtual environment is active.

Install the project dependencies by running:

pip install pybind11 chromadb openai

What are these packages?

pybind11 bridges C++ and Python so you can import compiled C++ as a Python module.
chromadb is a vector database that handles embedding and similarity search locally.
openai provides the SDK for calling LLM APIs (you will point it at LM Studio's local server).

First-run download for ChromaDB

ChromaDB downloads the all-MiniLM-L6-v2 embedding model (~90 MB) the first time you add documents to a collection. This happens automatically later in the project. No action needed now.

Verify pybind11 installed correctly by running:

python -c "import pybind11; print(pybind11.__version__)"

You should see 3.0.4 printed.

Seeing an error on pip install?

Make sure your virtual environment is active. You should see (.venv) in your prompt.
If pip is not found, try python -m pip install pybind11 chromadb openai instead.
Help me troubleshoot

Create the build configuration files

Python needs a build configuration to know how to compile your C++ code into an importable module. You will create three files that tell the build system where the source lives, which compiler flags to use, and which packages the project depends on.

Create requirements.txt in the root of your doc-intel-agent folder with this content:

pybind11
chromadb
openai

Create pyproject.toml in the same folder with this content:

[build-system]
requires = ["setuptools", "wheel", "pybind11"]
build-backend = "setuptools.build_meta"

What does pyproject.toml do?

This file tells Python's build system which tools it needs before it can compile anything. When you run pip install -e . later, pip reads this file to fetch setuptools and pybind11 as build dependencies.

Confirm both files exist by running:

dir requirements.txt pyproject.toml

You should see both files listed with their sizes.

Now create setup.py in the project root. This file configures the C++ compilation. Add the imports and compiler flag logic:

from setuptools import setup, Extension
from setuptools.command.build_ext import build_ext
import pybind11


class BuildExt(build_ext):
    def build_extensions(self):
        if self.compiler.compiler_type == "msvc":
            for ext in self.extensions:
                ext.extra_compile_args = ["/std:c++17", "/O2", "/openmp"]
        else:
            for ext in self.extensions:
                ext.extra_compile_args = ["-std=c++17", "-O2", "-fopenmp"]
                ext.extra_link_args = ["-fopenmp"]
        super().build_extensions()

What does this code do?

BuildExt overrides the default build command to inject the right compiler flags.
On Windows (MSVC), it passes /std:c++17 for modern C++ support, /O2 for optimization, and /openmp for parallel processing.
On macOS/Linux (GCC/Clang), the equivalent flags are -std=c++17, -O2, and -fopenmp.

Below the class, still in setup.py, add the extension module definition and setup call:

ext_modules = [
    Extension(
        "docengine",
        sources=["cpp/docengine.cpp"],
        include_dirs=[pybind11.get_include()],
        language="c++",
    ),
]

setup(
    name="docengine",
    version="1.0.0",
    description="High-performance C++ document preprocessing engine",
    ext_modules=ext_modules,
    cmdclass={"build_ext": BuildExt},
)

What does this code do?

ext_modules tells setuptools where your C++ source file lives (cpp/docengine.cpp) and that pybind11's headers are needed.
pybind11.get_include() returns the path to pybind11's C++ header files so the compiler can find them.
The setup() call registers the extension and hooks in the custom BuildExt class.

Save setup.py.

✔️ Awesome, I've got everything!

Great. Make sure you have saved requirements.txt, pyproject.toml, and setup.py.

ⓧ I'd like to double check the full code

Scroll up and confirm each file matches these contents in order:

requirements.txt has three lines: pybind11, chromadb, openai.
pyproject.toml has the [build-system] section with requires and build-backend.
setup.py has the BuildExt class (first chunk) followed by ext_modules and the setup() call (second chunk).

Compile the C++ extension module

With the build configuration in place, you need one more file: the actual C++ source. You will write a minimal module that pybind11 can compile, then build it using the MSVC toolchain.

Create cpp/docengine.cpp with this content:

#include <pybind11/pybind11.h>

namespace py = pybind11;

PYBIND11_MODULE(docengine, m) {
    m.doc() = "High-performance C++ document preprocessing engine";
}

What does this code do?

PYBIND11_MODULE(docengine, m) is the entry point macro that tells pybind11 to create a Python module named docengine.
m.doc() sets the module's docstring, which is what Python prints when you call help(docengine) or read docengine.__doc__.
This is a skeleton. You will add the real preprocessing functions in the next step.

✔️ Awesome, I've got everything!

Great. Make sure you have saved cpp/docengine.cpp.

ⓧ I'd like to double check the full code

#include <pybind11/pybind11.h>

namespace py = pybind11;

PYBIND11_MODULE(docengine, m) {
    m.doc() = "High-performance C++ document preprocessing engine";
}

Compiling C++ into a Python module requires the MSVC compiler environment. You need to run the build command from the Developer Command Prompt for Visual Studio, which sets up the correct compiler paths.

Press the Windows key to open your search bar.
Type Developer Command Prompt and open the result for your Visual Studio version (e.g., Developer Command Prompt for VS 2022).
Navigate to your project folder by running:

cd %USERPROFILE%\Desktop\doc-intel-agent

Activate your virtual environment in this new prompt by running:

.venv\Scripts\activate

Build and install the C++ extension in development mode by running:

pip install -e .

What does pip install -e . do?

The -e flag means "editable." It compiles your C++ code and links the result into your virtual environment so that import docengine works immediately. When you change the C++ source later, you just re-run this command to recompile.

You will see compiler output scrolling past. It ends with a success message confirming the install.

Before you run the verification command, what do you think docengine.__doc__ will print?

Test the compiled module by running:

python -c "import docengine; print(docengine.__doc__)"

You should see High-performance C++ document preprocessing engine printed. That is the docstring you set in the C++ source.

Build failed or module not found?

Make sure you are running inside Developer Command Prompt for Visual Studio. A regular Command Prompt does not have the MSVC compiler on its PATH.
Confirm your virtual environment is active (you should see (.venv) in the prompt).
Check that cpp/docengine.cpp exists in the right location. Run dir cpp to verify.
Help me troubleshoot the C++ build

What about macOS or Linux?

If you are on macOS or Linux, you do not need the Developer Command Prompt. Run pip install -e . directly in your regular terminal with the venv active. The setup.py detects a non-MSVC compiler and passes the GCC/Clang flags automatically.

Your toolchain is working. Python can now import compiled C++ code as a native module. Next up, you will fill that module with real text preprocessing functions that clean, split, and chunk documents at native speed.

Build the C++ Preprocessing Engine

Your module compiles and imports, but it does nothing yet. Raw document text is messy: inconsistent whitespace, control characters, and no structure.

A C++ engine can clean and chunk thousands of paragraphs in milliseconds where Python would take seconds. In this step, you will implement the preprocessing functions that power the entire document intelligence pipeline.

In this step, get ready to:

Implement text normalization, paragraph splitting, token estimation, and chunking functions in C++.
Add parallel batch processing with OpenMP.
Write a Python test script that benchmarks each function with real timing data.

Add the text cleaning and paragraph splitting functions

The first two functions handle the dirtiest part of document processing: normalizing whitespace and detecting paragraph boundaries.

In VS Code, open cpp/docengine.cpp.
Select all existing content and delete it. Replace it with the following includes and namespace by pasting this code:

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <string>
#include <vector>
#include <algorithm>
#include <sstream>
#include <cctype>

namespace py = pybind11;

What are these includes for?

pybind11/stl.h enables automatic conversion between C++ STL containers (like std::vector) and Python lists.
The standard library headers provide string manipulation, vector storage, and character classification.

Below the namespace line, add the clean_text function by pasting this code:

std::string clean_text(const std::string& text) {
    std::string result;
    result.reserve(text.size());

    bool prev_space = false;
    for (unsigned char c : text) {
        if (c == '\n' || c == '\r' || c == '\t') {
            if (!prev_space) {
                result += ' ';
                prev_space = true;
            }
        } else if (std::isspace(c)) {
            if (!prev_space) {
                result += ' ';
                prev_space = true;
            }
        } else if (std::isprint(c) || c >= 128) {
            result += c;
            prev_space = false;
        }
    }

    size_t start = result.find_first_not_of(' ');
    size_t end = result.find_last_not_of(' ');
    if (start == std::string::npos) return "";
    return result.substr(start, end - start + 1);
}

What does clean_text do?

This function collapses all whitespace (newlines, tabs, multiple spaces) into single spaces, strips non-printable control characters, and trims leading and trailing whitespace.

The result.reserve(text.size()) pre-allocates memory so the string does not reallocate as it grows. This is a key C++ performance pattern.

Below clean_text, add the split_paragraphs function by pasting this code:

std::vector<std::string> split_paragraphs(const std::string& text) {
    std::vector<std::string> paragraphs;
    std::string current;
    int newline_count = 0;

    for (size_t i = 0; i < text.size(); ++i) {
        if (text[i] == '\n') {
            newline_count++;
            if (newline_count >= 2) {
                std::string trimmed = clean_text(current);
                if (!trimmed.empty()) {
                    paragraphs.push_back(trimmed);
                }
                current.clear();
                newline_count = 0;
            } else {
                current += ' ';
            }
        } else {
            newline_count = 0;
            current += text[i];
        }
    }

    std::string trimmed = clean_text(current);
    if (!trimmed.empty()) {
        paragraphs.push_back(trimmed);
    }

    return paragraphs;
}

What does split_paragraphs do?

Documents separate ideas with blank lines (double newlines). This function detects those boundaries, cleans each paragraph with clean_text, and returns them as a vector of strings.

Below split_paragraphs, add a temporary PYBIND11_MODULE block that exports these two functions:

PYBIND11_MODULE(docengine, m) {
    m.doc() = "High-performance C++ document preprocessing engine";

    m.def("clean_text", &clean_text,
          "Fast text normalization: strips control chars, normalizes whitespace",
          py::arg("text"));

    m.def("split_paragraphs", &split_paragraphs,
          "Split text into paragraphs on double-newline boundaries",
          py::arg("text"));
}

Save cpp/docengine.cpp.
Rebuild the extension in your Developer Command Prompt for Visual Studio by running:

pip install -e .

Before you test, what do you think clean_text will do to a string containing newlines and extra spaces?

Test the two functions by running:

python -c "import docengine; print(docengine.clean_text('  hello\n\nworld  '))"

You should see hello world printed as a single cleaned string with no extra whitespace.

Seeing a compilation error?

Make sure you are running pip install -e . from the project root directory (where setup.py lives), not from the cpp/ subdirectory.
Confirm you are using Developer Command Prompt for Visual Studio, not a regular terminal. MSVC must be on the PATH.

Help me debug a pybind11 compilation error on Windows with MSVC

Add the chunking and batch processing functions

The next functions handle the intelligence side: estimating token counts and splitting text into overlapping chunks sized for LLM context windows.

In cpp/docengine.cpp, find the PYBIND11_MODULE block at the bottom of the file.
Add the estimate_tokens function ABOVE the PYBIND11_MODULE block by pasting:

size_t estimate_tokens(const std::string& text) {
    if (text.empty()) return 0;
    size_t word_count = 0;
    bool in_word = false;
    for (unsigned char c : text) {
        if (std::isspace(c)) {
            in_word = false;
        } else if (!in_word) {
            in_word = true;
            word_count++;
        }
    }
    return static_cast<size_t>(word_count * 1.3);
}

Why multiply by 1.3?

LLM tokenizers split words into subword tokens. On average, one English word becomes roughly 1.3 tokens. This gives a fast approximation without loading a full tokenizer model.

Below estimate_tokens (still above PYBIND11_MODULE), add the first part of the chunk_text function:

std::vector<std::string> chunk_text(const std::string& text, size_t max_tokens, size_t overlap_tokens) {
    std::vector<std::string> paragraphs = split_paragraphs(text);
    std::vector<std::string> chunks;

    std::string current_chunk;
    size_t current_tokens = 0;
    std::vector<std::string> current_paragraphs;

    for (const auto& para : paragraphs) {
        size_t para_tokens = estimate_tokens(para);

        if (current_tokens + para_tokens > max_tokens && !current_chunk.empty()) {
            chunks.push_back(current_chunk);

            std::string overlap_text;
            size_t overlap_count = 0;
            for (int i = static_cast<int>(current_paragraphs.size()) - 1; i >= 0; --i) {
                size_t pt = estimate_tokens(current_paragraphs[i]);
                if (overlap_count + pt > overlap_tokens) break;
                overlap_text = current_paragraphs[i] + " " + overlap_text;
                overlap_count += pt;
            }

            current_chunk = overlap_text;
            current_tokens = overlap_count;
            current_paragraphs.clear();
        }

What is the overlap logic doing?

When a chunk reaches the token limit, the function saves it and starts the next chunk. The overlap loop walks backward through the previous paragraphs, carrying a few over into the new chunk.

This overlap ensures that context is not lost at chunk boundaries. A question whose answer spans two paragraphs will find both in at least one chunk.

Immediately below the code you just pasted (still inside chunk_text), add the continuation that completes the function and the batch_clean function:

        if (!current_chunk.empty()) {
            current_chunk += " " + para;
        } else {
            current_chunk = para;
        }
        current_tokens += para_tokens;
        current_paragraphs.push_back(para);
    }

    if (!current_chunk.empty()) {
        chunks.push_back(current_chunk);
    }

    return chunks;
}

std::vector<std::string> batch_clean(const std::vector<std::string>& texts) {
    std::vector<std::string> results(texts.size());

    #pragma omp parallel for
    for (int i = 0; i < static_cast<int>(texts.size()); ++i) {
        results[i] = clean_text(texts[i]);
    }

    return results;
}

What does batch_clean do?

The #pragma omp parallel for directive tells the compiler to distribute loop iterations across CPU cores. When cleaning hundreds of paragraphs, this parallelism delivers a measurable speedup on multi-core machines.

Now replace the existing PYBIND11_MODULE block with the full version that exports all five functions:

PYBIND11_MODULE(docengine, m) {
    m.doc() = "High-performance C++ document preprocessing engine";

    m.def("clean_text", &clean_text,
          "Fast text normalization: strips control chars, normalizes whitespace",
          py::arg("text"));

    m.def("split_paragraphs", &split_paragraphs,
          "Split text into paragraphs on double-newline boundaries",
          py::arg("text"));

    m.def("estimate_tokens", &estimate_tokens,
          "Approximate token count for a text string",
          py::arg("text"));

    m.def("chunk_text", &chunk_text,
          "Split text into overlapping chunks for RAG retrieval",
          py::arg("text"), py::arg("max_tokens") = 256, py::arg("overlap_tokens") = 32);

    m.def("batch_clean", &batch_clean,
          "Parallel batch text cleaning using OpenMP",
          py::arg("texts"));
}

Save cpp/docengine.cpp.
Rebuild the module by running:

pip install -e .

Test the chunking by running:

python -c "import docengine; chunks = docengine.chunk_text('Para one.\n\nPara two.\n\nPara three.', 10, 2); print(len(chunks), 'chunks created')"

You should see chunks being created from the input text, confirming all five functions compiled and linked correctly.

Getting an unresolved symbol error?

Check that estimate_tokens and chunk_text are defined ABOVE the PYBIND11_MODULE block. C++ requires functions to be declared before they are referenced.
Verify every opening brace has a matching closing brace. A missing } at the end of chunk_text is a common issue when pasting in two parts.

Help me fix a linker or compilation error in my pybind11 module

Write the test script and run it against a sample document

The module compiles, but you need to see it process a real document with timing data to confirm it works at speed.

Create a new file at scripts/test_engine.py and start with the imports and the first half of the main() function:

"""Test the C++ document preprocessing engine."""

import time
from pathlib import Path

import docengine


def main():
    sample_path = Path(__file__).parent.parent / "documents" / "sample.txt"

    if not sample_path.exists():
        print("Creating sample document...")
        sample_path.parent.mkdir(parents=True, exist_ok=True)
        sample_path.write_text(SAMPLE_TEXT, encoding="utf-8")

    raw_text = sample_path.read_text(encoding="utf-8")
    print(f"Loaded document: {len(raw_text)} characters")

    start = time.perf_counter()
    cleaned = docengine.clean_text(raw_text)
    elapsed_clean = (time.perf_counter() - start) * 1000
    print(f"\nclean_text: {elapsed_clean:.2f}ms")
    print(f"  Result length: {len(cleaned)} chars")

What does this code do?

The script checks for a sample document and creates one from an embedded constant if it does not exist.
time.perf_counter() measures wall-clock time in seconds. Multiplying by 1000 converts to milliseconds.
Each function call is timed individually so you can see exactly how fast the C++ engine processes each stage.

Continue the main() function by pasting the remaining timing tests directly below:

    start = time.perf_counter()
    paragraphs = docengine.split_paragraphs(raw_text)
    elapsed_split = (time.perf_counter() - start) * 1000
    print(f"\nsplit_paragraphs: {elapsed_split:.2f}ms")
    print(f"  Found {len(paragraphs)} paragraphs")

    start = time.perf_counter()
    tokens = docengine.estimate_tokens(raw_text)
    elapsed_tokens = (time.perf_counter() - start) * 1000
    print(f"\nestimate_tokens: {elapsed_tokens:.2f}ms")
    print(f"  Estimated {tokens} tokens")

    start = time.perf_counter()
    chunks = docengine.chunk_text(raw_text, max_tokens=256, overlap_tokens=32)
    elapsed_chunk = (time.perf_counter() - start) * 1000
    print(f"\nchunk_text (max=256, overlap=32): {elapsed_chunk:.2f}ms")
    print(f"  Created {len(chunks)} chunks")
    for i, chunk in enumerate(chunks[:3]):
        print(f"\n  Chunk {i}: ({docengine.estimate_tokens(chunk)} tokens)")
        print(f"  {chunk[:80]}...")

    total = elapsed_clean + elapsed_split + elapsed_tokens + elapsed_chunk
    print(f"\nTotal processing time: {total:.2f}ms")

What does this section test?

Each of the four exported functions gets its own timed run. The chunk preview at the end prints the first 80 characters of the first three chunks so you can confirm the overlapping chunking strategy works.

Below the main() function, add the first half of the SAMPLE_TEXT constant that provides test data:

SAMPLE_TEXT = """Introduction to Distributed Systems

Distributed systems are collections of independent computers that appear to users
as a single coherent system. They enable organizations to scale their applications
beyond the limits of a single machine.

Consistency Models

In distributed computing, consistency models define the rules for how updates
propagate through the system. Strong consistency guarantees that all nodes see the
same data at the same time, but this comes at the cost of availability and latency.

Eventual consistency is a weaker model where updates will eventually propagate to
all nodes, but there may be temporary inconsistencies. This model is used by many
large-scale systems like Amazon DynamoDB and Apache Cassandra because it allows
for higher availability and partition tolerance.

The CAP theorem, formulated by Eric Brewer, states that a distributed system can
only guarantee two of the following three properties simultaneously: Consistency,
Availability, and Partition tolerance. This fundamental trade-off shapes the design
of every distributed database and messaging system.

Consensus Algorithms

Consensus algorithms enable distributed systems to agree on a single value even
when some nodes fail. The most well-known algorithms include Paxos, developed by
Leslie Lamport, and Raft, designed by Diego Ongaro as a more understandable
alternative.

Continue the SAMPLE_TEXT string (still inside the triple quotes) by pasting this directly below:

Raft divides the consensus problem into three sub-problems: leader election,
log replication, and safety. A Raft cluster contains several servers, one of which
is elected as the leader. The leader accepts client requests, replicates them to
follower nodes, and tells them when it is safe to apply the entries.

If the leader fails, a new election takes place. Followers that have not received
a heartbeat from the leader within a timeout period become candidates and request
votes from other nodes. The candidate that receives a majority of votes becomes
the new leader.

Fault Tolerance Patterns

Modern distributed systems employ several patterns to handle failures gracefully.
Circuit breakers prevent cascading failures by temporarily stopping requests to
a failing service. Retry mechanisms with exponential backoff allow transient
failures to resolve without overwhelming the system.

Bulkhead isolation separates system components so that a failure in one does not
affect others. This pattern is inspired by the bulkheads in ship hulls that prevent
water from flooding the entire vessel when one compartment is breached.

Health checks and heartbeat mechanisms allow systems to detect failures quickly
and route traffic away from unhealthy nodes. Combined with load balancers, these
patterns enable systems to maintain high availability even during partial failures.

Add the final section of SAMPLE_TEXT and close the string:

Message Queues and Event Streaming

Asynchronous communication through message queues decouples producers from consumers,
allowing each to scale independently. Apache Kafka, originally developed at LinkedIn,
has become the industry standard for event streaming, handling trillions of messages
per day at many organizations.

Kafka organizes messages into topics, which are split into partitions for parallel
processing. Consumer groups allow multiple consumers to share the work of processing
a topic, with each partition assigned to exactly one consumer in the group.

The log-based architecture of Kafka provides durability and replayability. Consumers
can re-read messages from any point in the log, enabling use cases like event sourcing
and change data capture that require processing historical events.

Service Discovery and Load Balancing

In dynamic environments where services scale up and down frequently, service discovery
mechanisms help components find each other. Tools like Consul, etcd, and ZooKeeper
maintain registries of available services and their locations.

Client-side load balancing distributes requests across available instances without
requiring a dedicated load balancer. Server-side load balancing, using tools like
NGINX or HAProxy, provides a single entry point and handles distribution transparently.

Observability and Monitoring

Distributed tracing tools like Jaeger and Zipkin track requests as they flow through
multiple services, helping developers understand performance bottlenecks and failure
points. Combined with metrics collection (Prometheus) and log aggregation (ELK stack),
these tools provide comprehensive observability into system behavior.
"""

Finally, add the entry point at the very bottom of the file:


if __name__ == "__main__":
    main()

Save scripts/test_engine.py.

Before you run this, predict: will the total processing time be closer to 1ms or 100ms for this document?

Run the test script:

python scripts/test_engine.py

You should see output showing each function's execution time, the number of paragraphs found, chunk count, and chunk previews. The total processing time should complete in well under 10ms for this document.

What should I see?

The output prints timing for each function (typically under 1ms each), finds around 15 paragraphs, estimates several hundred tokens, and creates multiple overlapping chunks. The first three chunks show previews with their token counts.

The script also creates documents/sample.txt automatically on the first run. This is the file the RAG pipeline will index in later steps.

Seeing ModuleNotFoundError for docengine?

Confirm you ran pip install -e . after saving the latest cpp/docengine.cpp. Every C++ change requires a rebuild.
Make sure your virtual environment is activated. You should see the venv name in your terminal prompt.

Help me fix a ModuleNotFoundError when importing my pybind11 docengine module

✔️ Awesome, I've got everything!

Great. Confirm you have saved both cpp/docengine.cpp and scripts/test_engine.py.

ⓧ I'd like to double check the full code

Your cpp/docengine.cpp should contain these sections in order (scroll up and confirm each is present):

7 #include headers and namespace py = pybind11;.
clean_text function (returns std::string).
split_paragraphs function (returns std::vector<std::string>).
estimate_tokens function (returns size_t).
chunk_text function (returns std::vector<std::string>).
batch_clean function with #pragma omp parallel for.
PYBIND11_MODULE block exporting all 5 functions with py::arg() annotations.

Your scripts/test_engine.py should contain:

Imports (time, Path, docengine).
main() function with timed calls to all four processing functions.
SAMPLE_TEXT constant with the full distributed systems article.
if __name__ == "__main__": main() entry point.

Your C++ engine is processing real documents in under 10ms. Next up, you will connect this engine to an LLM and see why just stuffing an entire document into the prompt is not enough.

Try Naive Context Stuffing

Your C++ engine can preprocess thousands of words in under 10ms. The most obvious way to answer questions about a document is to clean the entire thing, then paste it all into an LLM prompt alongside the user's question.

This "context stuffing" approach is dead simple. It also breaks spectacularly on real-sized documents. In this step, you will build it, see it work on a small document, and then watch it fail on a larger one.

In this step, get ready to:

Start LM Studio's local server for AI inference.
Write a naive Q&A script that stuffs entire documents into the LLM prompt.
Observe the approach fail on a longer document that exceeds the context window.

Start LM Studio and write the naive Q&A script

The naive approach needs an LLM to answer questions. LM Studio runs a local inference server that exposes an OpenAI-compatible API at http://localhost:1234. Your Python script will talk to it like any other chat completions endpoint.

Open LM Studio.
Make sure your Meta Llama 3.1 8B Instruct model is loaded.
Click the Developer tab in the left sidebar.
Toggle the Start server switch to start the API server.

You should see a confirmation that the server is running on port 1234.

Server not starting?

Make sure you have a model loaded in LM Studio before starting the server. The server needs at least one model available to accept requests.

Help me get LM Studio's local server running

Now you will write the naive Q&A script. It will load a document, preprocess it with the C++ engine, and send the entire cleaned text to the LLM.

Create scripts/naive_qa.py and add the imports and document loader function by pasting this code:

"""Naive Q&A: stuff entire document into context window."""

import sys
from pathlib import Path

from openai import OpenAI

import docengine


def load_document(filepath: str) -> str:
    # Read raw text and clean it with the C++ engine
    raw_text = Path(filepath).read_text(encoding="utf-8")
    cleaned = docengine.clean_text(raw_text)
    return cleaned

What does this code do?

The openai package provides the client library that talks to LM Studio's OpenAI-compatible endpoint. It is not specific to OpenAI's cloud service.

The load_document function reads a file from disk and immediately passes it through your C++ clean_text function to normalize whitespace and strip control characters.

Below your load_document function, add the ask_question function that sends the entire document to the LLM:

def ask_question(document_text: str, question: str) -> str:
    # Connect to LM Studio's local server
    client = OpenAI(base_url="http://localhost:1234/v1", api_key="lm-studio")

    tokens = docengine.estimate_tokens(document_text)
    print(f"[Engine] Document size: ~{tokens} tokens")

    # Stuff the ENTIRE document into the prompt
    response = client.chat.completions.create(
        model="meta-llama-3.1-8b-instruct",
        messages=[
            {
                "role": "system",
                "content": (
                    "Answer questions based on the provided document. "
                    "If the answer is not in the document, say so."
                ),
            },
            {
                "role": "user",
                "content": f"Document:\n{document_text}\n\nQuestion: {question}",
            },
        ],
        temperature=0.1,
    )

    return response.choices[0].message.content

What does this code do?

This function connects to LM Studio at http://localhost:1234/v1 using the OpenAI client library. The api_key is set to lm-studio because LM Studio does not require authentication by default.

The key line is the user message: it dumps the entire document_text into the prompt, then appends the question. This is context stuffing in action.

What model identifier should I use?

The code uses meta-llama-3.1-8b-instruct as the model identifier. Check your LM Studio Developer tab to confirm the exact identifier shown for your loaded model. If it differs, update the string to match.

Save naive_qa.py.
Confirm the server is still running in LM Studio by checking that the Developer tab shows the green status indicator.

Below ask_question, add the main entry point that wires everything together:

def main():
    if len(sys.argv) < 3:
        print("Usage: python naive_qa.py <document_path> <question>")
        sys.exit(1)

    filepath = sys.argv[1]
    question = sys.argv[2]

    print(f"[Engine] Loading document: {filepath}")
    document_text = load_document(filepath)
    print(f"[Engine] Preprocessed with C++ engine")

    print(f"[Agent] Asking: {question}")
    answer = ask_question(document_text, question)
    print(f"[Agent] Answer: {answer}")


if __name__ == "__main__":
    main()

What does this code do?

The main function takes two command-line arguments: a path to a document file and a question to ask about it. It loads the document, cleans it with the C++ engine, and passes the full text to the LLM.

Save naive_qa.py.

Before you run this, what do you think will happen when you ask a question about your 2000-word sample document? Will the LLM be able to handle it?

Test the script against your sample document by running:

python scripts/naive_qa.py documents/sample.txt "What is the CAP theorem?"

You should see the C++ engine report the document size in tokens, followed by a coherent answer about the CAP theorem from the LLM. The sample document is small enough to fit entirely in the context window.

Getting a connection error?

Make sure LM Studio's server is still running. Switch back to LM Studio and check the Developer tab shows the server is active on port 1234.

If you see a model not found error, verify the model identifier in your code matches what LM Studio shows in the Developer tab.

Help me connect my Python script to LM Studio

✔️ Awesome, I've got everything!

Great. Double check you have saved scripts/naive_qa.py and confirmed the LLM answered your question.

ⓧ I'd like to double check the full code

Here is the complete scripts/naive_qa.py file. Scroll through and confirm each section matches:

"""Naive Q&A: stuff entire document into context window."""

import sys
from pathlib import Path

from openai import OpenAI

import docengine


def load_document(filepath: str) -> str:
    raw_text = Path(filepath).read_text(encoding="utf-8")
    cleaned = docengine.clean_text(raw_text)
    return cleaned

def ask_question(document_text: str, question: str) -> str:
    client = OpenAI(base_url="http://localhost:1234/v1", api_key="lm-studio")

    tokens = docengine.estimate_tokens(document_text)
    print(f"[Engine] Document size: ~{tokens} tokens")

    response = client.chat.completions.create(
        model="meta-llama-3.1-8b-instruct",
        messages=[
            {
                "role": "system",
                "content": (
                    "Answer questions based on the provided document. "
                    "If the answer is not in the document, say so."
                ),
            },
            {
                "role": "user",
                "content": f"Document:\n{document_text}\n\nQuestion: {question}",
            },
        ],
        temperature=0.1,
    )

    return response.choices[0].message.content

def main():
    if len(sys.argv) < 3:
        print("Usage: python naive_qa.py <document_path> <question>")
        sys.exit(1)

    filepath = sys.argv[1]
    question = sys.argv[2]

    print(f"[Engine] Loading document: {filepath}")
    document_text = load_document(filepath)
    print(f"[Engine] Preprocessed with C++ engine")

    print(f"[Agent] Asking: {question}")
    answer = ask_question(document_text, question)
    print(f"[Agent] Answer: {answer}")


if __name__ == "__main__":
    main()

Watch context stuffing break on a larger document

The sample document is around 2000 words. That fits comfortably in the model's context window. But real documents are much longer. What happens when you try to stuff a 4000+ word document into the same prompt?

Create a longer test document by running this command, which repeats your sample text multiple times:

python -c "from pathlib import Path; t = Path('documents/sample.txt').read_text(); Path('documents/long_sample.txt').write_text(t * 4)"

What does this command do?

This one-liner reads your sample document and writes it four times back-to-back into a new file called documents/long_sample.txt. The result is a 8000+ word document that will exceed the model's effective context window.

Confirm the file was created by running:

ls documents/

You should see both sample.txt and long_sample.txt listed.

Now, before you run the script against this longer document, think about what you expect. The entire 8000+ word text will be pasted into the prompt alongside the system message and your question. Do you think the model will handle it cleanly?

Run the naive Q&A script against the longer document by running:

python scripts/naive_qa.py documents/long_sample.txt "What is the CAP theorem?"

This time, the output is different. The LLM either produces a truncated answer, hallucinates information not in the document, returns incoherent output, or refuses to answer entirely. The token count printed by the script reveals why: the document is far too large for the model's effective context window.

Why does this happen?

Even though Llama 3.1 8B technically supports a large context window, LM Studio's default settings often limit the effective context to 4096-8192 tokens depending on available RAM and model quantization. When your prompt exceeds that limit, the model either truncates the input silently or produces degraded output.

This is the fundamental problem with context stuffing. It works fine for short documents but collapses as documents grow. You cannot just throw everything at the model and hope for the best.

Why not just increase the context window?

Larger context windows use significantly more RAM and slow down inference. Even if you could fit the entire document, the model's attention degrades on very long inputs. It struggles to find the relevant needle in a haystack of text. The real solution is to give the model only the relevant pieces.

You just hit the wall that every document Q&A system runs into. Pasting everything into the prompt is a dead end for real-world documents. In the next step, you will solve this with RAG: instead of stuffing the entire document, you will retrieve only the relevant chunks and send those to the LLM.

Introduce RAG with ChromaDB

In the previous step, you saw the naive approach collapse when the document exceeded the model's context window. The LLM either refused, hallucinated, or produced gibberish because you tried to stuff everything in at once.

RAG (Retrieval-Augmented Generation) solves this by flipping the model: instead of sending the whole document, you send only the 3 most relevant chunks. ChromaDB handles embedding and similarity search locally, and your C++ engine provides the chunking.

In this step, get ready to:

Build a vector index from document chunks using ChromaDB.
Write a RAG query script that retrieves only relevant chunks before asking the LLM.
Verify that RAG produces correct answers where naive context stuffing failed.

Build the vector index script

The indexing script reads a document, chunks it with your C++ engine, and stores those chunks as vector embeddings in a persistent ChromaDB collection. ChromaDB's default embedding model (all-MiniLM-L6-v2) runs locally and handles vectorization automatically.

Create a new file called scripts/build_index.py and add the imports and build_index function by pasting this code:

"""Build a ChromaDB vector index from a document using C++ preprocessing."""

import sys
from pathlib import Path

import chromadb

import docengine


def build_index(filepath: str, collection_name: str = "documents"):
    raw_text = Path(filepath).read_text(encoding="utf-8")

    print(f"[Engine] Chunking document with C++ engine...")
    chunks = docengine.chunk_text(raw_text, max_tokens=256, overlap_tokens=32)
    print(f"[Engine] Created {len(chunks)} chunks")

    client = chromadb.PersistentClient(path="./chroma_db")

    try:
        client.delete_collection(name=collection_name)
    except ValueError:
        pass

    collection = client.create_collection(name=collection_name)

    ids = [f"chunk_{i}" for i in range(len(chunks))]
    metadatas = [{"source": filepath, "chunk_index": i} for i in range(len(chunks))]

    collection.add(documents=chunks, ids=ids, metadatas=metadatas)

What does this code do?

docengine.chunk_text() splits the raw document into overlapping chunks of ~256 tokens each, using your C++ engine for speed.
PersistentClient(path="./chroma_db") creates a local database on disk so your index survives between script runs.
The try/except ValueError block deletes any existing collection so you get a fresh index each time.
collection.add() stores the chunks along with metadata (source file and chunk position) and automatically generates vector embeddings for each.

Below the build_index function, add the print statements, return, the query_index helper, and the main entry point:

    print(f"[Index] Indexed {len(chunks)} chunks into ChromaDB")
    print(f"[Index] Database stored at ./chroma_db")

    return collection


def query_index(question: str, collection_name: str = "documents", n_results: int = 3):
    client = chromadb.PersistentClient(path="./chroma_db")
    collection = client.get_collection(name=collection_name)

    results = collection.query(query_texts=[question], n_results=n_results)

    return results["documents"][0], results["distances"][0]


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: python build_index.py <document_path>")
        sys.exit(1)

    build_index(sys.argv[1])
    print("\nTest query: 'What is the CAP theorem?'")
    docs, distances = query_index("What is the CAP theorem?")
    for i, (doc, dist) in enumerate(zip(docs, distances)):
        print(f"\n  Chunk {i+1} (distance: {dist:.4f}):")
        print(f"  {doc[:120]}...")

What does query_index do?

query_index() is a convenience function that queries the ChromaDB collection by semantic similarity. It returns the top matching chunks and their distance scores (lower distance means higher relevance).

The main block indexes the document, then immediately runs a test query so you can see retrieval working end-to-end.

Save scripts/build_index.py.
Run the indexing script against your sample document by running this command:

python scripts/build_index.py documents/sample.txt

First-run download

The first time you add documents, ChromaDB automatically downloads the all-MiniLM-L6-v2 embedding model (~80MB). This is a one-time download. Wait for it to complete before continuing.

You should see output showing the chunks created and a test query result with three matching chunks, each showing a distance score and a preview of the text.

Confirm a chroma_db directory now exists in your project root by running:

ls chroma_db

You should see files inside confirming the persistent database was created.

Not seeing any output or getting an import error?

Make sure your virtual environment is still activated. You should see the venv name in your terminal prompt.
If you see a ModuleNotFoundError for chromadb, run pip install chromadb again in the activated venv.
If you see an error about docengine, make sure you ran pip install -e . from the project root in a previous step.

Help me debug my build_index.py script

✔️ Awesome, I've got everything!

Great. Double check you saved scripts/build_index.py and confirmed the chroma_db directory exists.

ⓧ I'd like to double check the full code

Your scripts/build_index.py should contain these sections in order. Scroll up and confirm each is present:

Docstring and imports (sys, pathlib, chromadb, docengine).
build_index() function that chunks, creates the client, deletes old collection, creates new, adds chunks, prints status, and returns the collection.
query_index() function that queries the collection and returns documents and distances.
if __name__ == "__main__" block that indexes from argv, runs a test query for 'What is the CAP theorem?', and prints results.

Write the RAG query script

Now you have chunks indexed as vectors. The RAG script queries ChromaDB for the most relevant chunks, then sends only those to the LLM instead of the entire document.

Create a new file called scripts/rag_qa.py and add the imports and retrieval function by pasting this code:

"""RAG-based Q&A: retrieve relevant chunks then answer."""

import sys

import chromadb
from openai import OpenAI

import docengine


def retrieve_context(question: str, n_results: int = 3) -> str:
    client = chromadb.PersistentClient(path="./chroma_db")
    collection = client.get_collection(name="documents")

    results = collection.query(query_texts=[question], n_results=n_results)

    context_parts = []
    for i, doc in enumerate(results["documents"][0]):
        context_parts.append(f"[Section {i+1}]\n{doc}")

    return "\n\n".join(context_parts)

What does retrieve_context do?

This function takes a natural language question, searches the ChromaDB index for the 3 most semantically similar chunks, and formats them with section labels. The LLM will receive only these focused sections instead of the entire document.

Below retrieve_context, add the ask_with_rag function that sends the retrieved context to LM Studio:

def ask_with_rag(question: str) -> str:
    context = retrieve_context(question)
    context_tokens = docengine.estimate_tokens(context)
    print(f"[RAG] Retrieved context: ~{context_tokens} tokens (vs full document)")

    client = OpenAI(base_url="http://localhost:1234/v1", api_key="lm-studio")

    response = client.chat.completions.create(
        model="meta-llama-3.1-8b-instruct",
        messages=[
            {
                "role": "system",
                "content": (
                    "Answer questions based only on the provided context sections. "
                    "Cite which section number(s) your answer comes from. "
                    "If the answer is not in the provided context, say so."
                ),
            },
            {
                "role": "user",
                "content": f"Context:\n{context}\n\nQuestion: {question}",
            },
        ],
        temperature=0.1,
    )

    return response.choices[0].message.content

How is this different from naive_qa.py?

Instead of f"Document:\n{document_text}" (the entire document), this sends f"Context:\n{context}" (only 3 relevant chunks). The system prompt also instructs the model to cite which section numbers it draws from, giving you traceability.

Complete the file by adding the entry point below ask_with_rag:

def main():
    if len(sys.argv) < 2:
        print("Usage: python rag_qa.py <question>")
        print("  (Run build_index.py first to index a document)")
        sys.exit(1)

    question = " ".join(sys.argv[1:])
    print(f"[RAG] Question: {question}")
    answer = ask_with_rag(question)
    print(f"[RAG] Answer: {answer}")


if __name__ == "__main__":
    main()

Save scripts/rag_qa.py.
Before you run this, think: how many tokens do you expect the RAG approach to send to the LLM compared to the naive approach?
Test RAG against the same question that worked with naive_qa.py by running:

python scripts/rag_qa.py "What is the CAP theorem?"

You should see a line like [RAG] Retrieved context: ~XXX tokens (vs full document) showing a much smaller token count than the full document. The answer should correctly describe the CAP theorem and cite which section(s) the information came from.

Getting a connection error or empty response?

Make sure LM Studio's server is still running. Switch to LM Studio and check the **Developer** tab shows the server active at http://localhost:1234.
If the model name doesn't match what LM Studio has loaded, check the model identifier in LM Studio's Developer tab and update the model parameter in your script.
If you get a collection-not-found error, run build_index.py again first. The RAG script depends on an existing index.

Help me debug my rag_qa.py script

✔️ Awesome, I've got everything!

Great. Double check you saved scripts/rag_qa.py and received a grounded answer with section citations.

ⓧ I'd like to double check the full code

Your scripts/rag_qa.py should contain these sections in order. Scroll up and confirm each is present:

Docstring and imports (sys, chromadb, OpenAI, docengine).
retrieve_context() function that queries ChromaDB for top 3 chunks and formats them with section labels.
ask_with_rag() function that retrieves context, prints token count, and sends the system+user messages to LM Studio.
main() function with argument parsing and the if __name__ == "__main__" entry point.

Verify RAG handles the longer document

The real test is whether RAG succeeds where naive context stuffing failed. In the previous step, the longer document produced incoherent or truncated output.

Index the longer document you used in the previous step by running:

python scripts/build_index.py documents/sample.txt

Why re-index sample.txt?

If you used a longer version of sample.txt or a second document to trigger the naive approach's failure, index that same file here. The point is to show RAG handles the document that previously broke.

Before you run the RAG query, do you think it will succeed this time? Remember, it only retrieves 3 chunks regardless of total document size.
Now ask a question against the indexed document by running:

python scripts/rag_qa.py "What are the key differences between strong consistency and eventual consistency?"

You should see the system retrieve only ~300-500 tokens of context (the 3 most relevant chunks) and produce a grounded, coherent answer with section citations. This is the document that broke naive_qa.py, but RAG handles it because it never sends the full text.

Why does RAG solve the context window problem?

RAG decouples document size from prompt size. No matter how large your document grows, the LLM only ever sees the top 3 most relevant chunks. This keeps the prompt well within the context window while still providing focused, relevant information.

Your document intelligence system can now handle documents of any size. Next up, you will build an autonomous agent that decides what to search for, issues multiple queries, and reasons across results before answering.

Build the Agentic Tool-Calling Loop

Your RAG pipeline retrieves the top 3 chunks for a single query and sends them to the LLM. That works for straightforward questions, but complex questions need the system to search multiple times with different queries.

An agent solves this by reasoning about what to search for, issuing multiple searches with refined queries, and deciding when it has enough information to produce a final answer. In this step, you will build a tool-calling loop where the LLM autonomously calls your ChromaDB functions until it can synthesize a grounded response.

In this step, get ready to:

Define tool schemas in the OpenAI function-calling format for the agent to use.
Implement an agentic reasoning loop that calls tools, appends results, and iterates.
Test with multi-step questions that force the agent to search more than once before answering.

Define the agent's tool schemas

The agent needs a formal description of each tool it can call. LM Studio's /v1/chat/completions endpoint accepts a tools parameter in the same format as the OpenAI Function Calling API.

Create a new file called scripts/agent.py and add the file header and imports by pasting this code:

"""Document Intelligence Agent with tool-calling and RAG retrieval."""

import json
import sys

import chromadb
from openai import OpenAI

import docengine

What do these imports do?

json parses the function arguments the LLM returns as JSON strings.
chromadb connects to your existing vector index.
OpenAI talks to LM Studio's compatible endpoint.
docengine is your C++ preprocessing module (imported for potential use by tools).

Below the imports, add the TOOLS list with the search_documents tool definition by pasting this code:

TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "search_documents",
            "description": (
                "Search the document index for chunks relevant to a query. "
                "Use this when you need to find information in the documents."
            ),
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {
                        "type": "string",
                        "description": "The search query to find relevant document sections",
                    },
                    "n_results": {
                        "type": "integer",
                        "description": "Number of results to return (default 3)",
                    },
                },
                "required": ["query"],
            },
        },
    },
]

What is this schema telling the LLM?

Each entry in TOOLS describes one function the agent can call. The description field tells the LLM when to use it. The parameters object tells it what arguments to provide.

The LLM reads these schemas and autonomously decides whether to call a tool or produce a final answer on each iteration.

Now add a second tool to the list. In scripts/agent.py, find the closing ] of the TOOLS list.
Find this line:

    },
]

Replace it with the count_chunks tool definition followed by the closing bracket, so the end of TOOLS looks like this:

    },
    {
        "type": "function",
        "function": {
            "name": "count_chunks",
            "description": "Get the total number of indexed document chunks.",
            "parameters": {
                "type": "object",
                "properties": {},
            },
        },
    },
]

Implement the tool functions

Each tool schema needs a matching Python function that the agent loop will call when the LLM requests that tool.

Below the TOOLS list, add the search_documents and count_chunks implementations plus the dispatch dictionary by pasting this code:

def search_documents(query: str, n_results: int = 3) -> str:
    # Query ChromaDB for relevant chunks
    client = chromadb.PersistentClient(path="./chroma_db")
    collection = client.get_collection(name="documents")

    results = collection.query(query_texts=[query], n_results=n_results)

    # Format results with relevance scores
    output = []
    for i, (doc, dist) in enumerate(
        zip(results["documents"][0], results["distances"][0])
    ):
        output.append(f"[Result {i+1}, relevance: {1-dist:.2f}]\n{doc}")

    return "\n\n".join(output)


def count_chunks() -> str:
    client = chromadb.PersistentClient(path="./chroma_db")
    collection = client.get_collection(name="documents")
    return f"Total indexed chunks: {collection.count()}"


# Maps tool names to their Python implementations
TOOL_FUNCTIONS = {
    "search_documents": search_documents,
    "count_chunks": count_chunks,
}

What do these functions do?

search_documents queries your existing ChromaDB collection and formats results with relevance scores so the LLM can reason about quality.
count_chunks returns the total number of indexed chunks, giving the agent awareness of the index size.
TOOL_FUNCTIONS is a dispatch dictionary that maps string tool names (returned by the LLM) to their actual callable implementations.

Save scripts/agent.py and confirm Python can parse it by running:

python -c "exec(open('scripts/agent.py').read())"

You should see no output and no errors. If you see a SyntaxError, double-check your brackets and commas in the TOOLS list.

Seeing a SyntaxError or IndentationError?

Confirm every opening { in TOOLS has a matching } with a trailing comma.
Make sure the TOOL_FUNCTIONS dict is at the module level (no indentation), not nested inside a function.
help me debug a SyntaxError in my agent.py tool definitions

Wire up the agent reasoning loop

The agent loop is the core of the system. It sends messages to the LLM with the tools parameter, checks if the response contains tool calls, executes them, appends results back to the conversation, and repeats until the model produces a final text answer.

Below TOOL_FUNCTIONS, start the run_agent function by pasting its setup section:

def run_agent(question: str, max_steps: int = 5):
    # Connect to LM Studio's local server
    client = OpenAI(base_url="http://localhost:1234/v1", api_key="lm-studio")

    # Build the initial conversation with system instructions
    messages = [
        {
            "role": "system",
            "content": (
                "You are a document intelligence agent. You answer questions by "
                "searching through indexed documents using your tools. "
                "Always search for relevant information before answering. "
                "If you need more context, search again with a different query. "
                "Cite which results your answer is based on."
            ),
        },
        {"role": "user", "content": question},
    ]

    print(f"\n{'='*60}")
    print(f"Question: {question}")
    print(f"{'='*60}")

What does this setup do?

The system message instructs the LLM to always search before answering and to refine queries if needed. This nudges the model toward multi-step reasoning rather than guessing from its training data.

max_steps caps the loop at 5 iterations to prevent infinite tool-call cycles.

Directly below the print statements (still inside run_agent), add the for loop that sends each request and handles tool calls:

    for step in range(max_steps):
        print(f"\n--- Step {step + 1} ---")

        response = client.chat.completions.create(
            model="meta-llama-3.1-8b-instruct",
            messages=messages,
            tools=TOOLS,
            temperature=0.1,
        )

        message = response.choices[0].message

        if message.tool_calls:
            messages.append(message)

            for tool_call in message.tool_calls:
                func_name = tool_call.function.name
                func_args = json.loads(tool_call.function.arguments)

                print(f"  Tool call: {func_name}({func_args})")

                func = TOOL_FUNCTIONS.get(func_name)
                if func:
                    result = func(**func_args)
                else:
                    result = f"Error: Unknown tool '{func_name}'")

                print(f"  Result preview: {result[:100]}...")

How does the tool-calling loop work?

The LLM receives the conversation plus tool schemas. If it decides to call a tool, the response includes tool_calls with the function name and arguments.
json.loads deserializes the arguments string the LLM returns into a Python dict.
TOOL_FUNCTIONS dispatches the call to the right Python function.
The console prints let you watch the agent's reasoning process in real time.

Still inside the for tool_call loop, add the code that appends tool results back to the conversation. Then add the else branch that handles the final answer:

                messages.append(
                    {
                        "role": "tool",
                        "tool_call_id": tool_call.id,
                        "content": result,
                    }
                )
        else:
            print(f"\n{'='*60}")
            print(f"Final Answer:\n{message.content}")
            print(f"{'='*60}")
            return message.content

    print("\n[Agent] Max steps reached without final answer.")
    return None

What is the tool result message doing?

After executing a tool call, the result is appended with role: "tool" and the matching tool_call_id. This tells the LLM which function produced that output. The LLM then re-evaluates: it can call another tool or produce a final answer.

If the model responds WITHOUT tool calls, the loop ends and the final text is printed. If it never stops calling tools, max_steps terminates the loop after 5 iterations.

Finally, add the entry point at the bottom of the file:

def main():
    if len(sys.argv) < 2:
        print("Usage: python agent.py <question>")
        print("  (Make sure you have run build_index.py first)")
        sys.exit(1)

    question = " ".join(sys.argv[1:])
    run_agent(question)


if __name__ == "__main__":
    main()

Save scripts/agent.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved scripts/agent.py and that the file is about 130 lines long.

ⓧ I'd like to double check the full code

Your scripts/agent.py should have these sections in order. Scroll through and confirm each one is present:

Docstring and imports (json, sys, chromadb, OpenAI, docengine).
TOOLS list with two entries: search_documents and count_chunks.
search_documents() function querying ChromaDB.
count_chunks() function returning collection count.
TOOL_FUNCTIONS dictionary mapping names to callables.
run_agent() with messages setup, for loop, tool-call handling, else branch, and max-steps fallback.
main() parsing sys.argv and calling run_agent().
if __name__ == "__main__" guard at the very end.

Test with a comparison question

A comparison question forces the agent to search multiple times because the relevant information lives in different chunks of the document.

Make sure the LM Studio local server is still running at http://localhost:1234 with your model loaded.

Before you run this, do you think the agent will answer in one step or need multiple tool calls for a comparison question?

Run the agent with a comparison question by executing:

python scripts/agent.py "What are the key differences between Paxos and Raft consensus algorithms mentioned in the document?"

You should see the agent make 2 or more tool calls in the step-by-step console output before producing a final synthesized answer. Each step shows the tool name, arguments, and a preview of the results returned.

What should the output look like?

You will see output like:

Step 1: the agent calls search_documents with a query about Paxos or consensus.
Step 2: the agent searches again with a different or refined query to gather more context.
Final Answer: a synthesized response citing which results informed it.

The exact queries the agent chooses vary between runs. The key observation is that it makes multiple autonomous decisions rather than stopping after one search.

Agent only makes one tool call then answers?

Try a more specific comparison question that names two distinct topics from different sections of the document.
Check that the system prompt in your code includes the line about searching again with a different query if more context is needed.
The model's behavior is probabilistic. Run it a second time or try a different multi-faceted question.
help me get my agent to make multiple tool calls instead of just one

Test with a multi-hop question

A multi-hop question requires the agent to refine its search strategy across steps. The first search gives partial context, and the agent must decide what is still missing.

Run the agent with a question that requires connecting information from multiple sections:

python scripts/agent.py "How do the fault tolerance patterns in the document relate to the message queue architecture described later?"

Watch the console output. The agent should search for fault tolerance patterns first, then search separately for message queue content, and finally synthesize a connection between the two.

Why is this better than a single RAG retrieval?

Your rag_qa.py script from the previous step searches once with the raw user question. For a multi-hop question, the single search might return chunks about only one of the two topics.

The agent loop solves this by reasoning about what information is still missing after each retrieval, then issuing targeted follow-up searches. The result is a more complete, grounded answer that covers all aspects of the question.

Seeing 'Max steps reached without final answer'?

This means the agent used all 5 iterations calling tools without producing a final answer. The model may be stuck in a loop.
Try a simpler question first to confirm the loop works, then re-try the complex question.
Check that your documents/sample.txt actually contains content about both topics you are asking about.
help me fix my agent that hits max steps without producing an answer

Your document intelligence agent is now fully operational. It autonomously searches, retrieves, and reasons over your indexed documents using multi-step tool calling. You have a complete local system: C++ preprocessing for speed, ChromaDB for semantic retrieval, and an agentic LLM loop that ties it all together.

Secret mission

Add a Performance Benchmarking Tool

Your agent can search and reason over documents, but it has no idea how fast its own preprocessing engine is. Give it a benchmarking tool that pits C++ against pure Python in a head-to-head speed test, so it can report its own performance advantage in real-time.

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

Python virtual environment with pybind11, chromadb, and openai packages (~500MB including embedding model).
LM Studio local server (running in memory while active).
./chroma_db/ directory containing persistent vector database.
Project folder on your Desktop with cpp/, scripts/, and documents/ subdirectories.

Keep everything running

No action needed. Choose this if you're still actively building or want to experiment with different documents and questions.

Your virtual environment stays active for the current terminal session.
LM Studio's server continues running until you close it.
The ChromaDB database persists on disk and is ready for new queries.
No ongoing costs since everything is local.

Pause - I'll come back to this later

Shut down running processes to free up memory. Your files and database persist on disk.

In LM Studio, go to the Developer tab and toggle Start server off to stop the local server.
Deactivate your virtual environment by running:

deactivate

When you return, reactivate the virtual environment and restart LM Studio's server to pick up where you left off.

Delete - I don't want to use this again

Remove all project resources and start fresh. This cannot be undone.

In LM Studio, go to the Developer tab and toggle Start server off to stop the local server.
Deactivate the virtual environment by running:

deactivate

Delete the ChromaDB database by running:

rm -rf ./chroma_db

On Windows, use this instead:

rmdir /s /q chroma_db

Navigate to your Desktop and delete the entire project folder by running:

cd ~/Desktop
rm -rf doc-intelligence-agent

On Windows, use this instead:

cd ~\Desktop
rmdir /s /q doc-intelligence-agent

This removes all project files including the virtual environment, C++ source, compiled extension, scripts, and sample documents.

Nice Work!

You just built a hybrid C++/Python document intelligence system from scratch. Your agent can autonomously search, retrieve, and reason over any document you feed it, all running locally with zero API costs.

Here's what you learned:

Built a C++ text preprocessing engine with pybind11 that performs text normalization, paragraph splitting, token estimation, and overlapping chunk generation with OpenMP parallelism.
Connected to LM Studio's local Llama 3.1 8B model via the OpenAI-compatible API.
Experienced firsthand why naive context stuffing fails and how RAG solves it with ChromaDB vector search.
Built an autonomous agent loop with multi-step tool calling that searches, retrieves, and reasons over documents.
(Secret Mission) Added a benchmarking tool demonstrating the C++ engine's performance advantage over pure Python.

Ready to quiz yourself?