Build a Self-Hosted RAG Audit Agent

Build a private RAG pipeline that audits financial documents with a local LLM.

Introduction

30 Second Summary

Financial auditing involves sifting through invoices, receipts, and reports to verify every dollar amount and flag anything out of place. When you hand that job to a cloud AI service, your sensitive financial data travels across the internet to someone else's servers.

In this project, you will build a self-hosted document audit agent that ingests financial documents, answers audit questions with source-cited evidence, and returns structured JSON findings. The entire pipeline runs on your Windows server with zero cloud dependencies.

What You'll Build

You type an audit question like "What is the total amount on Invoice 1001?" and your agent pulls the exact dollar amount from the right document, returning a structured finding with the invoice number, date, and compliance status.

By the end of this project, you'll have:

Ask audit questions like "What is the total amount on Invoice 1001?" and get source-cited answers pulled from your documents through a RAG pipeline.
Restart the script and have your document index load instantly from a persistent vector store powered by ChromaDB, with no re-embedding needed.
Receive structured JSON audit findings with machine-readable invoice numbers, amounts, dates, and compliance flags instead of free-form paragraphs.
Secret Mission: Add metadata filtering to scope queries by document type and build an automated audit checklist that runs multiple structured queries in sequence.

Do I need any prerequisites?

This project assumes you have Python 3.10 or higher installed on a Windows server or PC. You should also be comfortable writing and running basic Python scripts.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters for your use case. This is your moment to commit to the goal before any hands-on work begins.

Set Up Ollama and the Python Environment

You've committed to building a self-hosted audit agent that keeps all your financial data on your own server. Now it's time to lay the foundation.

Your audit agent needs a local LLM to generate answers and an embedding model to understand your documents. Ollama provides both, running entirely on your hardware. In this step, you'll install Ollama, pull the models, and set up a Python environment with the libraries that tie everything together.

In this step, get ready to:

Install Ollama and verify it runs on your server.
Pull the LLM and embedding models that power the audit agent.
Create a Python virtual environment and install the required packages.

Install Ollama on your Windows server

Ollama is an open-source tool that runs large language models directly on your hardware. It handles model downloads, serves a local API, and keeps your data completely private.

Install Ollama by running this command in PowerShell:

irm https://ollama.com/install.ps1 | iex

What does this command do?

irm (short for Invoke-RestMethod) downloads the Ollama install script from the official site. iex (short for Invoke-Expression) runs it immediately.

The installer runs silently in the background. It does not require Administrator rights and installs in your home directory by default.

After installation completes, Ollama starts running in the background automatically. It serves its API on http://localhost:11434.

Prefer a GUI installer?

You can also download OllamaSetup.exe directly from ollama.com/download/windows and run it.

Verify Ollama is installed by running:

ollama -v

✔️ I see a version number

Ollama is installed and ready. You'll pull the AI models in the next substep.

ⓧ Command not found

Close PowerShell completely and reopen it. The installer updates your PATH, but existing terminals don't pick up the change.
Run ollama -v again in the fresh window.
If you still see an error, download OllamaSetup.exe from ollama.com/download/windows and run it manually.

Pull the LLM and embedding model

You need two models for the audit agent. The LLM generates answers from retrieved document context. The embedding model converts text into numerical vectors that enable similarity search.

Pull both models by running these commands one at a time:

ollama pull llama3.2:3b

You'll see a progress bar as the model downloads. This model is around 2GB, so it may take a few minutes depending on your connection speed.

Once llama3.2:3b finishes, pull the embedding model by running:

ollama pull nomic-embed-text

Why these two models?

llama3.2:3b is a 3-billion parameter LLM that runs well on CPU with 8GB or more of RAM. It's small enough to be fast on modest hardware while still generating useful audit answers.

nomic-embed-text is a lightweight embedding model (around 274MB) that converts document text into vectors. These vectors are what your agent searches through to find relevant context for each query.

Before you run the next command, what do you think will happen when you ask a local AI model to say hello?

Test that the LLM responds correctly by running:

ollama run llama3.2:3b "Say hello"

The model generates a greeting directly from your server. This confirms your local LLM is fully operational with no cloud connection required.

Model not responding or very slow?

Make sure the ollama pull llama3.2:3b command finished completely before running the test. If the download was interrupted, re-run the pull command.
The first time a model runs, it loads into memory, which can take 10-30 seconds. Subsequent runs are faster.
If your server has less than 8GB of RAM, the model may run very slowly or fail to load.

Help me troubleshoot why Ollama is not responding to my prompt on Windows.

Create your project folder and install Python packages

With Ollama serving models locally, you need a Python project to connect to them. You'll create a project folder, set up an isolated virtual environment, and install the LlamaIndex and ChromaDB packages that power the RAG pipeline.

Move to your Desktop and create your project folder by running these commands:

cd ~/Desktop
mkdir audit-agent
cd audit-agent

Create a Python virtual environment by running:

python -m venv audit-agent-env

Activate the virtual environment by running:

.\audit-agent-env\Scripts\Activate.ps1

Your PowerShell prompt now shows (audit-agent-env) at the beginning. This confirms the virtual environment is active and packages will install in isolation.

Seeing a script execution error?

If PowerShell says the script cannot be loaded because running scripts is disabled on this system, your execution policy is blocking it.

Run this command to allow scripts for your current session only:

Set-ExecutionPolicy -Scope Process -ExecutionPolicy RemoteSigned

Then try the activation command again.

Help me fix the PowerShell execution policy so I can activate my Python virtual environment.

Install all the required Python packages by running:

pip install llama-index-core llama-index-llms-ollama llama-index-embeddings-ollama llama-index-vector-stores-chroma chromadb

What are these packages?

llama-index-core is the core framework for building RAG pipelines. It handles document loading, chunking, indexing, and querying.
llama-index-llms-ollama and llama-index-embeddings-ollama connect LlamaIndex to your local Ollama models.
llama-index-vector-stores-chroma and chromadb provide the vector store where your document embeddings are stored and searched.

You'll see download progress for each package. This may take a minute or two.

Before you run the next command, do you think pip will find the llama-index-core package?

Confirm the packages installed successfully by running:

python -m pip show llama-index-core

You should see package details including the name llama-index-core and a version number. This confirms all your Python dependencies are installed and ready.

Package not found or install failed?

Make sure your virtual environment is active. Your prompt should start with (audit-agent-env). If it doesn't, run the activation command again.
If the install failed with a network error, check your internet connection and re-run the pip install command.
Make sure you are using Python 3.10 or higher. Run python --version to check.

Help me troubleshoot why my pip install failed for llama-index packages in my virtual environment.

Your environment is locked and loaded. Ollama is serving two models locally, and your Python environment has everything it needs. Next up, you'll create sample financial documents and run your first audit query.

Ingest Financial Documents and Run Your First Audit Query

Your local AI backbone is live. Ollama is serving both the llama3.2:3b LLM and the nomic-embed-text embedding model on your Windows server, and your Python environment has every package you need.

Before worrying about persistence or structured output, you need to see the full RAG pipeline work end-to-end. In this step, you will create sample financial documents, ingest them into a vector store index, and run your first audit query to get real answers pulled directly from your documents.

In this step, get ready to:

Create sample financial documents that simulate invoices and a quarterly report.
Build a RAG script that ingests, chunks, and queries those documents using your local LLM.
Run your first audit query and see source-cited answers with real dollar amounts.

Create sample financial documents

A RAG pipeline needs documents to work with. You will create three plain text files that simulate real financial records: two invoices with line items and totals, and one quarterly summary that references both invoices.

Create a data/ folder inside your project directory by running this command:

mkdir data

Create a new file called invoice_1001.txt inside the data/ folder. Open it in your text editor and paste in the following content:

INVOICE
Invoice Number: INV-1001
Date: 2026-07-15
Vendor: Acme Supplies Ltd.
Bill To: NextCorp Financial Services

Line Items:
1. Office Furniture - Qty: 10 - Unit Price: $250.00 - Total: $2,500.00
2. Computer Monitors - Qty: 5 - Unit Price: $450.00 - Total: $2,250.00
3. Printer Cartridges - Qty: 20 - Unit Price: $35.00 - Total: $700.00

Subtotal: $5,450.00
Tax (8%): $436.00
Total Due: $5,886.00

Payment Terms: Net 30
Due Date: 2026-08-14
Status: Unpaid

Save invoice_1001.txt.
Create a second file called invoice_1002.txt inside data/ and paste in this content:

INVOICE
Invoice Number: INV-1002
Date: 2026-07-22
Vendor: TechParts International
Bill To: NextCorp Financial Services

Line Items:
1. Server Rack Unit - Qty: 2 - Unit Price: $1,200.00 - Total: $2,400.00
2. Network Cables (Cat6) - Qty: 50 - Unit Price: $12.00 - Total: $600.00
3. UPS Battery Backup - Qty: 3 - Unit Price: $380.00 - Total: $1,140.00

Subtotal: $4,140.00
Tax (8%): $331.20
Total Due: $4,471.20

Payment Terms: Net 45
Due Date: 2026-09-05
Status: Paid

Save invoice_1002.txt.
Confirm both invoice files exist by running:

ls data/

You should see invoice_1001.txt and invoice_1002.txt listed in the output.

Create one more file called q3_2026_summary.txt inside data/ and paste in this content:

QUARTERLY FINANCIAL SUMMARY
Period: Q3 2026 (July - September)
Prepared by: Finance Department, NextCorp Financial Services

PROCUREMENT OVERVIEW:
Total procurement spend for Q3 2026: $10,357.20
Number of invoices processed: 2

Invoice Summary:
- INV-1001 (Acme Supplies Ltd.): $5,886.00 - Office supplies and equipment. Status: Unpaid, due 2026-08-14.
- INV-1002 (TechParts International): $4,471.20 - IT infrastructure components. Status: Paid in full on 2026-08-01.

BUDGET ANALYSIS:
Q3 procurement budget: $15,000.00
Actual spend: $10,357.20
Remaining budget: $4,642.80
Budget utilization: 69.05%

COMPLIANCE NOTES:
- All invoices received proper three-way match (PO, receipt, invoice).
- INV-1001 is approaching due date. Payment should be prioritized.
- No procurement policy violations detected this quarter.

VENDOR PERFORMANCE:
- Acme Supplies Ltd.: Delivery on time. Quality satisfactory.
- TechParts International: Delivery delayed by 3 days. Quality satisfactory. Vendor notified.

Save q3_2026_summary.txt.
Verify all three documents are in place by running:

ls data/

You should now see invoice_1001.txt, invoice_1002.txt, and q3_2026_summary.txt listed.

Why plain text files instead of PDFs?

Plain text files keep the project focused on the RAG pipeline itself, without needing extra PDF-parsing libraries. LlamaIndex's SimpleDirectoryReader handles PDFs natively, so you can swap in real documents later without changing any code.

Don't see all three files?

Make sure you created each file inside the data/ folder, not in the project root. The file names must match exactly: invoice_1001.txt, invoice_1002.txt, and q3_2026_summary.txt.

Help me check my data folder structure.

Write the core RAG script

Now you will build the script that ties everything together. It loads your documents, splits them into chunks, converts those chunks into vector embeddings, builds a searchable index, and queries that index with natural language audit questions.

Create a new file called audit_agent.py in your project folder (next to the data/ folder, not inside it). Add the imports and configuration:

import chromadb
import json

from llama_index.core import SimpleDirectoryReader, VectorStoreIndex, StorageContext
from llama_index.core.node_parser import SentenceSplitter
from llama_index.llms.ollama import Ollama
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.vector_stores.chroma import ChromaVectorStore
from llama_index.core.bridge.pydantic import BaseModel

# --- Configuration ---
DATA_DIR = "./data"
LLM_MODEL = "llama3.2:3b"
EMBED_MODEL = "nomic-embed-text"
OLLAMA_URL = "http://localhost:11434"

# --- Initialize LLM and embedding model ---
llm = Ollama(
    model=LLM_MODEL,
    base_url=OLLAMA_URL,
    temperature=0.0,
    request_timeout=120.0,
    context_window=8000,
)

embed_model = OllamaEmbedding(
    model_name=EMBED_MODEL,
    base_url=OLLAMA_URL,
)

What does this code do?

The imports bring in LlamaIndex's document reader, index builder, text splitter, and the Ollama integrations. Some imports (chromadb, StorageContext, BaseModel) are not used yet. You will use them in later steps.
The configuration section sets your document folder path, model names, and Ollama's local API address.
Ollama() connects to your local LLM for answering queries. temperature=0.0 makes responses deterministic, which is important for consistent audit results.
OllamaEmbedding() connects to your local embedding model, which converts text into numerical vectors for similarity search.

Add the document loading, indexing, and query logic below the code you just wrote in audit_agent.py:

# --- Load documents and build in-memory index ---
documents = SimpleDirectoryReader(DATA_DIR).load_data()
splitter = SentenceSplitter(chunk_size=512, chunk_overlap=50)
nodes = splitter.get_nodes_from_documents(documents)

index = VectorStoreIndex(nodes, embed_model=embed_model)

query_engine = index.as_query_engine(
    llm=llm,
    similarity_top_k=3,
)

# --- Run audit queries ---
audit_queries = [
    "What is the total amount due on Invoice INV-1001 and what is its payment status?",
    "What is the total procurement spend for Q3 2026 and how does it compare to the budget?",
    "Are there any compliance issues or vendor performance concerns noted in the quarterly summary?",
]

print("\n" + "=" * 60)
print("FINANCIAL DOCUMENT AUDIT REPORT")
print("=" * 60)

for i, query in enumerate(audit_queries, 1):
    print(f"\n--- Audit Query {i} ---")
    print(f"Question: {query}")
    response = query_engine.query(query)
    print(f"Finding: {response}")
    print()

What does this code do?

SimpleDirectoryReader loads every file in your data/ folder as a document object.
SentenceSplitter breaks each document into overlapping chunks of 512 characters. The 50-character overlap ensures important context is not split across chunk boundaries.
VectorStoreIndex takes those chunks, sends each one through the nomic-embed-text model to get a vector embedding, and stores them in memory for fast similarity search.
as_query_engine() creates a query engine that retrieves the 3 most relevant chunks (similarity_top_k=3) and passes them as context to the LLM.
The audit queries loop sends three different questions to the engine and prints each finding.

Save audit_agent.py.

✔️ Awesome, I've got everything!

Great. Make sure you have saved audit_agent.py before moving on.

ⓧ I'd like to double check the full code

Your audit_agent.py is built from two code blocks. Scroll up and confirm each section is present in this order:

Imports and configuration (starting with import chromadb, ending with the OllamaEmbedding initialization).
Document loading, indexing, and audit queries (starting with the # --- Load documents comment, ending with the for loop that prints each finding).

Run the script and observe the response

Your script is ready to run. It will read all three documents, embed every chunk through Ollama, build the index, and then fire three audit queries against it.

This will take a minute or two

The script embeds all document chunks and runs three LLM queries. On a CPU-only server, expect it to take 1-3 minutes depending on your hardware. You will see output appear as each query completes.

Before you run this, think about what you expect the agent to return for the first query about Invoice INV-1001's total amount. Do you think it will pull the correct dollar figure from the invoice?

Run the audit agent by running this command:

python audit_agent.py

You should see a FINANCIAL DOCUMENT AUDIT REPORT header followed by three audit findings. The first query should return the total amount of $5,886.00 from Invoice INV-1001, pulled directly from the document text. The second query should reference the $10,357.20 procurement spend and the $15,000.00 budget. The third should mention the vendor delivery delay.

What is happening under the hood?

When you run a query, the engine converts your question into a vector using nomic-embed-text. It then searches the index for the 3 most similar document chunks. Those chunks are passed as context to llama3.2:3b, which generates an answer grounded in the retrieved text.

This is the core of Retrieval-Augmented Generation: the LLM does not hallucinate financial data from its training. It answers using your actual documents as evidence.

Seeing a connection error?

If you see an error mentioning a refused connection to localhost:11434, Ollama is not running. Check your system tray for the Ollama icon. If it is not there, relaunch Ollama from the Start menu.

If you see a model-not-found error, confirm the models are pulled by running ollama list in a separate PowerShell window. You should see both llama3.2:3b and nomic-embed-text in the list.

If you see a file-not-found error for the data/ folder, make sure your PowerShell session is in your project folder (the one that contains audit_agent.py and data/).

Help me debug my audit agent script.

Your RAG pipeline is working end-to-end. Documents go in, audit answers come out, and everything runs on your local server with zero cloud calls. Next up, you will tackle a problem you might have already noticed: every time you run the script, it re-embeds all documents from scratch. You will fix that with persistent storage.

Add Persistent Storage with ChromaDB

Your audit agent works end-to-end. It loads your financial documents, embeds them, retrieves relevant chunks, and returns answers to audit queries.

There is one problem. Every time you run the script, it re-reads and re-embeds every document from scratch. For three small text files that takes a few seconds. For a real audit workflow with hundreds of invoices and reports, that could take minutes or hours.

In this step, get ready to:

Replace the in-memory vector store with a persistent ChromaDB store that saves to disk.
Add cache-or-build logic so the index loads instantly on subsequent runs.
Verify persistence by running the script multiple times and comparing behavior.

Experience the re-embedding delay

Before you fix anything, experience the problem firsthand. Your current script rebuilds the entire index every time it runs.

Before you run the next command, predict: will the script process documents any faster the second time, or will it take the same amount of time?
Run your script again by running this command:

python audit_agent.py

The script takes just as long as the first time. It re-reads every file, re-chunks every document, and re-generates every embedding from scratch. Nothing is cached or reused.

This is the gap you are closing now. You will store the index in a persistent ChromaDB database so it only builds once.

Add ChromaDB configuration and persistent storage

ChromaDB's PersistentClient saves your vector index to a folder on disk. On the next run, the script loads the existing index instead of re-embedding everything.

In audit_agent.py, find the configuration section at the top of the file.
Find this line:

DATA_DIR = "./data"
LLM_MODEL = "llama3.2:3b"

Add two new configuration lines between DATA_DIR and LLM_MODEL, so the config section now looks like this:

# --- Configuration ---
DATA_DIR = "./data"
CHROMA_DIR = "./chroma_db"
COLLECTION_NAME = "financial_audit"
LLM_MODEL = "llama3.2:3b"
EMBED_MODEL = "nomic-embed-text"
OLLAMA_URL = "http://localhost:11434"

Now replace the in-memory index section with the ChromaDB-backed version.

Scroll down past the embed_model initialization and find this section:

# --- Load documents and build in-memory index ---
documents = SimpleDirectoryReader(DATA_DIR).load_data()
splitter = SentenceSplitter(chunk_size=512, chunk_overlap=50)
nodes = splitter.get_nodes_from_documents(documents)

index = VectorStoreIndex(nodes, embed_model=embed_model)

query_engine = index.as_query_engine(
    llm=llm,
    similarity_top_k=3,
)

Delete everything from the # --- Load documents and build in-memory index --- comment down to and including index = VectorStoreIndex(nodes, embed_model=embed_model). Keep the query_engine lines below it.
In the same spot where you deleted the old code, paste this replacement:

# --- Build or load the ChromaDB index ---
chroma_client = chromadb.PersistentClient(path=CHROMA_DIR)

try:
    chroma_collection = chroma_client.get_collection(COLLECTION_NAME)
    print(f"Loaded existing collection '{COLLECTION_NAME}' from disk.")
    vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
    index = VectorStoreIndex.from_vector_store(
        vector_store,
        embed_model=embed_model,
    )
except Exception:
    print(f"Collection '{COLLECTION_NAME}' not found. Building new index...")
    chroma_collection = chroma_client.create_collection(COLLECTION_NAME)
    vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
    storage_context = StorageContext.from_defaults(vector_store=vector_store)

    documents = SimpleDirectoryReader(DATA_DIR).load_data()
    splitter = SentenceSplitter(chunk_size=512, chunk_overlap=50)
    nodes = splitter.get_nodes_from_documents(documents)

    index = VectorStoreIndex(
        nodes,
        storage_context=storage_context,
        embed_model=embed_model,
    )
    print(f"Indexed {len(nodes)} chunks into '{COLLECTION_NAME}'.")

What does this code do?

This block uses a try/except pattern to decide whether to build a new index or load an existing one.

chromadb.PersistentClient(path=CHROMA_DIR) creates a ChromaDB client that reads from and writes to the chroma_db/ folder on disk.
The try block calls get_collection() to check if the collection already exists. If it does, the index loads from disk without re-embedding a single document.
The except block runs only when the collection does not exist yet. It creates the collection, loads all documents, splits them into chunks, embeds them, and saves everything to disk.
StorageContext connects the new VectorStoreIndex to the ChromaDB collection so all embeddings are persisted automatically.

Make sure the query_engine block sits right after the new ChromaDB section. It should look like this:

query_engine = index.as_query_engine(
    llm=llm,
    similarity_top_k=3,
)

Save audit_agent.py.
Run the updated script by running this command:

python audit_agent.py

You should see the message Collection 'financial_audit' not found. Building new index... followed by Indexed X chunks into 'financial_audit'. at the top of the output. The audit queries still return answers just like before.

Confirm the chroma_db/ folder was created by running:

ls chroma_db

You should see files inside the folder. This is your persistent vector store on disk.

Not seeing the index build message?

Make sure you deleted the entire old in-memory index section, from the comment # --- Load documents and build in-memory index --- through index = VectorStoreIndex(nodes, embed_model=embed_model).
Check that CHROMA_DIR and COLLECTION_NAME are defined in your configuration section above the LLM initialization.
Verify that Ollama is still running in the background. The embedding step needs the nomic-embed-text model to be available.
help me debug why my ChromaDB index is not building

Verify persistence across restarts

The whole point of this change is that the second run should skip embedding entirely. Time to prove it.

Before you run the script again, predict: will you see the "Building new index" message, or a different message?
Run the script again by running this command:

python audit_agent.py

This time you should see Loaded existing collection 'financial_audit' from disk. instead of the build message. The script skips the entire embedding process and jumps straight to answering queries.

Why is this faster?

On the first run, your script sent every document chunk to the nomic-embed-text model to generate vector embeddings. That is the expensive step. On the second run, those embeddings already exist in the chroma_db/ folder, so the script loads them directly from disk.

For three small documents the speed difference is a few seconds. For hundreds of financial reports, it is the difference between minutes of waiting and an instant start.

Now verify that the persistence survives a full rebuild cycle.

Delete the chroma_db/ folder to simulate starting fresh by running this command:

Remove-Item -Path .\chroma_db -Recurse -Force

Run the script one more time by running this command:

python audit_agent.py

You should see the build message again: Collection 'financial_audit' not found. Building new index.... The script detects the missing collection, rebuilds everything from scratch, and saves it to a fresh chroma_db/ folder.

Still seeing the load message after deleting?

Confirm the chroma_db/ folder is actually gone by running ls in your project directory. If it still appears, the delete command may not have run successfully.
Make sure you ran Remove-Item from your project folder, not from a different directory.
help me verify that my ChromaDB folder was deleted and the index rebuilds

✔️ Awesome, I've got everything!

Great. Double check you have saved audit_agent.py and that a chroma_db/ folder exists in your project directory after the last run.

ⓧ I'd like to double check the full code

Scroll up through your step and confirm these sections are present in order:

The configuration section with DATA_DIR, CHROMA_DIR, COLLECTION_NAME, LLM_MODEL, EMBED_MODEL, and OLLAMA_URL.
The LLM and embedding model initialization (unchanged from Step 2).
The ChromaDB try/except block with PersistentClient, get_collection, and create_collection.
The query_engine and audit queries sections (unchanged from Step 2).

Your index now persists to disk and loads instantly on restart. Next up, you will upgrade the agent to return structured JSON audit findings instead of free-form prose paragraphs.

Extract Structured Audit Findings

Your audit agent now loads its index from disk in an instant. No more waiting for documents to re-embed on every run.

There is still a gap, though. Right now the agent returns free-form prose paragraphs. An auditor processing hundreds of findings needs machine-readable data, not paragraphs to read through manually. In this step, you will add structured output so the agent returns JSON objects with specific fields like invoice numbers, amounts, dates, and compliance flags.

In this step, get ready to:

Define a Pydantic model that describes the exact shape of an audit finding.
Update the query engine to return structured JSON instead of prose.
Run multiple audit queries and verify each returns a structured JSON object.

Define a Pydantic model for audit findings

To get structured output from an LLM, you need to tell it exactly what shape the response should take. Pydantic models define that shape as a Python class with typed fields.

In audit_agent.py, find the # --- Configuration --- section.
Copy the following class definition directly below the OLLAMA_URL line, before the LLM initialization section:

# --- Pydantic model for structured audit output ---
class AuditFinding(BaseModel):
    """A structured audit finding extracted from financial documents."""
    invoice_number: str
    total_amount: str
    date: str
    finding: str
    compliance_flag: str

What does this class do?

The AuditFinding class inherits from BaseModel, which was already imported in the previous step. Each field is a string that the LLM must fill in when it responds.

invoice_number captures the invoice ID (e.g. INV-1001).
compliance_flag lets the LLM tag whether a finding is compliant, non-compliant, or needs attention.
LlamaIndex will use this class to force the LLM's response into these exact fields, so you get predictable data every time.

Save audit_agent.py.
The class is just a definition for now. The next substep wires it into the query engine.

Update the query engine to use structured output

Two changes connect the Pydantic model to the LLM. First, you tell Ollama to constrain its output to valid JSON. Then you wrap the LLM with your AuditFinding model so every response matches your schema.

In audit_agent.py, scroll down to the # --- Initialize LLM and embedding model --- section.
Find this line inside the Ollama() call:

    request_timeout=120.0,
    context_window=8000,

Add json_mode=True, between those two lines, so it now looks like this:

    request_timeout=120.0,
    json_mode=True,
    context_window=8000,

What does json_mode do?

Setting json_mode=True tells Ollama to constrain the LLM's output to valid JSON. Without it, the model might return a mix of prose and JSON that breaks your parser.

Now you need to replace the query engine section so it uses a structured LLM instead of the raw one.

Scroll down to the line that reads query_engine = index.as_query_engine( and find this entire block:

query_engine = index.as_query_engine(
    llm=llm,
    similarity_top_k=3,
)

Replace that entire block with the following code:

# --- Create structured query engine ---
structured_llm = llm.as_structured_llm(AuditFinding)

query_engine = index.as_query_engine(
    llm=structured_llm,
    similarity_top_k=3,
)

What does this code do?

llm.as_structured_llm(AuditFinding) wraps your LLM so that every response is forced into the AuditFinding schema. The model cannot return free-form text anymore.
The query engine now passes structured_llm instead of the raw llm. The retrieval logic stays the same. Only the output format changes.

Save audit_agent.py.

Seeing a red underline under AuditFinding?

Make sure the AuditFinding class is defined above the line that calls llm.as_structured_llm(AuditFinding). Python reads top to bottom, so the class must come first.
Check that from llama_index.core.bridge.pydantic import BaseModel is in your imports at the top of the file.
Help me debug my AuditFinding class setup.

Run multiple audit queries and compare outputs

The three audit queries from the previous step are still in your script. The only difference now is how the agent formats its answers.

Before you run the script, what do you think the output will look like? Will it still be prose paragraphs, or something different?

Run the updated script by executing this command in your PowerShell terminal:

python audit_agent.py

Each audit query now returns a structured JSON object with the fields you defined in AuditFinding. Instead of a prose paragraph, you see invoice_number, total_amount, date, finding, and compliance_flag populated with values pulled from your documents.

Why structured output matters for auditing

Free-form prose requires a human to read every response and manually extract the data points. Structured JSON can be piped into a spreadsheet, a database, or another script.

The compliance_flag field is especially useful. A downstream script could filter findings by flag and escalate anything marked non-compliant automatically.

Getting an error or seeing empty fields?

If you see a timeout error, the llama3.2:3b model may need more time on your hardware. Try increasing request_timeout from 120.0 to 240.0.
If some fields come back empty or contain placeholder text, this is normal LLM behavior with smaller models. The important thing is that the response is valid JSON matching your schema, not prose.
Make sure json_mode=True is inside the Ollama() constructor. Without it, the model may return a mix of JSON and prose.
Help me debug my structured audit output.

Compare the structured outputs across the three queries. Verify that the extracted amounts match the values in your source documents. For example, the first query should return $5,886.00 for Invoice INV-1001, matching the data/invoice_1001.txt file.

✔️ Awesome, I've got everything!

Great. Double check you have saved audit_agent.py.

ⓧ I'd like to double check the full code

The full audit_agent.py file is too long to show in one block. Scroll up through your file and confirm each section is present in this order:

Imports at the top (chromadb, json, all llama_index modules, and BaseModel).
Configuration section with DATA_DIR, CHROMA_DIR, COLLECTION_NAME, LLM_MODEL, EMBED_MODEL, and OLLAMA_URL.
The AuditFinding Pydantic class with five string fields.
LLM initialization with json_mode=True and embedding model initialization.
ChromaDB build-or-load logic with the try/except block.
Structured query engine created with structured_llm = llm.as_structured_llm(AuditFinding) and llm=structured_llm in the query engine.
Three audit queries and the print loop at the bottom.

Your audit agent now speaks JSON. Every query returns a clean, structured finding with the exact fields an auditor needs. Next up, you will take this further in the Secret Mission by adding metadata filtering and an automated audit checklist.

Secret mission

Add Metadata Filtering and an Audit Checklist

A real auditor does not run one query at a time. They have a checklist of items to verify across specific document types. In this mission, you will add metadata tags to your documents, scope queries to specific categories like invoices or reports, and build an automated multi-query audit checklist that outputs a full JSON report.

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally with no cloud resources or paid services, so there are no ongoing costs.

Resources you used:

Ollama application with llama3.2:3b (~2GB) and nomic-embed-text (~274MB) models stored at %HOMEPATH%\.ollama.
chroma_db/ folder containing your persistent vector store data.
audit-agent-env Python virtual environment with all installed packages.
Project files: audit_agent.py, audit_checklist.py, and the data/ folder with sample financial documents.

Keep everything running

No action needed. Choose this if you're still actively building or plan to add more financial documents to your audit agent.

Ollama continues running in the background and starts automatically with Windows. Your models stay downloaded and ready to use.
Your ChromaDB index persists on disk in the chroma_db/ folder. You can add new documents to data/ and re-run the indexing script to update the index.
Your virtual environment and all Python packages remain installed and ready.

Pause - I'll come back to this later

Shut down the Ollama background service to free up memory. Your models, vector store data, and project files all remain on disk.

Right-click the Ollama icon in the Windows system tray (bottom-right corner of your taskbar) and select Quit.
When you're ready to resume, relaunch Ollama from the Start menu. Your models and ChromaDB data will still be there.

Delete - I don't want to use this again

Remove all project resources and the Ollama application from your server.

Uninstall Ollama by opening Settings in Windows, selecting Apps, then Apps & features. Find Ollama in the list and click Uninstall.
Delete the downloaded models by removing the %HOMEPATH%\.ollama folder. Open PowerShell and run:

Remove-Item -Recurse -Force "$env:HOMEPATH\.ollama"

Delete the ChromaDB vector store, virtual environment, and all project files. Navigate to your project folder in PowerShell and run:

# First, deactivate the virtual environment if it is active
deactivate

# Move up to the parent folder (e.g. your Desktop)
cd ..

# Delete the entire project folder
Remove-Item -Recurse -Force .\audit-agent

Adjust the folder name

Replace audit-agent in the command above with whatever you named your project folder when you created it in Step 1.

Nice Work!

Nice work! You just built a fully self-hosted financial document audit agent that runs entirely on your private Windows server.

You've learned how to:

Build a RAG pipeline with Ollama and LlamaIndex that ingests financial documents, chunks and embeds them locally, and answers audit queries using a local LLM with zero cloud dependencies.
Add persistent storage with ChromaDB so your vector index survives script restarts and loads instantly from disk without re-embedding.
Implement structured JSON output using Pydantic models and JSON mode so the agent returns machine-readable audit findings instead of prose.
Secret Mission: Add metadata filtering to scope queries by document type and build an automated audit checklist that runs multiple structured queries in sequence.

Ready to quiz yourself?