Local RAG Chat App with Ollama

Build a local AI chat app that answers questions from your documents.

Introduction

30 Second Summary

AI chatbots like ChatGPT are powerful, but they can't read your private documents. What if you could build your own document Q&A assistant that runs entirely on your machine with zero cloud costs?

In this project, you will build a local AI chat app that lets you upload PDF, Word, or TXT files and ask questions about their content. Everything runs on your computer, so your data never leaves your machine.

What You'll Build

A Streamlit web app that processes documents into searchable chunks, stores them in a ChromaDB vector database, and uses a locally-running LLM via Ollama to answer questions based on the document content.

By the end of this project, you'll have:

  • A working Streamlit chat interface where you upload documents and ask questions about their content.
  • A RAG pipeline that chunks documents, creates vector embeddings, and retrieves relevant context for answer generation.
  • A fully local setup with Ollama running an LLM on your machine. No API keys or cloud costs required.
  • Secret Mission: Add conversation memory so your chatbot can handle follow-up questions that reference previous answers.

Are there any prerequisites?

You need basic Python knowledge (variables, functions, imports). No prior AI or machine learning experience is required.

Install Ollama and Pull a Model

To build a local RAG chat app, you need a language model running on your machine. Ollama is a tool that makes running large language models locally as easy as running a single terminal command.

In this step, you will install Ollama and download the Qwen3 8B model that your app will use to generate answers from document context.

In this step, get ready to:
  • Install Ollama on your computer.
  • Download the Qwen3 8B language model.
  • Test the model with a question to confirm it works.
Install Ollama

Ollama runs a local server on your machine that hosts language models. Once installed, it serves an API at http://localhost:11434 that your Python app will connect to later.

  • Open your terminal by pressing Cmd+Space (macOS) or the Windows key (Windows), typing Terminal (macOS) or PowerShell (Windows), and pressing Enter.

macOS

  • Install Ollama by running this command:
curl -fsSL https://ollama.com/install.sh | sh

What does this command do?

This downloads and runs the official Ollama install script. It places the ollama command in your system path so you can use it from any terminal window.

Requires macOS 14 Sonoma or later. This same command also works on Linux.

Windows

  • Install Ollama by running this command in PowerShell:
irm https://ollama.com/install.ps1 | iex

What does this command do?

This downloads and runs the official Ollama installer for Windows. It installs the Ollama application and makes the ollama command available in your terminal.

Requires Windows 10 22H2 or newer.

  • Verify Ollama is installed by running:
ollama list

You should see an empty table or a list of models. This confirms Ollama is running and ready.

Command not found?

If you see an error, try closing your terminal and opening a new one. The install script adds ollama to your system path, but existing terminal sessions may not pick up the change.

On Windows, make sure you ran the command in PowerShell (not Command Prompt).

help me troubleshoot my Ollama installation

Pull the Qwen3 8B model

Ollama hosts a library of open-source models. You need to download one before your app can generate answers. The Qwen3 8B model is a strong choice for question-answering tasks.

  • Download the model by running this command:
ollama pull qwen3:8b

What does this command do?

This downloads the Qwen3 8B model (approximately 5GB) to your local machine. The model files are stored in Ollama's data directory so you only need to download them once.

The download may take several minutes depending on your internet speed.

  • Wait for the download to complete. You will see a progress bar in your terminal.
  • Confirm the model is ready by running:
ollama list

You should see qwen3:8b listed in the output.

Download seems stuck?

Large model downloads can pause or slow down on unstable connections. If the progress bar stops, press Ctrl+C to cancel, then re-run the same ollama pull qwen3:8b command. Ollama resumes from where it left off.

help me fix my Ollama model download

Test the model

Before building your app, confirm the model can generate responses. Ollama provides an interactive chat mode for quick testing.

  • Start an interactive chat with the model by running:
ollama run qwen3:8b
  • Type a simple question like What is Python? and press Enter.

You should see the model generate a response in your terminal. This confirms the model is loaded and working correctly.

  • Type /bye and press Enter to exit the chat.

Why test now?

If something is wrong with your Ollama setup, it is much easier to catch it now than after you have built the entire app. A working ollama run confirms that the model loaded into memory and can generate text.

Model responding very slowly?

Ollama uses your GPU if available, but falls back to CPU. On CPU-only machines, the first response may take 30-60 seconds as the model loads into memory. Subsequent responses will be faster.

If you have less than 8GB of RAM, the 8B model may struggle. You can try a smaller model like qwen3:4b instead, though response quality may be lower.

help me fix slow Ollama model performance

Ollama is installed and your model is ready to go. Next up, you will set up the Python project and install the libraries that connect your code to this local LLM.

Set Up the Python Project

Your local LLM is ready to go. Now you need a Python project to talk to it.

In this step, you will create a project folder, set up a virtual environment, and install all the libraries your RAG pipeline needs. These include LangChain for orchestrating the pipeline, ChromaDB for storing vector embeddings, and Streamlit for the chat interface.

In this step, get ready to:
  • Create a Python virtual environment for isolated package management.
  • Install RAG pipeline dependencies (LangChain, ChromaDB, Streamlit, and more).
  • Verify Streamlit is working by launching a demo app.
Create the project folder and virtual environment

This project requires Python 3.10 or higher. Let's confirm you have the right version installed.

  • Check your Python version by running this command in your terminal:

macOS

python3 --version

Windows

python --version

✔️ I see version 3.10 or higher

You are all set. Move on to creating your project folder below.

ⓧ I see an older version

You need to upgrade to Python 3.10 or higher.

  • Visit python.org/downloads and download the latest Python installer for your operating system.
  • Run the installer. On Windows, make sure to check the box that says Add python.exe to PATH before clicking Install Now.
  • After installing, close and reopen your terminal, then rerun the version check command above.

Why a newer Python?

Streamlit 1.58.0 and ChromaDB 1.5.9 both require Python 3.10 or newer. An older version will cause install failures.

ⓧ Command not found

Python is not installed on your machine yet.

  • Visit python.org/downloads and download the latest Python installer for your operating system.
  • Run the installer. On Windows, make sure to check the box that says Add python.exe to PATH before clicking Install Now.
  • On macOS, the installer will guide you through the process automatically.
  • After installing, close and reopen your terminal, then run the version check command above to confirm it works.

Now let's create your project folder on the Desktop so it's easy to find.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir rag-chat-app
cd rag-chat-app
  • Confirm the folder exists by running:
ls

You should see an empty output since the folder is brand new.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Visual Studio Code and press Enter to open it.
  • In VS Code, click File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop and select the rag-chat-app folder, then click Open.

Using a different editor?

Any code editor works for this project. Open the rag-chat-app folder from your Desktop in whichever editor you prefer.

A virtual environment keeps this project's packages separate from your system Python. This prevents version conflicts with other projects.

  • Create a virtual environment by running this command in your terminal:

macOS

python3 -m venv venv

Windows

python -m venv venv
  • Activate the virtual environment:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate

You should see (venv) appear at the beginning of your terminal prompt. This confirms the virtual environment is active.

What is a virtual environment?

A virtual environment is an isolated Python installation inside your project folder. Packages you install with pip only apply to this project, not your entire computer.

The (venv) prefix in your terminal tells you the environment is active. Always activate it before running project commands.

Don't see (venv) in your prompt?

Make sure you ran the activate command for your operating system. On Windows, you may need to allow script execution first by running Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned in PowerShell, then trying the activate command again.

Help me troubleshoot my virtual environment

Install the project dependencies

Your RAG app needs several packages. You will list them all in a single file so they can be installed together.

  • In VS Code, create a new file called requirements.txt in the root of your rag-chat-app folder.
  • Paste the following content into requirements.txt:
streamlit==1.58.0
chromadb==1.5.9
langchain==0.3.25
langchain-community==0.3.25
langchain-core==0.3.51
sentence-transformers==3.4.1
pypdf==5.4.0
python-docx==1.1.2

What are these packages?

  • streamlit provides the web-based chat interface.
  • chromadb is the vector database that stores document embeddings.
  • langchain and langchain-community orchestrate the retrieval and generation pipeline.
  • sentence-transformers creates vector embeddings locally on your machine.
  • pypdf and python-docx handle reading PDF and Word documents.
  • Save requirements.txt.
  • Install all the packages by running this command in your terminal (make sure you see (venv) in your prompt):
pip install -r requirements.txt

This will download and install all eight packages plus their dependencies. It may take a few minutes depending on your internet speed.

What should I see?

Your terminal will show progress bars as each package downloads. When it finishes, you should see a line like Successfully installed followed by a list of all installed packages.

Seeing install errors?

Make sure your virtual environment is active (you see (venv) in your prompt). If you see permission errors, do NOT use sudo. Deactivate and recreate the virtual environment instead.

If a specific package fails, check that your Python version is 3.10 or higher. Some packages require a minimum version.

Help me fix my pip install errors

Verify the installation

Let's confirm everything is working by launching Streamlit's built-in demo app.

  • Run the Streamlit demo by entering this command:
streamlit hello

A new browser tab should open automatically showing Streamlit's demo page. You will see a welcome screen with example apps you can explore.

Browser didn't open?

Check your terminal output for a URL that looks like http://localhost:8501. Copy and paste it into your browser manually.

If you see a "command not found" error, your virtual environment may not be active. Run the activate command again and retry.

Help me troubleshoot my Streamlit installation

  • Once you have confirmed Streamlit is running, press Ctrl+C in your terminal to stop the demo server.

✔️ Awesome, I've got everything!

Great. Make sure you have saved requirements.txt and that your virtual environment is still active.

ⓧ I'd like to double check the full code

Here is the complete requirements.txt file for comparison:

streamlit==1.58.0
chromadb==1.5.9
langchain==0.3.25
langchain-community==0.3.25
langchain-core==0.3.51
sentence-transformers==3.4.1
pypdf==5.4.0
python-docx==1.1.2

Your development environment is locked and loaded. Next up, you will build the document processing pipeline that turns uploaded files into searchable chunks.

Build the Document Processing Pipeline

Your Python environment is ready and all dependencies are installed. Now it's time to build the core engine that powers your chatbot.

A RAG pipeline takes a document, breaks it into small pieces, converts those pieces into numbers that capture their meaning, and stores them for fast retrieval. In this step, you'll write a Python module that handles all of this.

In this step, get ready to:
  • Write document loading functions that support PDF, Word, and TXT files.
  • Split documents into overlapping chunks for better retrieval.
  • Store chunks as vector embeddings in a ChromaDB collection.
Create the document loading and splitting functions

Your RAG pipeline needs to read different file formats and break them into smaller pieces. Smaller chunks let the system find precisely the right paragraph to answer a question, instead of searching an entire document at once.

  • Create a new file called rag_pipeline.py in your project folder.
  • Add the imports and the first two functions by pasting this code:
import os
import tempfile
from langchain_community.document_loaders import PyPDFLoader, Docx2txtLoader, TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.vectorstores import Chroma


def load_document(file_path: str, file_type: str):
    """Load a document based on its file type."""
    if file_type == "pdf":
        loader = PyPDFLoader(file_path)
    elif file_type == "docx":
        loader = Docx2txtLoader(file_path)
    elif file_type == "txt":
        loader = TextLoader(file_path, encoding="utf-8")
    else:
        raise ValueError(f"Unsupported file type: {file_type}")
    return loader.load()


def split_documents(documents):
    """Split documents into chunks."""
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=1000,
        chunk_overlap=200,
        length_function=len,
    )
    return text_splitter.split_documents(documents)

What does this code do?

The imports pull in document loaders for each file type: PyPDFLoader for PDFs, Docx2txtLoader for Word files, and TextLoader for plain text.

The load_document function picks the right loader based on the file extension and returns the document content. The split_documents function uses RecursiveCharacterTextSplitter to break the text into 1000-character chunks with 200 characters of overlap between adjacent chunks.

The overlap ensures that if important information spans two chunks, it appears in both so the search can still find it.

  • Save rag_pipeline.py.
  • Verify the imports work by running this command in your terminal:
python -c "from rag_pipeline import load_document, split_documents; print('Module loaded successfully')"
  • You should see Module loaded successfully printed in the terminal.

Seeing a ModuleNotFoundError?

  • Make sure your virtual environment is activated. You should see (venv) at the start of your terminal prompt.
  • Confirm you're running the command from inside your project folder where rag_pipeline.py lives.
  • If you see an error about langchain_community, re-run pip install -r requirements.txt to reinstall the packages.

Why is my rag_pipeline.py failing to import?

Set up the ChromaDB vector store

With documents loaded and chunked, you need somewhere to store the chunks so they can be searched by meaning. A vector store converts each chunk into a list of numbers (an embedding) and lets you find the chunks most similar to a user's question.

  • Add the following two functions below your existing code in rag_pipeline.py:
def create_vector_store(chunks):
    """Create a ChromaDB vector store from document chunks."""
    embeddings = HuggingFaceEmbeddings(
        model_name="all-MiniLM-L6-v2"
    )
    vector_store = Chroma.from_documents(
        documents=chunks,
        embedding=embeddings,
    )
    return vector_store


def process_uploaded_file(uploaded_file):
    """Process an uploaded file and return a QA chain."""
    file_name = uploaded_file.name
    file_type = file_name.split(".")[-1].lower()

    with tempfile.NamedTemporaryFile(delete=False, suffix=f".{file_type}") as tmp_file:
        tmp_file.write(uploaded_file.getbuffer())
        tmp_file_path = tmp_file.name

    try:
        documents = load_document(tmp_file_path, file_type)
        chunks = split_documents(documents)
        vector_store = create_vector_store(chunks)
        return vector_store, len(chunks)
    finally:
        os.unlink(tmp_file_path)

What does this code do?

The create_vector_store function uses the all-MiniLM-L6-v2 model to convert each chunk into a 384-dimensional embedding. It stores these embeddings in ChromaDB, an in-memory vector database that enables fast similarity search.

The process_uploaded_file function is the glue that ties everything together. It saves an uploaded file to a temporary location, runs it through the full pipeline (load, chunk, embed), and cleans up the temp file when done.

  • Save rag_pipeline.py.
  • Verify all four functions import cleanly by running:
python -c "from rag_pipeline import load_document, split_documents, create_vector_store, process_uploaded_file; print('All functions loaded')"
  • You should see All functions loaded in the terminal.

Seeing an import error about HuggingFaceEmbeddings?

  • This usually means sentence-transformers did not install correctly. Run pip install sentence-transformers==3.4.1 to reinstall it.
  • If you see a warning about a deprecated import, that is safe to ignore for now.

Help me fix my HuggingFaceEmbeddings import error.

Test the pipeline with a sample file

Before connecting this pipeline to a chat interface, you should verify it works end-to-end. You'll create a sample text file and run it through the full load, chunk, embed, and retrieve cycle.

  • Create a file called test_sample.txt in your project folder with some sample content by running:
echo "Retrieval-Augmented Generation (RAG) is a technique that combines large language models with external knowledge retrieval. Instead of relying solely on what a model learned during training, RAG systems fetch relevant information from a document collection before generating an answer. The RAG pipeline consists of several stages. First, documents are loaded and split into smaller chunks. This chunking process ensures that each piece of text is small enough to be meaningful but large enough to retain context. Common chunk sizes range from 500 to 1500 characters, with overlapping regions to prevent information loss at boundaries. Next, each chunk is converted into a vector embedding that captures the semantic meaning of the text as a list of numbers. Similar concepts end up close together in vector space, even if they use different words. The vector embeddings are stored in a vector database like ChromaDB. When a user asks a question, the question is also converted into an embedding. The system then finds the chunks whose embeddings are most similar to the question embedding. These relevant chunks are passed to the language model as context for generating an accurate answer." > test_sample.txt
  • Create a test script called test_pipeline.py and paste the following code:
from rag_pipeline import load_document, split_documents, create_vector_store

# Load the sample text file
documents = load_document("test_sample.txt", "txt")
print(f"Loaded {len(documents)} document(s)")

# Split into chunks
chunks = split_documents(documents)
print(f"Split into {len(chunks)} chunk(s)")

# Create vector store and search for relevant chunks
vector_store = create_vector_store(chunks)
results = vector_store.similarity_search("What is RAG?", k=3)
print(f"\nRetrieved {len(results)} relevant chunk(s):")
for i, doc in enumerate(results):
    print(f"\n--- Chunk {i+1} ---")
    print(doc.page_content[:200])

What does this test do?

This script exercises each function in sequence: loading a text file, splitting it into chunks, creating embeddings, and performing a similarity search. The similarity_search call finds the chunks most relevant to the query "What is RAG?" and prints them.

  • Run the test script:
python test_pipeline.py

First run downloads the embedding model

The first time you run this, sentence-transformers downloads the all-MiniLM-L6-v2 model (about 90MB). This only happens once. Future runs use the cached model.

  • You should see output showing documents loaded, chunks created, and retrieved chunks printed with text about RAG.

Seeing zero chunks or an empty result?

  • Make sure test_sample.txt was created in the same folder where you're running the script. Run ls to confirm the file exists.
  • If you see a network error during the first run, check your internet connection. The embedding model needs to download once.
  • If you see only 1 chunk, that is fine for short text. The retriever still returns results.

Why is my pipeline test returning unexpected results?

✔️ Awesome, I've got everything!

Great. Double check you've saved rag_pipeline.py and that your test printed retrieved chunks successfully.

ⓧ I'd like to double check the full code

Here is the complete rag_pipeline.py file at this point:

import os
import tempfile
from langchain_community.document_loaders import PyPDFLoader, Docx2txtLoader, TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.vectorstores import Chroma


def load_document(file_path: str, file_type: str):
    """Load a document based on its file type."""
    if file_type == "pdf":
        loader = PyPDFLoader(file_path)
    elif file_type == "docx":
        loader = Docx2txtLoader(file_path)
    elif file_type == "txt":
        loader = TextLoader(file_path, encoding="utf-8")
    else:
        raise ValueError(f"Unsupported file type: {file_type}")
    return loader.load()


def split_documents(documents):
    """Split documents into chunks."""
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=1000,
        chunk_overlap=200,
        length_function=len,
    )
    return text_splitter.split_documents(documents)


def create_vector_store(chunks):
    """Create a ChromaDB vector store from document chunks."""
    embeddings = HuggingFaceEmbeddings(
        model_name="all-MiniLM-L6-v2"
    )
    vector_store = Chroma.from_documents(
        documents=chunks,
        embedding=embeddings,
    )
    return vector_store


def process_uploaded_file(uploaded_file):
    """Process an uploaded file and return a QA chain."""
    file_name = uploaded_file.name
    file_type = file_name.split(".")[-1].lower()

    with tempfile.NamedTemporaryFile(delete=False, suffix=f".{file_type}") as tmp_file:
        tmp_file.write(uploaded_file.getbuffer())
        tmp_file_path = tmp_file.name

    try:
        documents = load_document(tmp_file_path, file_type)
        chunks = split_documents(documents)
        vector_store = create_vector_store(chunks)
        return vector_store, len(chunks)
    finally:
        os.unlink(tmp_file_path)

Your document processing pipeline is complete and tested. Next up, you'll connect this pipeline to Ollama so the local LLM can generate answers from the retrieved chunks.

Connect to Ollama for LLM Responses

Your pipeline can load documents, split them into chunks, and store embeddings in a vector database. The missing piece is the brain that reads retrieved chunks and generates a human-readable answer.

In this step, you will connect your pipeline to the Ollama LLM running on your machine. You will write a prompt template that instructs the model to only answer from the provided context, then wire everything together into a single function call.

In this step, get ready to:
  • Create a QA chain that connects your vector store to the local Ollama model.
  • Write a prompt template that keeps the LLM grounded in your document context.
  • Test the full end-to-end pipeline from document to generated answer.
Create the QA chain

A RetrievalQA chain combines three components: a retriever that fetches relevant chunks, a prompt that formats the context and question, and an LLM that generates the answer. You will add all three inside a single function.

  • In rag_pipeline.py, find the existing import lines at the top of the file.
  • Add these three new imports directly below the from langchain_community.vectorstores import Chroma line:
from langchain_community.chat_models import ChatOllama
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate

What do these imports do?

  • ChatOllama connects to your local Ollama server and sends prompts to the model.
  • RetrievalQA is a LangChain chain that combines retrieval and question answering into one call.
  • PromptTemplate lets you define a reusable template with placeholder variables for context and the user's question.
  • Below your create_vector_store function and above process_uploaded_file, add the following get_qa_chain function:
def get_qa_chain(vector_store):
    """Create a retrieval QA chain with Ollama."""
    retriever = vector_store.as_retriever(
        search_type="similarity",
        search_kwargs={"k": 3},
    )

    prompt_template = """Use the following pieces of context to answer the question at the end.
If you don't know the answer based on the context, say "I don't have enough information in the document to answer that question."
Do not make up answers that are not supported by the context.

Context:
{context}

Question: {question}

Answer:"""

    prompt = PromptTemplate(
        template=prompt_template,
        input_variables=["context", "question"],
    )

    llm = ChatOllama(
        model="qwen3:8b",
        base_url="http://localhost:11434",
        temperature=0.2,
    )

    qa_chain = RetrievalQA.from_chain_type(
        llm=llm,
        chain_type="stuff",
        retriever=retriever,
        return_source_documents=True,
        chain_type_kwargs={"prompt": prompt},
    )
    return qa_chain

What does this function do?

  • vector_store.as_retriever(search_kwargs={"k": 3}) creates a retriever that fetches the top 3 most relevant chunks for any given question.
  • ChatOllama points to your locally running Ollama server at http://localhost:11434 and uses the qwen3:8b model you pulled earlier.
  • temperature=0.2 keeps responses factual and consistent. Lower temperature means less randomness.
  • return_source_documents=True tells the chain to include the retrieved chunks in its output, so you can show users where the answer came from.
  • Save rag_pipeline.py.

You will not see a visible change yet. The next substep wires this function into your file processing pipeline.

Seeing a red underline on the imports?

Make sure your virtual environment is activated. If the imports show as unresolved, your editor may be using the system Python instead of the one inside venv.

help me fix import errors in my virtual environment

Write the prompt template

The prompt template you just wrote is the most important piece of the RAG pipeline. It tells the LLM exactly how to behave when answering questions.

Why does the prompt say to refuse if the answer isn't in the context?

Without this instruction, the LLM will happily make up answers from its training data. That defeats the purpose of RAG. The whole point is to ground answers in YOUR documents, not the model's general knowledge.

The {context} placeholder gets filled with retrieved chunks. The {question} placeholder gets filled with the user's query. LangChain handles this substitution automatically at runtime.

Now you need to wire the get_qa_chain function into process_uploaded_file so it returns a ready-to-use QA chain.

  • In rag_pipeline.py, scroll down to the process_uploaded_file function.
  • Find these two lines inside the try: block:
        vector_store = create_vector_store(chunks)
        return vector_store, len(chunks)
  • Replace those two lines with these three lines, which call get_qa_chain before returning:
        vector_store = create_vector_store(chunks)
        qa_chain = get_qa_chain(vector_store)
        return qa_chain, len(chunks)

What changed?

Previously, process_uploaded_file returned the raw vector store. Now it passes the vector store into get_qa_chain and returns a fully assembled QA chain that is ready to answer questions.

  • Save rag_pipeline.py.

Not sure where to place the new line?

The qa_chain = get_qa_chain(vector_store) line goes directly after vector_store = create_vector_store(chunks) and before the return statement. Make sure it is indented at the same level (two levels in, inside the try: block).

help me fix the indentation in my process_uploaded_file function

Test the end-to-end pipeline

Your pipeline now covers the full journey: load a document, chunk it, embed it, retrieve relevant chunks, and generate an answer. Let's confirm everything works by running a quick test.

  • Create a sample text file for testing by running this command in your terminal:
echo "Python is a programming language created by Guido van Rossum. It was first released in 1991. Python emphasizes code readability and simplicity." > test_doc.txt
  • Run this Python script in your terminal to test the full pipeline:
python -c "
from rag_pipeline import load_document, split_documents, create_vector_store, get_qa_chain

documents = load_document('test_doc.txt', 'txt')
chunks = split_documents(documents)
vector_store = create_vector_store(chunks)
qa_chain = get_qa_chain(vector_store)

result = qa_chain.invoke({'query': 'Who created Python?'})
print('Answer:', result['result'])
print('Sources found:', len(result['source_documents']))
"

What should I see?

After a few seconds of processing, you should see an answer mentioning Guido van Rossum and a source count of 1 (since the test document is small enough to fit in a single chunk). The first run may take longer as the embedding model downloads (~90MB).

  • Confirm that the answer references information from your test document rather than general knowledge.

Seeing a connection error?

Make sure Ollama is still running in the background. Open a separate terminal and run:

ollama list

If you see your qwen3:8b model listed, Ollama is running. If you get an error, restart Ollama and try the test again.

help me fix connection issues between my Python script and Ollama

✔️ Awesome, I've got everything!

Great. Double check you have saved rag_pipeline.py with all your changes.

ⓧ I'd like to double check the full code

Here is the complete rag_pipeline.py file at this point:

import os
import tempfile
from langchain_community.document_loaders import PyPDFLoader, Docx2txtLoader, TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.vectorstores import Chroma
from langchain_community.chat_models import ChatOllama
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate


def load_document(file_path: str, file_type: str):
    """Load a document based on its file type."""
    if file_type == "pdf":
        loader = PyPDFLoader(file_path)
    elif file_type == "docx":
        loader = Docx2txtLoader(file_path)
    elif file_type == "txt":
        loader = TextLoader(file_path, encoding="utf-8")
    else:
        raise ValueError(f"Unsupported file type: {file_type}")
    return loader.load()


def split_documents(documents):
    """Split documents into chunks."""
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=1000,
        chunk_overlap=200,
        length_function=len,
    )
    return text_splitter.split_documents(documents)


def create_vector_store(chunks):
    """Create a ChromaDB vector store from document chunks."""
    embeddings = HuggingFaceEmbeddings(
        model_name="all-MiniLM-L6-v2"
    )
    vector_store = Chroma.from_documents(
        documents=chunks,
        embedding=embeddings,
    )
    return vector_store


def get_qa_chain(vector_store):
    """Create a retrieval QA chain with Ollama."""
    retriever = vector_store.as_retriever(
        search_type="similarity",
        search_kwargs={"k": 3},
    )

    prompt_template = """Use the following pieces of context to answer the question at the end.
If you don't know the answer based on the context, say "I don't have enough information in the document to answer that question."
Do not make up answers that are not supported by the context.

Context:
{context}

Question: {question}

Answer:"""

    prompt = PromptTemplate(
        template=prompt_template,
        input_variables=["context", "question"],
    )

    llm = ChatOllama(
        model="qwen3:8b",
        base_url="http://localhost:11434",
        temperature=0.2,
    )

    qa_chain = RetrievalQA.from_chain_type(
        llm=llm,
        chain_type="stuff",
        retriever=retriever,
        return_source_documents=True,
        chain_type_kwargs={"prompt": prompt},
    )
    return qa_chain


def process_uploaded_file(uploaded_file):
    """Process an uploaded file and return a QA chain."""
    file_name = uploaded_file.name
    file_type = file_name.split(".")[-1].lower()

    with tempfile.NamedTemporaryFile(delete=False, suffix=f".{file_type}") as tmp_file:
        tmp_file.write(uploaded_file.getbuffer())
        tmp_file_path = tmp_file.name

    try:
        documents = load_document(tmp_file_path, file_type)
        chunks = split_documents(documents)
        vector_store = create_vector_store(chunks)
        qa_chain = get_qa_chain(vector_store)
        return qa_chain, len(chunks)
    finally:
        os.unlink(tmp_file_path)

Your RAG pipeline is now fully assembled. Next up, you will build the Streamlit chat interface so users can upload documents and ask questions through a browser.

Build the Streamlit Chat Interface

Your RAG pipeline can load documents, chunk them, create embeddings, and generate answers from a local LLM. Now you need a way for users to interact with it.

In this step, you will build a Streamlit web app that gives your pipeline a proper chat interface. Users will upload files through a sidebar and ask questions in a familiar chat-style UI.

In this step, get ready to:
  • Create app.py with page configuration and a file upload sidebar.
  • Build a chat interface that displays conversation history and generates LLM responses.
  • Show source document chunks after each answer so users can verify where information came from.
Create app.py with the page layout and sidebar

Streamlit reruns your entire script from top to bottom on every user interaction. Without a way to persist data between reruns, you would lose the QA chain and chat messages every time someone clicks a button.

Streamlit's session state solves this by storing values in a dictionary-like object that survives reruns for the same browser session.

  • Create a new file called app.py in your project folder.
  • Add the following code to set up the page, initialize session state, and build the file upload sidebar:
import streamlit as st
from rag_pipeline import process_uploaded_file

st.set_page_config(page_title="Document Chat", page_icon="\ud83d\udcc4", layout="wide")
st.title("\ud83d\udcc4 Chat with Your Documents")
st.caption("Upload a PDF, Word, or TXT file and ask questions about its content. Powered by a local LLM via Ollama.")

# Initialize session state
if "messages" not in st.session_state:
    st.session_state.messages = []
if "qa_chain" not in st.session_state:
    st.session_state.qa_chain = None

# Sidebar for file upload
with st.sidebar:
    st.header("Upload Document")
    uploaded_file = st.file_uploader(
        "Choose a file",
        type=["pdf", "docx", "txt"],
        help="Supported formats: PDF, Word (.docx), Text (.txt)",
    )

    if uploaded_file and st.button("Process Document", type="primary"):
        with st.spinner("Processing document..."):
            qa_chain, num_chunks = process_uploaded_file(uploaded_file)
            st.session_state.qa_chain = qa_chain
            st.session_state.messages = []
            st.success(f"Document processed into {num_chunks} chunks. Start asking questions!")

    if st.session_state.qa_chain:
        st.divider()
        st.caption("\u2705 Document loaded. Ask questions below.")

What does this code do?

  • st.set_page_config sets the browser tab title, icon, and page layout before any other Streamlit call.
  • The two st.session_state checks create a messages list and a qa_chain slot that persist between reruns.
  • st.file_uploader restricts uploads to PDF, DOCX, and TXT files.
  • When the user clicks Process Document, the app calls your process_uploaded_file function and stores the returned QA chain in session state.
  • Save app.py.
  • Confirm the file exists by running:
ls app.py

You should see app.py listed in the output.

Don't see app.py?

Make sure you created the file inside your project folder (the same folder that contains rag_pipeline.py).

If you created it in the wrong location, delete it and recreate it in the correct folder. Help me find where my file was created.

Add the chat history display

Streamlit's st.chat_message renders messages in styled bubbles. A simple for loop over your stored messages recreates the full conversation on every rerun.

  • In app.py, add the following code below the sidebar block to display previous messages:
# Display chat history
for message in st.session_state.messages:
    with st.chat_message(message["role"]):
        st.write(message["content"])
        if "sources" in message:
            with st.expander("View Source Chunks"):
                for i, source in enumerate(message["sources"]):
                    st.markdown(f"**Chunk {i + 1}:**")
                    st.text(source[:500])
                    st.divider()

What does this code do?

  • The loop iterates over every message stored in st.session_state.messages, rendering each one as a chat bubble with the correct role (user or assistant).
  • For assistant messages that have source data attached, an expandable section shows the document chunks that informed the answer.
  • source[:500] trims each chunk to 500 characters so the display stays readable.
  • Save app.py.

You won't see a visual change yet. The next chunk adds the input field where users type their questions.

Handle user input and display source context

The final piece connects user input to your RAG pipeline. When a user submits a question, the app invokes the QA chain, displays the answer, and shows which document chunks were used.

  • In app.py, add the following code below the chat history loop to handle new questions:
# Chat input
if prompt := st.chat_input("Ask a question about your document..."):
    if not st.session_state.qa_chain:
        st.warning("Please upload and process a document first.")
    else:
        st.session_state.messages.append({"role": "user", "content": prompt})
        with st.chat_message("user"):
            st.write(prompt)

        with st.chat_message("assistant"):
            with st.spinner("Thinking..."):
                result = st.session_state.qa_chain.invoke({"query": prompt})
                answer = result["result"]
                sources = [doc.page_content for doc in result["source_documents"]]

            st.write(answer)
            with st.expander("View Source Chunks"):
                for i, source in enumerate(sources):
                    st.markdown(f"**Chunk {i + 1}:**")
                    st.text(source[:500])
                    st.divider()

        st.session_state.messages.append({
            "role": "assistant",
            "content": answer,
            "sources": sources,
        })

What does this code do?

  • st.chat_input renders a text field pinned to the bottom of the page. The walrus operator (:=) captures the user's input into prompt when submitted.
  • If no document has been processed yet, the app shows a warning instead of crashing.
  • st.spinner("Thinking...") shows a loading indicator while the LLM generates its response.
  • After the answer appears, an st.expander reveals the top 3 source chunks so users can verify the answer against the original document.
  • Both the user message and the assistant response (with sources) get appended to session state so they persist on the next rerun.
  • Save app.py.

You won't see the app yet. In the next step, you will launch it with streamlit run app.py and test it with real documents.

Seeing a syntax error when you save?

The walrus operator (:=) requires Python 3.8 or higher. If your editor flags it, confirm your virtual environment is using Python 3.10+.

Also check that the indentation inside the with blocks is consistent. Mixing tabs and spaces will cause an IndentationError. Help me debug my app.py syntax error.

✔️ Awesome, I've got everything!

Great. Double check you have saved app.py before moving on.

ⓧ I'd like to double check the full code

import streamlit as st
from rag_pipeline import process_uploaded_file

st.set_page_config(page_title="Document Chat", page_icon="\ud83d\udcc4", layout="wide")
st.title("\ud83d\udcc4 Chat with Your Documents")
st.caption("Upload a PDF, Word, or TXT file and ask questions about its content. Powered by a local LLM via Ollama.")

# Initialize session state
if "messages" not in st.session_state:
    st.session_state.messages = []
if "qa_chain" not in st.session_state:
    st.session_state.qa_chain = None

# Sidebar for file upload
with st.sidebar:
    st.header("Upload Document")
    uploaded_file = st.file_uploader(
        "Choose a file",
        type=["pdf", "docx", "txt"],
        help="Supported formats: PDF, Word (.docx), Text (.txt)",
    )

    if uploaded_file and st.button("Process Document", type="primary"):
        with st.spinner("Processing document..."):
            qa_chain, num_chunks = process_uploaded_file(uploaded_file)
            st.session_state.qa_chain = qa_chain
            st.session_state.messages = []
            st.success(f"Document processed into {num_chunks} chunks. Start asking questions!")

    if st.session_state.qa_chain:
        st.divider()
        st.caption("\u2705 Document loaded. Ask questions below.")

# Display chat history
for message in st.session_state.messages:
    with st.chat_message(message["role"]):
        st.write(message["content"])
        if "sources" in message:
            with st.expander("View Source Chunks"):
                for i, source in enumerate(message["sources"]):
                    st.markdown(f"**Chunk {i + 1}:**")
                    st.text(source[:500])
                    st.divider()

# Chat input
if prompt := st.chat_input("Ask a question about your document..."):
    if not st.session_state.qa_chain:
        st.warning("Please upload and process a document first.")
    else:
        st.session_state.messages.append({"role": "user", "content": prompt})
        with st.chat_message("user"):
            st.write(prompt)

        with st.chat_message("assistant"):
            with st.spinner("Thinking..."):
                result = st.session_state.qa_chain.invoke({"query": prompt})
                answer = result["result"]
                sources = [doc.page_content for doc in result["source_documents"]]

            st.write(answer)
            with st.expander("View Source Chunks"):
                for i, source in enumerate(sources):
                    st.markdown(f"**Chunk {i + 1}:**")
                    st.text(source[:500])
                    st.divider()

        st.session_state.messages.append({
            "role": "assistant",
            "content": answer,
            "sources": sources,
        })

Your chat interface is ready. Next up, you will launch the app and test it with real PDF, Word, and TXT files to see your full RAG pipeline in action.

Run and Test Your App

You have a complete RAG pipeline and a Streamlit chat interface ready to go. Now it is time to launch everything and see it in action.

In this step, you will run your app, upload documents in each supported format, and confirm that the retrieval pipeline returns relevant source chunks alongside the LLM's answers.

In this step, get ready to:
  • Launch the Streamlit app and see it in your browser.
  • Test all three file types (PDF, DOCX, TXT) by uploading and asking questions.
  • Verify that source chunks appear in the expander after each answer.
Launch the app

Streamlit apps start with a single terminal command. The app will open automatically in your default browser.

  • Launch your app by running this command in your terminal:
streamlit run app.py

Your browser should open automatically to http://localhost:8501. You should see the title "Chat with Your Documents" and a sidebar with an Upload Document section.

Browser didn't open?

If your browser did not open automatically, copy http://localhost:8501 from your terminal output and paste it into your browser's address bar.

If you see an error about the port already being in use, another process is running on port 8501. Close any other Streamlit apps or restart your terminal and try again.

Help me troubleshoot my Streamlit app not opening in the browser.

Test with different file types

Your app supports PDF, DOCX, and TXT uploads. Testing each type confirms that all three document loaders are working correctly.

  • In the left sidebar, click **Browse files** (or drag a file) to upload a .txt file. Any plain text file on your computer works.
  • Click the **Process Document** button.
  • Wait for the success message showing how many chunks were created.
  • Type a question about the file's content in the chat input at the bottom.
  • Press **Enter** and wait for the LLM to respond.

You should see the assistant's answer appear below your question. The first response may take 10-30 seconds as the local LLM generates its answer.

Why is the first response slow?

Ollama loads the model into memory the first time you send a query. Subsequent responses in the same session are faster because the model stays loaded.

  • Click the **X** on the uploaded file in the sidebar to remove it.
  • Upload a .pdf file and click **Process Document**.
  • Ask a question about the PDF's content and confirm you get a relevant answer.
  • Repeat with a .docx file to confirm all three loaders work.

Getting an error when uploading?

If you see an import error mentioning pypdf or docx2txt, one of your dependencies may not have installed correctly. Return to your terminal (keep the Streamlit process running), open a new terminal tab, activate your virtual environment, and run pip install -r requirements.txt again.

If you see a connection error mentioning Ollama or port 11434, make sure Ollama is still running in the background. Open a new terminal and run ollama list to confirm it responds.

Help me fix an error when uploading a document to my Streamlit RAG app.

Verify source attribution

The source chunks expander lets users verify where the answer came from. This transparency is a core feature of any trustworthy RAG application.

  • After the assistant answers a question, look for the **View Source Chunks** expander below the answer.
  • Click the expander to open it.
  • Read through the displayed chunks and confirm they contain text relevant to your question.
  • Try asking a question that the document does not cover.
  • Confirm the LLM responds with something like "I don't have enough information in the document to answer that question."

What should the source chunks look like?

Each chunk is a passage of up to 1000 characters from your original document. The retriever returns the top 3 most similar chunks based on your question.

If the chunks look unrelated to your question, the document may be too short or the question too vague. Try asking a more specific question that uses words from the document.

Source chunks look empty or irrelevant?

Make sure your test document has enough content. Very short files (under 100 words) may not produce meaningful chunks for the retriever to match against.

If the chunks appear but the answer ignores them, the LLM may be generating from its own knowledge. Check that the prompt template in rag_pipeline.py instructs the model to answer only from the provided context.

Help me understand why my RAG app's source chunks don't match my question.

Your local RAG chatbot is fully functional. You can upload documents, ask questions, and verify answers against the source material. Next up, the Secret Mission challenges you to add conversation memory so the chatbot can handle follow-up questions.

Secret mission

Add Conversation Memory

Your chatbot answers questions, but it forgets everything between messages. Ask "Tell me more about that" and it has no idea what "that" refers to. In this secret mission, you'll add conversation memory so the LLM can handle follow-up questions naturally.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing cloud costs or subscriptions to worry about. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Project folder on your Desktop (contains app.py, rag_pipeline.py, requirements.txt, and the venv virtual environment).
  • Ollama with the qwen3:8b model (~5 GB on disk).
  • sentence-transformers cached embedding model (all-MiniLM-L6-v2, ~90 MB in your HuggingFace cache).

Keep everything running

No action needed. Choose this if you plan to keep experimenting with the RAG app or uploading new documents.

  • Ollama stays installed and ready for other local LLM projects.
  • Your project files remain in their folder on your Desktop.
  • The qwen3:8b model stays downloaded for instant access.

Pause - I'll come back to this later

Free up system memory by stopping the Ollama background process while keeping all files in place.

  • Close any running Streamlit server by pressing Ctrl+C in the terminal where it is running.
  • Deactivate your virtual environment by running:
deactivate
  • Close any open Ollama chat sessions (if running ollama run, type /bye to exit).
  • Your project files, model, and virtual environment will stay on disk for when you return.

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow all the steps below to fully clean up.

  • Close any running Streamlit server by pressing Ctrl+C in the terminal.
  • Deactivate your virtual environment by running:
deactivate
  • Delete the project folder from your Desktop. On macOS/Linux, run:
rm -rf ~/Desktop/rag-chat-app
  • On Windows, run:
Remove-Item -Recurse -Force "$HOME\Desktop\rag-chat-app"
  • Remove the qwen3:8b model from Ollama by running:
ollama rm qwen3:8b
  • To uninstall Ollama entirely on Windows, open Settings, go to Apps, then Add or remove programs. Find Ollama in the list and click Uninstall. The uninstaller will ask if you want to remove downloaded models as well.
  • Clear the HuggingFace embedding model cache. On macOS/Linux, run:
rm -rf ~/.cache/huggingface/hub/models--sentence-transformers--all-MiniLM-L6-v2
  • On Windows, run:
Remove-Item -Recurse -Force "$HOME\.cache\huggingface\hub\models--sentence-transformers--all-MiniLM-L6-v2"

Nice Work!

Nice Work!

You just built a fully local, privacy-preserving document Q&A chatbot powered by your own machine. No API keys, no cloud costs, no data leaving your computer.

You've learned how to:

  • Build a RAG pipeline that chunks documents, creates vector embeddings, and retrieves relevant context for an LLM to answer questions.
  • Connect a local Ollama LLM to a Streamlit chat interface that processes PDF, Word, and TXT files.
  • Store and search document chunks using ChromaDB vector embeddings with source attribution so users can verify answers.
  • Added conversation memory so the chatbot handles follow-up questions that reference previous answers.

Ready to quiz yourself?