Build Your AI Second Brain

Build a local RAG system that answers questions from your own documents.

Introduction

30 Second Summary

You have notes, research, and documents scattered across your computer. Finding the exact piece of information you need means manually searching through dozens of files.

In this project, you will build a local AI assistant that reads your personal documents and answers questions about them in natural language. No cloud services, no API keys, no cost.

What You'll Build

A fully local RAG (Retrieval-Augmented Generation) system that indexes your documents into a vector database and generates grounded answers with source citations.

By the end of this project, you'll have:

  • A document ingestion pipeline that loads, chunks, and embeds your Markdown and text files into ChromaDB.
  • A RAG query engine that retrieves relevant passages and generates answers grounded in your own documents.
  • A Streamlit chat interface to converse with your personal knowledge base.
  • Secret Mission: Add hybrid search combining BM25 keyword matching with vector semantic search for maximum retrieval accuracy.

Are there any prerequisites?

You should be comfortable with intermediate Python (functions, file I/O, lists, and dictionaries). No prior experience with AI, vector embeddings, or Ollama is needed.

Set Up Your Environment

Your AI second brain needs two core capabilities: understanding language (to process your questions) and storing knowledge (to index your documents).

In this step, you will install Ollama to run AI models locally, set up a Python project with the required packages, and create sample documents to test your system.

In this step, get ready to:
  • Install Ollama and pull the embedding and chat models.
  • Create a Python virtual environment and install dependencies.
  • Create a project folder with sample documents for testing.
Install Ollama and pull the required models

Ollama is a tool that lets you run large language models entirely on your own machine. You will use it for two things: generating vector embeddings (to index documents) and generating chat responses (to answer questions).

  • Install Ollama by running the install command for your operating system:

macOS

  • Open Terminal by pressing Cmd+Space, typing Terminal, and pressing Enter.
  • Install Ollama by running this command:
curl -fsSL https://ollama.com/install.sh | sh

What does this command do?

This downloads and runs the official Ollama install script from ollama.com. It installs the ollama CLI to your system so you can pull and run AI models.

Windows

  • Open PowerShell by pressing the Windows key, typing PowerShell, and pressing Enter.
  • Install Ollama by running this command:
irm https://ollama.com/install.ps1 | iex

What does this command do?

This downloads and runs the official Ollama installer for Windows. It installs the ollama CLI so you can pull and run AI models from PowerShell or Command Prompt.

  • Verify Ollama installed correctly by running:
ollama --version

✔️ I see a version number

Ollama is installed and ready. Continue to the next command below.

ⓧ Command not found

The install may not have completed successfully. Try these fixes:

  • Close your terminal and open a new one, then try the command again.
  • On macOS, re-run the install script. On Windows, re-run the PowerShell command from the previous tab.
  • On Windows, make sure you are running PowerShell as Administrator.

If you see a version number after retrying, continue below.

Now you will download two models. The embedding model converts text into vectors for search. The chat model generates answers from retrieved context.

  • Pull the embedding model by running:
ollama pull nomic-embed-text

What is nomic-embed-text?

This is a 274MB embedding model that converts text into numerical vectors. These vectors let you search by meaning rather than exact keywords.

  • Pull the chat model by running:
ollama pull llama3.2

What is llama3.2?

This is a 2.0GB language model with 3 billion parameters. It generates natural language answers and supports a 128K context window, which is more than enough for RAG responses.

  • Verify both models are ready by running:
ollama list
  • You should see both nomic-embed-text and llama3.2 in the output list.

Don't see both models listed?

  • Make sure each ollama pull command completed without errors. If a pull was interrupted, run it again.
  • Ensure you have at least 4GB of free disk space for the models.

Help me troubleshoot why my Ollama models aren't showing up when I run ollama list.

Create your Python virtual environment and install dependencies

This project requires Python 3.10 or higher because Streamlit dropped support for older versions.

  • Check your Python version by running:
python3 --version

✔️ I see version 3.10 or higher

Your Python version is compatible. Continue to the project folder creation below.

ⓧ I see a version lower than 3.10

You need Python 3.10 or higher. Your current version will cause package installation failures.

  • On macOS, the system Python is often outdated. Install a newer version using Homebrew:
  • Run brew install python@3.12 and then use python3.12 instead of python3 for the rest of this project.
  • On Windows, download and install the latest Python from python.org. Make sure to check Add Python to PATH during installation.

After upgrading, close your terminal, open a new one, and run python3 --version again to confirm.

ⓧ Command not found

Python is not installed on your system yet.

  • On macOS, install Python using Homebrew by running brew install python@3.12 in Terminal.
  • On Windows, download and install Python from python.org. Check Add Python to PATH during installation.

After installing, close your terminal, open a new one, and run python3 --version to confirm you see version 3.10 or higher.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir ai-second-brain
cd ai-second-brain
  • Create a Python virtual environment by running:
python3 -m venv venv

What is a virtual environment?

A virtual environment is an isolated Python installation inside your project folder. Packages you install here stay separate from your system Python, preventing version conflicts between projects.

  • Activate the virtual environment:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate

You should see (venv) appear at the beginning of your terminal prompt. This confirms the virtual environment is active.

  • Install the three packages your project needs by running:
pip install ollama==0.6.2 chromadb==1.5.9 streamlit==1.58.0

What are these packages?

  • ollama is the Python client for interacting with your local Ollama models.
  • chromadb is a vector database that stores and searches document embeddings.
  • streamlit is a framework for building web UIs with Python. You will use it for your chat interface.
  • Verify all packages installed correctly by running:
python -c "import ollama, chromadb, streamlit; print('All good!')"
  • You should see All good! printed in your terminal.

Seeing an import error?

  • Make sure you see (venv) in your terminal prompt. If not, re-run the activation command above.
  • If a specific package failed to install, try running the pip install command again. Network issues can interrupt downloads.
  • Confirm you are running Python 3.10+ by checking python --version inside the active venv.

Help me debug why my Python packages aren't importing correctly after pip install.

Create project structure and sample documents

Your RAG system needs documents to index. You will create a docs/ folder with sample Markdown files that contain varied content for testing.

  • Create the documents folder by running:
mkdir docs
  • Open your project in VS Code by pressing Cmd+Space (macOS) or the Windows key (Windows), typing VS Code, and pressing Enter.
  • In VS Code, click File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop and select the ai-second-brain folder, then click Open.

Now create 2-3 sample Markdown files in the docs/ folder. These can be about any topic you know well. The more varied the content, the better you can test your search later.

  • In the VS Code Explorer sidebar, right-click the docs folder and select New File.
  • Name your first file cooking-tips.md (or any topic you prefer).
  • Add content using Markdown headings and paragraphs. Here is an example:
# Cooking Tips

## Knife Skills

A sharp knife is safer than a dull one. Dull knives require more force and are more likely to slip. Keep your knives honed before each use with a steel rod.

## Seasoning

Salt your food in layers as you cook, not just at the end. Each ingredient benefits from being seasoned at the right time. Pasta water should taste like the sea.

## Heat Management

Let your pan get hot before adding oil. Add oil, then let the oil get hot before adding food. This prevents sticking and creates better browning.
  • Create a second file in the docs folder. Right-click docs again and select New File.
  • Name it productivity.md (or another topic of your choice).
  • Add varied content with headings. Here is another example:
# Productivity Methods

## Pomodoro Technique

Work in focused 25-minute blocks followed by 5-minute breaks. After four blocks, take a longer 15-30 minute break. This prevents burnout while maintaining deep focus.

## Time Blocking

Assign specific tasks to specific time slots on your calendar. Treat these blocks as appointments you cannot miss. This eliminates decision fatigue about what to work on next.

## Two-Minute Rule

If a task takes less than two minutes, do it immediately. Small tasks pile up and create mental overhead. Clearing them instantly keeps your task list manageable.

Use your own knowledge

These examples are just starting points. Replace them with notes about topics you actually know. Your second brain works best when it contains YOUR knowledge. Try meeting notes, study summaries, or hobby research.

  • Create at least one more file with different content so you have enough variety for meaningful search results.
  • Save all files by pressing Cmd+S (macOS) or Ctrl+S (Windows).
  • Confirm your project structure by running this in the terminal:
ls docs/
  • You should see your Markdown files listed (e.g. cooking-tips.md, productivity.md).

Don't see your files?

  • Make sure you saved each file in VS Code (look for the dot on unsaved tabs).
  • Confirm you created the files inside the docs/ subfolder, not in the root ai-second-brain folder.
  • Make sure your terminal is in the ai-second-brain folder. Run pwd to check your current location.

Help me figure out why my docs folder is empty or missing.

Your environment is fully set up. You have Ollama running two AI models, a Python virtual environment with all dependencies, and sample documents ready for indexing. Next up, you will build the ingestion pipeline that reads your documents and stores them as searchable vectors.

Build the Document Ingestion Pipeline

Your environment is set up and your sample documents are waiting in the docs/ folder. Now it's time to teach your AI how to read them.

Before an LLM can answer questions about your documents, it needs to convert raw text into something searchable. In this step, you'll build a pipeline that loads your files, splits them into meaningful chunks, converts those chunks into vector embeddings, and stores everything in a local database.

In this step, get ready to:
  • Write functions that load and chunk your Markdown files.
  • Embed each chunk into a vector using Ollama.
  • Store everything in a local ChromaDB database.
Write the document loader and chunker

The first stage of any RAG pipeline is ingestion. You need to read your files from disk, then split them into small, self-contained chunks that the LLM can digest later.

  • In VS Code, create a new file called ingest.py in your ai-second-brain folder.
  • Add the imports and the document loader function by pasting the following code:
import os
import ollama
import chromadb


def load_documents(docs_path):
    """Load all .md and .txt files from a directory."""
    documents = []
    for filename in os.listdir(docs_path):
        if filename.endswith((".md", ".txt")):
            filepath = os.path.join(docs_path, filename)
            with open(filepath, "r", encoding="utf-8") as f:
                text = f.read()
            documents.append({"text": text, "source": filename})
    return documents

What does this code do?

The imports bring in three libraries: os for file system access, ollama for embeddings, and chromadb for vector storage.

The load_documents() function walks through every file in a folder and reads any .md or .txt file into memory. It stores each document's text and its source filename in a dictionary.

  • Next, add the chunking function below load_documents in the same file:
def chunk_document(text, source, max_chunk_size=800):
    """Split document into chunks by headings or paragraph breaks."""
    chunks = []
    current_chunk = ""
    current_heading = ""

    lines = text.split("\n")

    for line in lines:
        if line.startswith("#"):
            if current_chunk.strip():
                chunks.append({
                    "text": current_chunk.strip(),
                    "source": source,
                    "heading": current_heading,
                })
            current_chunk = line + "\n"
            current_heading = line.strip("# ").strip()
        else:
            if len(current_chunk) + len(line) > max_chunk_size and current_chunk.strip():
                chunks.append({
                    "text": current_chunk.strip(),
                    "source": source,
                    "heading": current_heading,
                })
                current_chunk = ""
            current_chunk += line + "\n"

    if current_chunk.strip():
        chunks.append({
            "text": current_chunk.strip(),
            "source": source,
            "heading": current_heading,
        })

    return chunks

What does the chunker do?

Markdown documents have natural structure via headings. This function splits text at every # heading line, keeping related content together.

If a section grows longer than 800 characters, it splits again at paragraph boundaries. Each chunk carries metadata about its source file and heading, so you can cite sources later.

  • Save ingest.py.

Seeing a red underline on the imports?

Make sure your virtual environment is active. You should see (venv) at the start of your terminal prompt. If not, reactivate it with venv\Scripts\activate.

Help me fix import errors in my ingest.py file.

Write the embedding and indexing logic

Your chunker produces text passages, but an LLM can't search raw text efficiently. You need to convert each chunk into a vector embedding so the database can find semantically similar content later.

  • In ingest.py, add the embedding function below chunk_document:
def embed_chunks(chunks):
    """Generate embeddings for a list of chunks using Ollama."""
    texts = [chunk["text"] for chunk in chunks]
    embeddings = []
    for text in texts:
        response = ollama.embed(model="nomic-embed-text", input=text)
        embeddings.append(response.embeddings[0])
    return embeddings

What is a vector embedding?

A vector embedding is a list of numbers that captures the meaning of a piece of text. Similar ideas produce vectors that are close together in mathematical space.

The nomic-embed-text model (only 274MB) converts each chunk into a vector. Later, when you ask a question, the system embeds your question the same way and finds the closest matching chunks.

  • Now add the main pipeline function that ties everything together. Paste this below embed_chunks:
def index_documents(docs_path="./docs"):
    """Main ingestion pipeline: load, chunk, embed, store."""
    print(f"Loading documents from {docs_path}...")
    documents = load_documents(docs_path)
    print(f"Found {len(documents)} documents.")

    all_chunks = []
    for doc in documents:
        chunks = chunk_document(doc["text"], doc["source"])
        all_chunks.extend(chunks)
    print(f"Created {len(all_chunks)} chunks.")

    print("Generating embeddings (this may take a moment)...")
    embeddings = embed_chunks(all_chunks)

    print("Storing in ChromaDB...")
    client = chromadb.PersistentClient(path="./chroma_db")
    collection = client.get_or_create_collection(name="second_brain")

    ids = [f"{chunk['source']}::chunk{i}" for i, chunk in enumerate(all_chunks)]
    docs = [chunk["text"] for chunk in all_chunks]
    metadatas = [{"source": chunk["source"], "heading": chunk["heading"]} for chunk in all_chunks]

    collection.upsert(
        ids=ids,
        documents=docs,
        embeddings=embeddings,
        metadatas=metadatas,
    )

    print(f"Done! Indexed {len(all_chunks)} chunks into ChromaDB.")

What does index_documents do?

This function orchestrates the full pipeline: load files, chunk them, embed them, and store everything.

  • chromadb.PersistentClient(path="./chroma_db") creates a local database folder that persists between script runs.
  • get_or_create_collection(name="second_brain") gives you a named table inside that database.
  • collection.upsert(...) stores each chunk's text, embedding vector, and metadata. Using upsert means re-running the script updates existing entries instead of duplicating them.
  • Save ingest.py.

Unsure about the ChromaDB path?

The path ./chroma_db is relative to where you run the script. As long as you run python ingest.py from inside your ai-second-brain folder, the database appears there.

Help me understand how ChromaDB stores data locally.

Run the ingestion pipeline

Your functions are ready. The last piece is a small entry point that kicks off the pipeline when you run the script directly.

  • At the very bottom of ingest.py, add this entry point:
if __name__ == "__main__":
    index_documents()
  • Save ingest.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file, then continue below to run it.

ⓧ I'd like to double check the full code

Here's the complete ingest.py file. Make sure yours matches exactly:

import os
import ollama
import chromadb


def load_documents(docs_path):
    """Load all .md and .txt files from a directory."""
    documents = []
    for filename in os.listdir(docs_path):
        if filename.endswith((".md", ".txt")):
            filepath = os.path.join(docs_path, filename)
            with open(filepath, "r", encoding="utf-8") as f:
                text = f.read()
            documents.append({"text": text, "source": filename})
    return documents


def chunk_document(text, source, max_chunk_size=800):
    """Split document into chunks by headings or paragraph breaks."""
    chunks = []
    current_chunk = ""
    current_heading = ""

    lines = text.split("\n")

    for line in lines:
        if line.startswith("#"):
            if current_chunk.strip():
                chunks.append({
                    "text": current_chunk.strip(),
                    "source": source,
                    "heading": current_heading,
                })
            current_chunk = line + "\n"
            current_heading = line.strip("# ").strip()
        else:
            if len(current_chunk) + len(line) > max_chunk_size and current_chunk.strip():
                chunks.append({
                    "text": current_chunk.strip(),
                    "source": source,
                    "heading": current_heading,
                })
                current_chunk = ""
            current_chunk += line + "\n"

    if current_chunk.strip():
        chunks.append({
            "text": current_chunk.strip(),
            "source": source,
            "heading": current_heading,
        })

    return chunks


def embed_chunks(chunks):
    """Generate embeddings for a list of chunks using Ollama."""
    texts = [chunk["text"] for chunk in chunks]
    embeddings = []
    for text in texts:
        response = ollama.embed(model="nomic-embed-text", input=text)
        embeddings.append(response.embeddings[0])
    return embeddings


def index_documents(docs_path="./docs"):
    """Main ingestion pipeline: load, chunk, embed, store."""
    print(f"Loading documents from {docs_path}...")
    documents = load_documents(docs_path)
    print(f"Found {len(documents)} documents.")

    all_chunks = []
    for doc in documents:
        chunks = chunk_document(doc["text"], doc["source"])
        all_chunks.extend(chunks)
    print(f"Created {len(all_chunks)} chunks.")

    print("Generating embeddings (this may take a moment)...")
    embeddings = embed_chunks(all_chunks)

    print("Storing in ChromaDB...")
    client = chromadb.PersistentClient(path="./chroma_db")
    collection = client.get_or_create_collection(name="second_brain")

    ids = [f"{chunk['source']}::chunk{i}" for i, chunk in enumerate(all_chunks)]
    docs = [chunk["text"] for chunk in all_chunks]
    metadatas = [{"source": chunk["source"], "heading": chunk["heading"]} for chunk in all_chunks]

    collection.upsert(
        ids=ids,
        documents=docs,
        embeddings=embeddings,
        metadatas=metadatas,
    )

    print(f"Done! Indexed {len(all_chunks)} chunks into ChromaDB.")


if __name__ == "__main__":
    index_documents()
  • Run the ingestion script by entering this command in your terminal:
python ingest.py

You should see output like this (the exact numbers depend on your sample documents):

Loading documents from ./docs...
Found 3 documents.
Created 8 chunks.
Generating embeddings (this may take a moment)...
Storing in ChromaDB...
Done! Indexed 8 chunks into ChromaDB.
  • Confirm the database folder was created by running:
ls chroma_db

You should see files inside the chroma_db folder, confirming your documents are now indexed.

Script errors or no chunks created?

  • Make sure you have .md or .txt files inside your docs/ folder. Empty folders produce zero chunks.
  • Confirm Ollama is still running. If you closed the terminal since last step, open a new terminal and run ollama list to verify.
  • Check that you're running the command from inside your ai-second-brain folder, not one level above.

My ingest.py script failed, help me debug.

Your documents are now living inside a searchable vector database. Next up, you'll build a query engine that retrieves the right chunks and generates grounded answers from them.

Build the RAG Query Engine

Your ingestion pipeline stored document chunks as vector embeddings in ChromaDB. Now you need a way to ask questions and get answers from that knowledge base.

The query engine is the core of your RAG system. It takes a question, finds the most relevant chunks using vector similarity search, then passes those chunks to an LLM that generates a grounded answer with source citations.

In this step, get ready to:
  • Build a retrieval function that finds relevant chunks for any question.
  • Build a generation function that produces answers grounded in retrieved context.
  • Test the full RAG pipeline from the command line.
Write the retrieval function

Retrieval is the "R" in RAG. Your retrieval function takes a question, converts it into a vector embedding, and searches ChromaDB for the most similar document chunks.

  • Create a new file called query.py in your ai-second-brain folder, and add the following code:
import ollama
import chromadb


def get_collection():
    """Connect to the existing ChromaDB collection."""
    client = chromadb.PersistentClient(path="./chroma_db")
    return client.get_or_create_collection(name="second_brain")

What does this code do?

  • The imports bring in the same ollama and chromadb libraries you used in ingest.py.
  • get_collection() connects to the same ChromaDB database and collection where your ingestion pipeline stored the document chunks.
  • Below the get_collection() function, add the retrieval function:
def retrieve(question, n_results=5):
    """Retrieve the most relevant chunks for a question."""
    response = ollama.embed(model="nomic-embed-text", input=question)
    query_embedding = response.embeddings[0]

    collection = get_collection()
    results = collection.query(
        query_embeddings=[query_embedding],
        n_results=n_results,
    )

    return results

What does this code do?

This function performs semantic search over your indexed documents.

  • ollama.embed() converts your question into a vector embedding using the same nomic-embed-text model that embedded the documents. This puts the question in the same "vector space" as your chunks.
  • collection.query(query_embeddings=[...]) searches ChromaDB for the 5 most similar chunks by comparing the question's embedding to every stored embedding.
  • The results include the matching document texts and their metadata (source filenames and headings).
  • Save query.py.

You won't see output yet. The next substep adds the generation function that turns retrieved chunks into natural language answers.

Seeing an import error?

Make sure your virtual environment is still activated. You should see (venv) at the start of your terminal prompt. If not, re-activate it with:

venv\Scripts\activate

help me fix my import error in query.py

Write the grounded generation function

Retrieval finds relevant chunks, but you still need an LLM to synthesize those chunks into a coherent answer. The generation function formats retrieved context into a prompt and instructs the model to answer only from that context.

  • In query.py, add the following function below retrieve():
def generate_answer(question, results):
    """Generate a grounded answer using retrieved context."""
    documents = results["documents"][0]
    metadatas = results["metadatas"][0]

    context_parts = []
    for i, (doc, meta) in enumerate(zip(documents, metadatas)):
        source = meta["source"]
        context_parts.append(f"[Source: {source}]\n{doc}")

    context = "\n\n---\n\n".join(context_parts)

    system_prompt = """You are a helpful assistant that answers questions using ONLY the provided context.
Rules:
- Answer based ONLY on the context below. Do not use outside knowledge.
- If the context does not contain the answer, say "I don't have information about that in my documents."
- Cite your sources by mentioning the filename in brackets, e.g. [notes.md].
- Be concise and direct."""

    user_prompt = f"""Context:
{context}

Question: {question}"""

    stream = ollama.chat(
        model="llama3.2",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_prompt},
        ],
        stream=True,
    )

    return stream

What does this code do?

This function is the "grounded generation" part of RAG. It ensures the LLM only uses YOUR documents to answer.

  • It formats each retrieved chunk with its source filename as a label, so the model knows where each piece of context comes from.
  • The system_prompt strictly instructs the model to answer ONLY from the provided context and cite its sources.
  • ollama.chat() calls llama3.2 with streaming enabled, so the answer appears word-by-word instead of waiting for the full response.

Why ground the model with rules?

Without the system prompt, the LLM would happily answer from its training data. The rules prevent hallucination by telling the model to say "I don't know" when the context doesn't contain the answer.

This is what separates RAG from a plain chatbot. The model becomes a reader of your documents, not a general knowledge oracle.

  • Save query.py.

You won't see output yet. The next substep ties retrieval and generation together into a runnable pipeline.

Test the query engine from the command line

You have both pieces of the RAG pipeline: retrieval and generation. Now you need a function that connects them and a way to run the whole thing from your terminal.

  • In query.py, add the following code below generate_answer():
def ask(question):
    """Full RAG pipeline: retrieve and generate."""
    print(f"\nQuestion: {question}")
    print("-" * 50)

    results = retrieve(question)

    print("Sources found:")
    for meta in results["metadatas"][0]:
        print(f"  - {meta['source']} ({meta['heading']})")
    print("-" * 50)
    print("Answer:")

    stream = generate_answer(question, results)
    for chunk in stream:
        print(chunk.message.content, end="", flush=True)
    print("\n")


if __name__ == "__main__":
    question = input("Ask your second brain: ")
    ask(question)

What does this code do?

  • ask() is the full RAG pipeline in one function. It calls retrieve() to find relevant chunks, prints which source files were matched, then calls generate_answer() and streams the response to the terminal.
  • The __main__ block prompts you for a question and runs the pipeline when you execute the script directly.
  • chunk.message.content extracts each streamed token from the Ollama response so the answer prints word-by-word.
  • Save query.py.
  • Run the query engine by running this command:
python query.py
  • When prompted with Ask your second brain:, type a question about the content in your sample documents and press Enter.

You should see the sources it found (with filenames and headings), followed by a streamed answer that references your documents by name.

Getting an empty answer or wrong sources?

  • Make sure you ran python ingest.py successfully in the previous step. The ./chroma_db folder must exist with indexed data.
  • Check that Ollama is still running. You can verify by running ollama list in a separate terminal.
  • Ask a question that directly relates to content in your sample documents. Generic questions will not match if the context is too different.

help me troubleshoot my RAG query engine not returning relevant results

✔️ Awesome, I've got everything!

Great. Double check you've saved query.py and that your test question returned a grounded answer with source citations.

ⓧ I'd like to double check the full code

Here is the complete query.py file:

import ollama
import chromadb


def get_collection():
    """Connect to the existing ChromaDB collection."""
    client = chromadb.PersistentClient(path="./chroma_db")
    return client.get_or_create_collection(name="second_brain")


def retrieve(question, n_results=5):
    """Retrieve the most relevant chunks for a question."""
    response = ollama.embed(model="nomic-embed-text", input=question)
    query_embedding = response.embeddings[0]

    collection = get_collection()
    results = collection.query(
        query_embeddings=[query_embedding],
        n_results=n_results,
    )

    return results


def generate_answer(question, results):
    """Generate a grounded answer using retrieved context."""
    documents = results["documents"][0]
    metadatas = results["metadatas"][0]

    context_parts = []
    for i, (doc, meta) in enumerate(zip(documents, metadatas)):
        source = meta["source"]
        context_parts.append(f"[Source: {source}]\n{doc}")

    context = "\n\n---\n\n".join(context_parts)

    system_prompt = """You are a helpful assistant that answers questions using ONLY the provided context.
Rules:
- Answer based ONLY on the context below. Do not use outside knowledge.
- If the context does not contain the answer, say "I don't have information about that in my documents."
- Cite your sources by mentioning the filename in brackets, e.g. [notes.md].
- Be concise and direct."""

    user_prompt = f"""Context:
{context}

Question: {question}"""

    stream = ollama.chat(
        model="llama3.2",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_prompt},
        ],
        stream=True,
    )

    return stream


def ask(question):
    """Full RAG pipeline: retrieve and generate."""
    print(f"\nQuestion: {question}")
    print("-" * 50)

    results = retrieve(question)

    print("Sources found:")
    for meta in results["metadatas"][0]:
        print(f"  - {meta['source']} ({meta['heading']})")
    print("-" * 50)
    print("Answer:")

    stream = generate_answer(question, results)
    for chunk in stream:
        print(chunk.message.content, end="", flush=True)
    print("\n")


if __name__ == "__main__":
    question = input("Ask your second brain: ")
    ask(question)

Your query engine is working. It finds the right context from your documents and generates grounded answers. Next up, you'll wrap this in a Streamlit chat interface so you can have ongoing conversations with your second brain.

Build the Streamlit Chat Interface

Your RAG pipeline works. You can ask questions from the terminal and get grounded answers with source citations.

Now it is time to give your second brain a proper interface. In this step, you will build a Streamlit chat UI that lets you converse with your documents, complete with streaming responses and source expanders.

In this step, get ready to:
  • Create a Streamlit app with page config and session state.
  • Build a chat loop that retrieves context and streams answers.
  • Run and test the full end-to-end system in your browser.
Create the Streamlit app layout

Streamlit apps are Python scripts that run top-to-bottom on every interaction. Each time the user sends a message, Streamlit re-executes the entire file. That means you need a way to persist chat history between reruns.

  • Create a new file called app.py in your ai-second-brain folder.
  • Add the following code to set up the imports, page config, and session state:
import streamlit as st
from query import retrieve, generate_answer

# Configure the browser tab title and icon
st.set_page_config(page_title="AI Second Brain", page_icon="\U0001f9e0")

# Display the app header
st.title("\U0001f9e0 AI Second Brain")
st.caption("Ask questions about your documents. Everything runs locally.")

# Initialize chat history in session state (persists across reruns)
if "messages" not in st.session_state:
    st.session_state.messages = []

What does this code do?

  • from query import retrieve, generate_answer imports the RAG functions you built in the previous step.
  • st.set_page_config() sets the browser tab title and icon. This must be the first Streamlit command in the file.
  • st.session_state.messages is a list that survives Streamlit reruns. Without it, your chat history would disappear every time you send a new message.

What is session state?

Streamlit re-executes your script on every user interaction. Session state is a dictionary that persists data between those reruns. It is how Streamlit apps remember things like form values, toggles, and chat messages.

  • Save app.py.

You will not see a change yet. The next chunk adds the visible chat elements.

Build the chat interaction loop

The chat loop has two jobs. First, it re-renders any messages that already exist in history. Second, it handles new user input by retrieving context and generating a streamed answer.

  • In app.py, add the following code below the session state initialization to display existing chat history:
# Re-display all previous messages on each rerun
for message in st.session_state.messages:
    with st.chat_message(message["role"]):
        st.markdown(message["content"])
        if "sources" in message:
            with st.expander("Sources"):
                for source in message["sources"]:
                    st.markdown(f"- **{source}**")

What does this code do?

  • The for loop iterates through every message stored in session state.
  • st.chat_message() renders each message with the correct avatar (user or assistant) based on the role field.
  • If a message has source citations, they appear in a collapsible expander so the UI stays clean.
  • Save app.py.

You will not see a change yet. The next chunk wires up the input box and RAG pipeline.

  • Still in app.py, add the following code below the chat history loop to handle new user messages:
# Handle new user input
if prompt := st.chat_input("Ask your second brain..."):
    # Add user message to history and display it
    st.session_state.messages.append({"role": "user", "content": prompt})
    with st.chat_message("user"):
        st.markdown(prompt)

    # Generate and display assistant response
    with st.chat_message("assistant"):
        # Retrieve relevant chunks from your documents
        results = retrieve(prompt)

        # Collect unique source filenames
        sources = list(set(
            meta["source"] for meta in results["metadatas"][0]
        ))

        # Generate a streamed answer grounded in the retrieved context
        stream = generate_answer(prompt, results)

        # Wrap the Ollama stream in a generator that yields text chunks
        def response_generator():
            for chunk in stream:
                yield chunk.message.content

        # Display the response with a typewriter effect
        response = st.write_stream(response_generator())

        # Show source documents in a collapsible section
        with st.expander("Sources"):
            for source in sources:
                st.markdown(f"- **{source}**")

    # Save the full response and sources to chat history
    st.session_state.messages.append({
        "role": "assistant",
        "content": response,
        "sources": sources,
    })

What does this code do?

  • st.chat_input() displays a text input pinned to the bottom of the page. When the user presses Enter, it returns their message.
  • retrieve(prompt) embeds the question and finds the most relevant chunks from ChromaDB.
  • generate_answer(prompt, results) calls llama3.2 with the retrieved context and returns a stream.
  • response_generator() wraps the Ollama stream into a standard Python generator that st.write_stream() can consume.
  • st.write_stream() renders each text chunk as it arrives, creating a typewriter effect. It returns the full concatenated response string.
  • The final append saves the response and sources so they persist when Streamlit reruns the script.
  • Save app.py.

Seeing a syntax error on the walrus operator?

The := operator requires Python 3.8 or newer. If you see a syntax error, confirm you are running inside your virtual environment with Python 3.10+.

Help me fix the walrus operator syntax error

✔️ Awesome, I've got everything!

Great. Double check you have saved app.py before moving on.

ⓧ I'd like to double check the full code

Here is the complete app.py file:

import streamlit as st
from query import retrieve, generate_answer

st.set_page_config(page_title="AI Second Brain", page_icon="\U0001f9e0")

st.title("\U0001f9e0 AI Second Brain")
st.caption("Ask questions about your documents. Everything runs locally.")

if "messages" not in st.session_state:
    st.session_state.messages = []

for message in st.session_state.messages:
    with st.chat_message(message["role"]):
        st.markdown(message["content"])
        if "sources" in message:
            with st.expander("Sources"):
                for source in message["sources"]:
                    st.markdown(f"- **{source}**")

if prompt := st.chat_input("Ask your second brain..."):
    st.session_state.messages.append({"role": "user", "content": prompt})
    with st.chat_message("user"):
        st.markdown(prompt)

    with st.chat_message("assistant"):
        results = retrieve(prompt)

        sources = list(set(
            meta["source"] for meta in results["metadatas"][0]
        ))

        stream = generate_answer(prompt, results)

        def response_generator():
            for chunk in stream:
                yield chunk.message.content

        response = st.write_stream(response_generator())

        with st.expander("Sources"):
            for source in sources:
                st.markdown(f"- **{source}**")

    st.session_state.messages.append({
        "role": "assistant",
        "content": response,
        "sources": sources,
    })
Run and test the app

Everything is wired up. Time to launch the chat interface and have a conversation with your documents.

  • Make sure Ollama is still running in the background. If you closed it, reopen it.
  • In your terminal (with the virtual environment still active), start the Streamlit app by running:
streamlit run app.py

What should I see?

Streamlit will print a local URL (usually http://localhost:8501) and automatically open it in your browser. You will see the title, caption, and a chat input bar pinned at the bottom of the page.

  • Type a question about your sample documents in the chat input and press Enter.
  • Watch the response stream in word by word.
  • After the answer finishes, click the Sources expander to see which files were used.
  • Ask 2-3 more questions to confirm the chat history persists between messages.

Questions to try

Try asking something specific that only your documents would know. Then try asking something completely unrelated to see the LLM respond with its grounding refusal.

Seeing a ModuleNotFoundError?

Make sure your virtual environment is active (you should see (venv) in your terminal prompt). Run venv\Scripts\activate if needed, then try the command again.

Help me fix the ModuleNotFoundError

App opens but nothing happens when you type?

Confirm Ollama is running by opening a new terminal and running ollama list. You should see nomic-embed-text and llama3.2 listed. If Ollama is not running, start it again.

Help me debug why my app is not responding

Your AI second brain is live. You now have a full RAG system running entirely on your machine. Next up, the Secret Mission takes your retrieval accuracy to the next level with hybrid search.

Secret mission

Add Hybrid Search (BM25 + Vector)

Your vector search finds semantically similar passages, but it can miss results that contain exact keywords. What if you combined vector search with classic keyword matching to get the best of both worlds? In this secret mission, you'll implement hybrid search using BM25 and Reciprocal Rank Fusion.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • ai-second-brain project folder on your Desktop (contains ingest.py, query.py, app.py, docs/, chroma_db/, and venv/).
  • Ollama application and downloaded models (llama3.2 and nomic-embed-text).

Keep everything running

No action needed. Choose this if you plan to keep adding documents to your second brain or want to experiment with different models and prompts.

  • Ollama stays installed and ready to serve models whenever you need it.
  • Your chroma_db/ folder retains all indexed documents for instant querying.
  • You can add new files to docs/ and re-run python ingest.py at any time to grow your knowledge base.

Pause - I'll come back to this later

Shut down running processes to free up memory while keeping all files intact for later.

  • Stop the Streamlit app by pressing Ctrl+C in the terminal where it is running.
  • Deactivate your virtual environment by running:
deactivate
  • Ollama runs as a background process. You can stop it by closing the Ollama application from your system tray.
  • All your files and the chroma_db/ database remain on disk for next time.

Delete - I don't want to use this again

Remove all project resources and start fresh. This permanently deletes your indexed documents and project code.

  • Stop the Streamlit app by pressing Ctrl+C in the terminal where it is running.
  • Deactivate your virtual environment by running:
deactivate
  • Delete the entire project folder by running:
cd ~/Desktop
rm -rf ai-second-brain

On Windows, use this command instead:

cd ~\Desktop
Remove-Item -Recurse -Force ai-second-brain
  • To uninstall Ollama on Windows, open Settings, then Apps, then Add or remove programs. Find Ollama in the list and click Uninstall. The uninstaller will offer to remove downloaded models as well.

What about macOS or Linux?

On macOS, drag the Ollama app from your Applications folder to the Trash. On Linux, remove the binary and service files. In both cases, delete the model data stored in ~/.ollama to reclaim disk space.

That's a Wrap!

That's a Wrap!

Nice work! You just built a fully local AI assistant that knows everything in your documents and answers questions with source citations.

You've learned how to:

  • Built a complete RAG pipeline from scratch with document loading, chunking, embedding via Ollama, and vector storage in ChromaDB.
  • Created a query engine that retrieves relevant context using semantic search and generates grounded answers with source citations.
  • Built a Streamlit chat interface to interact with your personal knowledge base, running 100% locally with zero cost and full privacy.
  • Secret Mission: Implemented hybrid search combining BM25 keyword matching with vector semantic search, fused using Reciprocal Rank Fusion for maximum retrieval accuracy.

Ready to quiz yourself?