Build a Local RAG Assistant

Build a local Python assistant that answers handbook questions with sources.

Introduction

30 Second Summary

Company handbooks are full of precise answers. A chatbot that has never seen one can still invent a confident policy.

In this project, you will build a terminal-based support assistant for Orbit Labs with Python and Ollama. Retrieval-augmented generation will ground each answer in the fictional employee handbook.

What You'll Build

Picture asking an employee handbook question and seeing the exact policy title beside the answer in your terminal.

By the end of this project, you'll have:

  • A baseline comparison that shows how a general model responds when Orbit Labs evidence is missing.
  • A grounded handbook assistant that answers questions from retrieved policies and displays its sources.
  • A retrieval inspection that ranks policies by similarity score. A repeatable evaluation that prints pass or fail for each known question.
  • Secret Mission: Add a confidence guardrail that refuses unrelated questions when retrieval evidence is too weak.

Are there any prerequisites?

You need a Mac running macOS 14 Sonoma or newer. Python 3, Visual Studio Code, and internet access cover the remaining prerequisites.

Before We Start

This first checkpoint locks in what you are building before any hands-on work begins. You are creating a local Orbit Labs handbook assistant whose answers can be traced to retrieved policy sections.

Set Up Ollama and Local Models

Your Orbit Labs assistant needs a model that can generate answers. It also needs an embedding model that can represent handbook text for retrieval.

Ollama provides a local inference server that Python can call later. This step prepares both models for that workflow.

In this step, get ready to:
  • Confirm your Mac meets Ollama's macOS requirement.
  • Install the Ollama local service.
  • Make both local models available through Ollama.
Check your macOS version

Ollama requires macOS 14 Sonoma or newer. Checking your version now confirms that your Mac can run the local service.

  • Click the Apple menu in the top-left corner of your screen.
  • Select About This Mac.
  • Check the macOS name shown in the window.
  • Check the version number shown below the macOS name.

✔️ I see macOS 14 or newer

Your Mac meets Ollama's current macOS requirement.

ⓧ I see macOS 13 or older

This Mac does not meet the project's Ollama requirement.

Install Ollama and start the local service

Ollama runs the generation model and embedding model on your Mac. This local installation requires no paid account.

  • Press Command-Space bar to open Spotlight.
  • Type Terminal in the search field.
  • Press Return to open Terminal.
  • Install Ollama from Terminal by running this command:
curl -fsSL https://ollama.com/install.sh | sh

What does this command do?

The curl command downloads Ollama's official installation script. The sh command runs that script on your Mac.

The installer adds Ollama so you can control the local service from Terminal.

  • Approve the prompt to open Ollama if it appears.
  • Wait for Terminal to return to its command prompt.
  • Check that the Ollama service responds by running this command:
ollama ls

What does this command do?

The ollama ls command asks the local service for its available models. A response confirms that Ollama is installed locally on the Mac.

You should see a model list with no connection failure. The list can be empty because you download the models next.

Ollama service not responding?

Press Command-Space bar to open Spotlight. Search for Ollama to start the local service.

Repeat the service check after Ollama starts.

Help me start my local Ollama service.

Download and verify both models

The gemma3:1b model generates the assistant's text. The all-minilm model converts handbook sections into vectors for semantic retrieval.

The first command downloads an 815 MB model before opening an interactive chat. A quiet wait during this download is normal.

  • Download the generation model by running this command:
ollama run gemma3:1b

What does this command do?

The command downloads gemma3:1b when needed. It then starts an interactive chat with the model.

This gives you a direct check that local text generation works before Python enters the workflow.

Once the download finishes, you should see an interactive prompt ready for your message.

  • Test the generation model by entering this prompt:
What is a handbook?

What does this test prove?

A generated response proves that gemma3:1b is available through your local Ollama service.

You should see a short answer generated by the model.

  • Leave the interactive chat by entering:
/bye

What does this command do?

The /bye command closes the Ollama chat. The downloaded model remains available on your Mac.

You should return to your normal Terminal prompt.

The embedding model is a smaller 46 MB download. It stays local with the generation model.

  • Download the embedding model by running this command:
ollama pull all-minilm

What does this command do?

The command downloads all-minilm into Ollama's local model storage. Python uses this model later through Ollama's embedding endpoint.

You should see the download complete before Terminal returns to its prompt.

Before you check, which two model identifiers do you expect to see in the local model list?

  • Verify both downloads by running this command:
ollama ls

What does this check prove?

The model list confirms which downloads are available through the running Ollama service. Both models must appear before the assistant can generate text or create embeddings.

You should see gemma3:1b and all-minilm listed as local models.

Missing one of the models?

Repeat the download command above for the missing model. Keep Ollama running until the download completes.

Check your internet connection if the download stops before completion.

Help me finish my Ollama model downloads.

You now have both halves of the local inference workflow available on your Mac.

Your local inference service is ready. Next up, you'll ask the generation model a company question without giving it the Orbit Labs handbook.

Build an Ungrounded Baseline

Your Ollama service is ready. Your local Gemma 3 1B model can now answer its first Orbit Labs question.

Orbit Labs is fictional. A general model has no reliable access to its private handbook. This baseline exposes that evidence gap before you add semantic retrieval.

In this step, get ready to:
  • Create the local-rag-assistant workspace.
  • Write five fictional Orbit Labs policy chunks.
  • Run a Python baseline without handbook evidence.
Create the project workspace

A dedicated workspace keeps both project files together. Opening that folder in Visual Studio Code also gives its terminal the correct starting location.

  • In Visual Studio Code, select File from the menu bar.
  • Select Open Folder....
  • Select Desktop in the folder picker.
  • Select New Folder.
  • Enter local-rag-assistant as the folder name.
  • Select Open.
  • Select Yes, I trust the authors if the Workspace Trust dialog appears.

Good. You should see local-rag-assistant at the top of the Explorer sidebar.

Folder missing from Explorer?

Repeat the folder selection from the File menu. Make sure you select the local-rag-assistant folder itself before choosing Open.

If the folder still does not appear, help me open my project folder in Visual Studio Code.

Write the fictional handbook

Each policy section becomes one retrievable chunk later. A single blank line creates a clear boundary between neighbouring policies.

  • Select New File in the Explorer sidebar.
  • Enter handbook.txt as the file name.
  • Add the five policy chunks by pasting this content:
# Remote Work Abroad
Employees may work outside their home country for up to 18 calendar days per calendar year. They must receive written approval from People Operations at least 10 business days before travel.

# Learning Budget
Each employee has an annual learning budget of $1,200 for approved courses, books, and certification exams. Manager approval is required before purchase.

# Security Incidents
A lost or stolen company laptop must be reported to the Security team within one hour. The employee must also notify their manager.

# Paid Leave
Full-time employees receive 22 days of paid leave per calendar year.

# Home Office Equipment
Employees can claim up to $500 for home office equipment once every two calendar years.

How is the handbook structured?

  • Each line beginning with # names a policy.
  • Each paragraph beneath a heading contains the policy evidence.
  • Each blank line separates one policy chunk from the next.
  • Save handbook.txt.
  • Confirm that the editor shows five policy headings.
  • Confirm that one blank line separates each policy chunk.

That gives your future retrieval pipeline five distinct pieces of evidence to compare.

Policy chunks running together?

Check for one empty line before every heading after Remote Work Abroad. Extra spaces on an otherwise empty line can hide a missing chunk boundary.

If the structure still looks different, help me format my handbook into five policy chunks.

✔️ Awesome, I've got everything!

Great. Double check that you saved handbook.txt inside local-rag-assistant.

ⓧ I'd like to double check the full code

# Remote Work Abroad
Employees may work outside their home country for up to 18 calendar days per calendar year. They must receive written approval from People Operations at least 10 business days before travel.

# Learning Budget
Each employee has an annual learning budget of $1,200 for approved courses, books, and certification exams. Manager approval is required before purchase.

# Security Incidents
A lost or stolen company laptop must be reported to the Security team within one hour. The employee must also notify their manager.

# Paid Leave
Full-time employees receive 22 days of paid leave per calendar year.

# Home Office Equipment
Employees can claim up to $500 for home office equipment once every two calendar years.

How to use this reference

Compare this file with your saved copy. Check every heading plus every blank line.

Build the ungrounded baseline

The baseline uses Python's standard library to call Ollama's local REST API. The model receives the employee question through a generation request.

Why direct HTTP?

Ollama already provides a local API for model requests. Python's standard library can call that API without adding a framework.

This keeps your attention on the evidence flow. Every request remains visible in your own code.

  • Select New File in the Explorer sidebar.
  • Enter app.py as the file name.
  • Add the API configuration plus the request helper by pasting this code:
import json
import sys
from urllib.request import Request, urlopen

BASE_URL = "http://localhost:11434"
GENERATION_MODEL = "gemma3:1b"
EMBEDDING_MODEL = "all-minilm"
HANDBOOK_FILE = "handbook.txt"


def post_json(path, payload):
    body = json.dumps(payload).encode("utf-8")
    request = Request(
        BASE_URL + path,
        data=body,
        headers={"Content-Type": "application/json"},
    )
    with urlopen(request, timeout=120) as response:
        return json.loads(response.read().decode("utf-8"))

What does this code do?

  • The json module converts Python data into request content.
  • The sys module reads the mode supplied when the script starts.
  • Request builds the local HTTP request.
  • urlopen sends the request to Ollama.
  • BASE_URL points to the local Ollama service.
  • post_json() sends a JSON payload to a chosen API path.
  • Save app.py.
  • Confirm that app.py appears beside handbook.txt in the Explorer sidebar.

Seeing a syntax warning?

Check the parentheses inside Request(). Check that the headers value keeps its braces.

For a closer comparison, help me debug the API helper in app.py.

  • Add the generation function plus the baseline function below post_json() by pasting this code:
def generate(prompt):
    result = post_json(
        "/api/generate",
        {
            "model": GENERATION_MODEL,
            "prompt": prompt,
            "stream": False,
            "options": {"temperature": 0},
        },
    )
    return result["response"].strip()


def run_baseline():
    prompt = (
        "You are answering without access to Orbit Labs' private handbook. "
        "Answer the employee question. If the company-specific policy is unknown, say so.\n\n"
        "Question: How many days per year can an Orbit Labs employee work "
        "outside their home country?"
    )
    print("Baseline answer:\n")
    print(generate(prompt))

How does the baseline work?

  • generate() sends the prompt to the local generation endpoint.
  • GENERATION_MODEL selects gemma3:1b for the response.
  • stream keeps the response in one JSON result.
  • temperature uses a fixed low-randomness setting for this comparison.
  • run_baseline() asks the policy question without adding handbook content.
  • Save app.py.
  • Confirm that the file now contains generate() beneath post_json().
  • Confirm that run_baseline() appears beneath generate().

Functions in the wrong place?

Make sure both function definitions begin at the far left of the editor. Keep the lines inside each function indented by four spaces.

If Visual Studio Code highlights either function, help me fix the indentation in my Python functions.

  • Add the command dispatcher at the bottom of app.py by pasting this code:
def main():
    if sys.argv[1:] == ["--baseline"]:
        run_baseline()


if __name__ == "__main__":
    main()

What starts the baseline?

  • main() checks the command-line arguments.
  • --baseline selects the baseline mode.
  • __main__ starts main() when Python runs this file directly.
  • Save app.py.

✔️ Awesome, I've got everything!

Great. Double check that both files are saved inside local-rag-assistant.

ⓧ I'd like to double check the full code

import json
import sys
from urllib.request import Request, urlopen

BASE_URL = "http://localhost:11434"
GENERATION_MODEL = "gemma3:1b"
EMBEDDING_MODEL = "all-minilm"
HANDBOOK_FILE = "handbook.txt"


def post_json(path, payload):
    body = json.dumps(payload).encode("utf-8")
    request = Request(
        BASE_URL + path,
        data=body,
        headers={"Content-Type": "application/json"},
    )
    with urlopen(request, timeout=120) as response:
        return json.loads(response.read().decode("utf-8"))


def generate(prompt):
    result = post_json(
        "/api/generate",
        {
            "model": GENERATION_MODEL,
            "prompt": prompt,
            "stream": False,
            "options": {"temperature": 0},
        },
    )
    return result["response"].strip()


def run_baseline():
    prompt = (
        "You are answering without access to Orbit Labs' private handbook. "
        "Answer the employee question. If the company-specific policy is unknown, say so.\n\n"
        "Question: How many days per year can an Orbit Labs employee work "
        "outside their home country?"
    )
    print("Baseline answer:\n")
    print(generate(prompt))


def main():
    if sys.argv[1:] == ["--baseline"]:
        run_baseline()


if __name__ == "__main__":
    main()

How to use this reference

Compare the function order with your saved file. Check the indentation inside every function.

  • Control-click local-rag-assistant in the Explorer sidebar.
  • Select Open in Integrated Terminal.

Before you run the baseline, do you think the model can prove an Orbit Labs policy from the question alone?

The first local response can take a little longer while Ollama loads Gemma 3 1B into memory.

  • Run the ungrounded baseline with this command:
python3 app.py --baseline

What does this command do?

Python runs app.py with the --baseline argument. The script sends the baseline prompt to the local Ollama generation endpoint.

You should see Baseline answer: followed by a model response. The exact wording can vary between runs.

The response contains no retrieved policy section. It contains no source score.

You have now exposed the key grounding gap. The answer has no traceable company evidence.

Baseline not returning an answer?

Make sure the Ollama service from the previous step is still running. A connection failure means Python cannot reach the local generation endpoint.

Check that your terminal is inside local-rag-assistant. Check that app.py is saved.

If the baseline still fails, help me debug my local Ollama baseline.

Your baseline can call a local model, but its answer carries no handbook evidence. Next, you will retrieve the policy sections that make an answer traceable.

Retrieve Relevant Policies

The baseline returned an answer from Gemma 3 1B. However, it could not point to evidence from the fictional Orbit Labs handbook.

Semantic retrieval closes that gap by converting each policy into an embedding. Cosine similarity then ranks the handbook sections against the employee's question.

In this step, get ready to:
  • Load the handbook policies as separate chunks.
  • Build a semantic ranking pipeline for the handbook chunks.
  • Display a scored policy ranking in the terminal.
Embed the handbook chunks

An embedding represents the meaning of text as a list of numbers. Ollama can create one vector for each policy through its local embedding endpoint.

  • In app.py, locate generate() above run_baseline().
  • Add both data-preparation functions below generate() by copying this code:
def embed(texts):
    result = post_json("/api/embed", {"model": EMBEDDING_MODEL, "input": texts})
    return result["embeddings"]


def load_chunks():
    with open(HANDBOOK_FILE, "r", encoding="utf-8") as handbook:
        return [chunk.strip() for chunk in handbook.read().split("\n\n") if chunk.strip()]

What does this code do?

  • The embed() function sends text to /api/embed using the local all-minilm model. It returns the vectors from the embeddings field in the JSON response.
  • The load_chunks() function reads handbook.txt. It treats each blank-line-separated policy as one retrievable chunk.
  • Save app.py.

Before you run the script, predict whether the original baseline path still completes after Python reads the new function definitions.

  • Confirm the baseline path still runs by using:
python3 app.py --baseline

What does this check prove?

This command confirms that Python can read the new functions without a syntax problem. It also confirms that the existing local generation path still works.

You should see Baseline answer: followed by another unsupported model response.

Did the baseline stop working?

Check that embed() starts at the left edge of app.py. Make sure the body of each new function uses consistent indentation.

Confirm that each separator inside split("\n\n") contains two escaped newline characters.

Help me fix the new embedding and handbook-loading functions.

Rank chunks by meaning

Each policy vector needs a score against the question vector. Cosine similarity measures how closely the two vectors point in the same direction.

  • In app.py, locate load_chunks().
  • Add the similarity function below load_chunks() by copying this code:
def cosine_similarity(left, right):
    numerator = sum(a * b for a, b in zip(left, right))
    left_length = sum(value * value for value in left) ** 0.5
    right_length = sum(value * value for value in right) ** 0.5
    denominator = left_length * right_length
    return numerator / denominator if denominator else 0.0

How is the score calculated?

  • The numerator combines matching positions from the two vectors.
  • The length calculations normalize the result so vectors with larger raw values do not dominate the comparison.
  • The denominator guard returns 0.0 if either vector has no length.
  • Save app.py.
  • Check that the file still parses by running the baseline command again:
python3 app.py --baseline

Why rerun the baseline?

The baseline remains the quickest runnable path while you assemble retrieval. A successful response confirms that the new similarity function has valid Python syntax.

You should see the baseline heading followed by a generated response. Your existing model path still works after the similarity calculation was added.

Did the similarity function break the script?

Compare the indentation inside cosine_similarity() with the code above. Check that the final expression ends with 0.0.

Help me debug the cosine similarity function.

A similarity function scores one pair of vectors. The retrieval function applies that comparison to every handbook chunk before sorting the results.

  • Add retrieve() below cosine_similarity() by copying this code:
def retrieve(question, chunks, chunk_vectors, top_k=2):
    question_vector = embed(question)[0]
    scored_chunks = []
    for chunk, vector in zip(chunks, chunk_vectors):
        scored_chunks.append((chunk, cosine_similarity(question_vector, vector)))
    scored_chunks.sort(key=lambda item: item[1], reverse=True)
    return scored_chunks[:top_k]

How does retrieval work?

  • The question passes through embed() so it uses the same vector format as the handbook chunks.
  • Each chunk receives a similarity score against the question.
  • The reverse sort places the highest score first. The top_k value controls how many ranked chunks are returned.
  • Save app.py.
  • Confirm the script still runs by using the baseline command:
python3 app.py --baseline

What does this run confirm?

Python now accepts the complete ranking function. The baseline path also continues to reach the local generation model.

You should see another baseline response. The new retrieval functions are ready to be connected to an inspection mode.

Did retrieval introduce a syntax problem?

Check that the for loop body is indented beneath the loop. Keep the sort and return lines aligned with the loop itself.

Help me debug the retrieve function.

Inspect the retrieval ranking

A ranked list becomes useful once you can read the policy names beside their scores. Inspection mode exposes those results before any answer is generated.

  • In app.py, locate run_baseline().
  • Add the title helper plus the inspection function above run_baseline() by copying this code:
def source_title(chunk):
    return chunk.splitlines()[0].lstrip("# ").strip()


def run_inspection(chunks, chunk_vectors):
    question = "How long can I work from another country?"
    print("Question: " + question)
    print("\nRetrieval ranking:")
    for chunk, score in retrieve(question, chunks, chunk_vectors, top_k=3):
        print("- {0}: {1:.3f}".format(source_title(chunk), score))

What does inspection mode show?

  • The source_title() helper extracts the first line from a chunk. It removes the handbook heading marker.
  • The run_inspection() function retrieves three policies for a fixed test question.
  • Each score is formatted to three decimal places so the ranking is easy to compare.
  • Save app.py.
  • Confirm the new inspection helpers have valid syntax by running the baseline path:
python3 app.py --baseline

What does this syntax check cover?

Python reads both inspection helpers before it starts main(). Reaching the baseline response confirms those definitions are valid.

You should see the familiar baseline output. The last task is to route inspection mode through the new retrieval pipeline.

Did the inspection helpers stop the baseline?

Check the quote marks around the fixed question. Confirm that the for loop is indented inside run_inspection().

Help me debug the inspection helpers.

  • In app.py, scroll to main() near the bottom of the file.
  • Find this current function:
def main():
    if sys.argv[1:] == ["--baseline"]:
        run_baseline()

What needs to change?

The current main() only handles baseline mode. Inspection needs the handbook chunks plus their vectors before it can rank anything.

  • Replace the current main() with this updated version:
def main():
    if sys.argv[1:] == ["--baseline"]:
        run_baseline()
        return
    chunks = load_chunks()
    chunk_vectors = embed(chunks)
    if sys.argv[1:] == ["--inspect"]:
        run_inspection(chunks, chunk_vectors)

How is inspection mode connected?

  • The early return keeps baseline mode separate from handbook retrieval.
  • The normal path loads every policy chunk from handbook.txt.
  • A single batch embedding request creates the reusable policy vectors. The --inspect branch passes those vectors into run_inspection().
  • Save app.py.

The first embedding request can pause briefly while Ollama loads all-minilm into memory.

Before you run inspection mode, predict which policy ranks first for a question about another country.

  • Run the completed retrieval inspection with:
python3 app.py --inspect

What does this command test?

The command runs the complete retrieval path. It loads the handbook before embedding every policy.

The test question receives its own embedding. The script then prints the three closest policy sections with their similarity scores.

You should see Question: How long can I work from another country? above a scored ranking. Remote Work Abroad should appear first.

That retrieval ranking is your first evidence-backed result. Your application can now select a policy before it generates a single word.

Is the ranking missing or incorrect?

If the command cannot reach the embedding service, confirm Ollama is still running. Check that the all-minilm model remains available locally.

If Remote Work Abroad is missing from first place, confirm that handbook.txt still contains five policies separated by blank lines.

Help me fix the retrieval ranking.

  • Choose the full-code tab to compare app.py with the exact cumulative version.
  • Update any differences before completing the checkpoint.

✔️ Awesome, I've got everything!

Great. Double check that you saved app.py after confirming the retrieval ranking.

ⓧ I'd like to double check the full code

import json
import sys
from urllib.request import Request, urlopen

BASE_URL = "http://localhost:11434"
GENERATION_MODEL = "gemma3:1b"
EMBEDDING_MODEL = "all-minilm"
HANDBOOK_FILE = "handbook.txt"


def post_json(path, payload):
    body = json.dumps(payload).encode("utf-8")
    request = Request(BASE_URL + path, data=body, headers={"Content-Type": "application/json"})
    with urlopen(request, timeout=120) as response:
        return json.loads(response.read().decode("utf-8"))


def generate(prompt):
    result = post_json("/api/generate", {"model": GENERATION_MODEL, "prompt": prompt, "stream": False, "options": {"temperature": 0}})
    return result["response"].strip()


def embed(texts):
    result = post_json("/api/embed", {"model": EMBEDDING_MODEL, "input": texts})
    return result["embeddings"]


def load_chunks():
    with open(HANDBOOK_FILE, "r", encoding="utf-8") as handbook:
        return [chunk.strip() for chunk in handbook.read().split("\n\n") if chunk.strip()]


def cosine_similarity(left, right):
    numerator = sum(a * b for a, b in zip(left, right))
    left_length = sum(value * value for value in left) ** 0.5
    right_length = sum(value * value for value in right) ** 0.5
    denominator = left_length * right_length
    return numerator / denominator if denominator else 0.0


def retrieve(question, chunks, chunk_vectors, top_k=2):
    question_vector = embed(question)[0]
    scored_chunks = []
    for chunk, vector in zip(chunks, chunk_vectors):
        scored_chunks.append((chunk, cosine_similarity(question_vector, vector)))
    scored_chunks.sort(key=lambda item: item[1], reverse=True)
    return scored_chunks[:top_k]


def source_title(chunk):
    return chunk.splitlines()[0].lstrip("# ").strip()


def run_baseline():
    prompt = "You are answering without access to Orbit Labs' private handbook. Answer the employee question. If the company-specific policy is unknown, say so.\n\nQuestion: How many days per year can an Orbit Labs employee work outside their home country?"
    print("Baseline answer:\n")
    print(generate(prompt))


def run_inspection(chunks, chunk_vectors):
    question = "How long can I work from another country?"
    print("Question: " + question)
    print("\nRetrieval ranking:")
    for chunk, score in retrieve(question, chunks, chunk_vectors, top_k=3):
        print("- {0}: {1:.3f}".format(source_title(chunk), score))


def main():
    if sys.argv[1:] == ["--baseline"]:
        run_baseline()
        return
    chunks = load_chunks()
    chunk_vectors = embed(chunks)
    if sys.argv[1:] == ["--inspect"]:
        run_inspection(chunks, chunk_vectors)


if __name__ == "__main__":
    main()

Your retriever can now turn a question into ranked handbook evidence. Next up, you'll pass that evidence to the generation model and display the policies used as sources.

Generate Grounded Answers

Your semantic retrieval pipeline now ranks handbook sections by relevance. You can see which evidence best matches an employee's question.

Retrieval alone only identifies evidence. The next layer sends that evidence to Gemma 3 1B through Ollama. It uses grounding to constrain each answer. It also adds source attribution so the supporting policies remain visible.

In this step, get ready to:
  • Build a handbook-only prompt from the top two retrieved sections.
  • Add an interactive loop that displays answers with source titles and scores.
  • Compare an in-scope question with a question the handbook does not answer.
Build the grounded answer path

A grounded prompt gives the model a defined evidence boundary. Your function joins the top two retrieved sections into that boundary before asking the model to answer.

  • In app.py, locate the end of run_inspection().
  • Add answer_question() below run_inspection() by copying this code:
def answer_question(question, chunks, chunk_vectors):
    retrieved = retrieve(question, chunks, chunk_vectors)
    context = "\n\n---\n\n".join(chunk for chunk, score in retrieved)
    prompt = "You are Orbit Labs' support assistant. Answer only from the CONTEXT. If the answer is not in the context, say: I could not find that in the handbook. Keep the answer concise and name the relevant policy.\n\nCONTEXT:\n" + context + "\n\nQUESTION:\n" + question
    return generate(prompt), retrieved

What does this code do?

  • The retrieve() call selects the two policy sections with the highest similarity scores.
  • The context value joins those sections with a visible separator.
  • The prompt requires the model to use only the retrieved handbook text.
  • The return value keeps the generated answer beside the evidence used to create it.
  • Save app.py.
  • Confirm the existing retrieval path still works by running:
python3 app.py --inspect

What does this check prove?

The inspection mode exercises the same embedding and ranking path that answer_question() depends on. A complete ranking confirms that the new function has not introduced a syntax problem.

You should still see Remote Work Abroad ranked first. Its similarity score appears beside the policy title.

Did the inspection stop working?

Check that answer_question() begins at the left edge of app.py. Make sure each line inside the function uses the same indentation.

Confirm that the running Ollama service still has all-minilm available. The inspection path needs that model to embed the question.

Help me fix the grounded answer function.

Add the interactive question loop

A question loop keeps the handbook embeddings available for repeated requests. Each response can reuse those vectors without restarting the script.

  • Add run_interactive() immediately below answer_question() by copying this code:
def run_interactive(chunks, chunk_vectors):
    print("Orbit Labs Handbook Assistant")
    print("Type q to quit.\n")
    while True:
        question = input("Question: ").strip()
        if question.lower() == "q":
            break
        if not question:
            continue
        answer, retrieved = answer_question(question, chunks, chunk_vectors)
        print("\nAnswer:\n" + answer)
        print("\nSources:")
        for chunk, score in retrieved:
            print("- {0} ({1:.3f})".format(source_title(chunk), score))
        print()

How does the loop work?

  • The loop waits at the Question: prompt for each employee question.
  • Entering q exits the loop.
  • An empty question restarts the loop without calling either model.
  • Each completed request prints the answer before listing the retrieved policy titles with three-decimal scores.
  • In main(), locate the branch that calls run_inspection(chunks, chunk_vectors).
  • Add an else branch beneath that call so the branch looks like this:
    if sys.argv[1:] == ["--inspect"]:
        run_inspection(chunks, chunk_vectors)
    else:
        run_interactive(chunks, chunk_vectors)

Why add the else branch?

The existing branch preserves inspection mode when the matching command argument is present. Every normal run now starts run_interactive() with the handbook chunks and their vectors.

  • Save app.py.
Test answers and source attribution

Before you run the assistant, which handbook fact do you expect it to return for the remote-work question? Which policy title should support that answer?

  • Start the interactive assistant by running:
python3 app.py

What happens when the assistant starts?

The script loads the handbook before embedding all five sections. It then displays the assistant title with a Question: prompt.

The first response can take longer while gemma3:1b loads into memory. The terminal is still working while you wait.

  • Enter How many days can I work outside my home country? at the Question: prompt.

You should see an answer stating 18 calendar days. You should also see Remote Work Abroad in the Sources list.

  • Enter What is the office dress code? at the next Question: prompt.

The handbook contains no dress-code policy. You should see I could not find that in the handbook. while the Sources list still shows the closest retrieved sections.

  • Enter q at the next prompt to exit the assistant.

You've closed the evidence gap. The assistant now answers from selected handbook text with visible policy evidence.

Missing the answer or Sources list?

Check that the normal path in main() calls run_interactive(chunks, chunk_vectors). Confirm that the call sits inside the else branch.

If the assistant accepts a question but cannot generate an answer, confirm the running Ollama service still has gemma3:1b available.

Help me debug the interactive assistant.

✔️ Awesome, I've got everything!

Great. Double check that app.py is saved after your successful grounded-answer test.

ⓧ I'd like to double check the full code

import json
import sys
from urllib.request import Request, urlopen

BASE_URL = "http://localhost:11434"
GENERATION_MODEL = "gemma3:1b"
EMBEDDING_MODEL = "all-minilm"
HANDBOOK_FILE = "handbook.txt"


def post_json(path, payload):
    body = json.dumps(payload).encode("utf-8")
    request = Request(BASE_URL + path, data=body, headers={"Content-Type": "application/json"})
    with urlopen(request, timeout=120) as response:
        return json.loads(response.read().decode("utf-8"))


def generate(prompt):
    result = post_json("/api/generate", {"model": GENERATION_MODEL, "prompt": prompt, "stream": False, "options": {"temperature": 0}})
    return result["response"].strip()


def embed(texts):
    return post_json("/api/embed", {"model": EMBEDDING_MODEL, "input": texts})["embeddings"]


def load_chunks():
    with open(HANDBOOK_FILE, "r", encoding="utf-8") as handbook:
        return [chunk.strip() for chunk in handbook.read().split("\n\n") if chunk.strip()]


def cosine_similarity(left, right):
    numerator = sum(a * b for a, b in zip(left, right))
    denominator = (sum(value * value for value in left) ** 0.5) * (sum(value * value for value in right) ** 0.5)
    return numerator / denominator if denominator else 0.0


def retrieve(question, chunks, chunk_vectors, top_k=2):
    question_vector = embed(question)[0]
    scored_chunks = [(chunk, cosine_similarity(question_vector, vector)) for chunk, vector in zip(chunks, chunk_vectors)]
    scored_chunks.sort(key=lambda item: item[1], reverse=True)
    return scored_chunks[:top_k]


def source_title(chunk):
    return chunk.splitlines()[0].lstrip("# ").strip()


def run_baseline():
    prompt = "You are answering without access to Orbit Labs' private handbook. Answer the employee question. If the company-specific policy is unknown, say so.\n\nQuestion: How many days per year can an Orbit Labs employee work outside their home country?"
    print("Baseline answer:\n")
    print(generate(prompt))


def run_inspection(chunks, chunk_vectors):
    question = "How long can I work from another country?"
    print("Question: " + question)
    print("\nRetrieval ranking:")
    for chunk, score in retrieve(question, chunks, chunk_vectors, top_k=3):
        print("- {0}: {1:.3f}".format(source_title(chunk), score))


def answer_question(question, chunks, chunk_vectors):
    retrieved = retrieve(question, chunks, chunk_vectors)
    context = "\n\n---\n\n".join(chunk for chunk, score in retrieved)
    prompt = "You are Orbit Labs' support assistant. Answer only from the CONTEXT. If the answer is not in the context, say: I could not find that in the handbook. Keep the answer concise and name the relevant policy.\n\nCONTEXT:\n" + context + "\n\nQUESTION:\n" + question
    return generate(prompt), retrieved


def run_interactive(chunks, chunk_vectors):
    print("Orbit Labs Handbook Assistant")
    print("Type q to quit.\n")
    while True:
        question = input("Question: ").strip()
        if question.lower() == "q":
            break
        if not question:
            continue
        answer, retrieved = answer_question(question, chunks, chunk_vectors)
        print("\nAnswer:\n" + answer)
        print("\nSources:")
        for chunk, score in retrieved:
            print("- {0} ({1:.3f})".format(source_title(chunk), score))
        print()


def main():
    if sys.argv[1:] == ["--baseline"]:
        run_baseline()
        return
    chunks = load_chunks()
    chunk_vectors = embed(chunks)
    if sys.argv[1:] == ["--inspect"]:
        run_inspection(chunks, chunk_vectors)
    else:
        run_interactive(chunks, chunk_vectors)


if __name__ == "__main__":
    main()

Your assistant can now turn retrieved policies into grounded answers with visible sources. Next, you'll build a repeatable check for whether retrieval selects the correct evidence.

Evaluate Retrieval Quality

Your grounded assistant can now answer handbook questions. It also displays the policy sections used as evidence.

A convincing answer can still rely on the wrong evidence. In this step, you'll add a repeatable retrieval evaluation that checks the top policy for three known questions.

In this step, get ready to:
  • Define three questions with their expected policy titles.
  • Measure whether retrieval selects the correct policy for every question.
  • Confirm the evaluation passes without breaking the interactive assistant.
Define the evaluation cases

Generated answers can change between runs. Expected policy titles give you a stable result that directly measures evidence selection.

  • Find HANDBOOK_FILE = "handbook.txt" near the top of app.py.
  • Add the evaluation cases directly below that line by pasting this code:
EVAL_CASES = [
    ("How long can I work from another country?", "Remote Work Abroad"),
    ("What can I spend on courses each year?", "Learning Budget"),
    ("How quickly must I report a stolen work laptop?", "Security Incidents"),
]

What do these cases represent?

  • Each tuple pairs a natural-language question with its expected handbook policy.
  • The expected titles match the headings returned by source_title().
  • Together, the three tuples form a small test dataset for retrieval.
  • Save app.py.
  • Confirm the editor shows three question-title pairs below HANDBOOK_FILE.

Do the cases show a syntax warning?

Check that every case has opening and closing parentheses. Confirm the list ends with a closing square bracket.

Compare each policy title with its heading in handbook.txt.

Help me check my evaluation cases.

Build the retrieval evaluator

The evaluator should use the same retrieval path as the assistant. This keeps the test focused on the evidence your real workflow selects.

  • Find the end of run_interactive() in app.py.
  • Add evaluate_retrieval() below it by pasting this code:
def evaluate_retrieval(chunks, chunk_vectors):
    passed = 0
    for question, expected_title in EVAL_CASES:
        top_chunk, score = retrieve(question, chunks, chunk_vectors, top_k=1)[0]
        actual_title = source_title(top_chunk)
        result = "PASS" if actual_title == expected_title else "FAIL"
        if result == "PASS":
            passed += 1
        print("{0}: expected {1}, got {2} ({3:.3f})".format(result, expected_title, actual_title, score))
    print("Retrieval accuracy: {0}/{1}".format(passed, len(EVAL_CASES)))

How does the evaluator work?

  • The loop sends every test question through retrieve() with top_k=1.
  • The retrieval path embeds each question. It compares that vector with every handbook vector using cosine similarity.
  • The evaluator converts the winning chunk into a title with source_title().
  • The final line reports how many expected titles matched the retrieved titles.
  • Save app.py.
  • Confirm evaluate_retrieval() appears between run_interactive() and main().

Does the evaluator look nested?

Place def evaluate_retrieval at the left edge of the editor. A leading indent would place it inside run_interactive().

Help me place evaluate_retrieval correctly.

The function now exists, so main() needs a route for the evaluation flag. That route keeps evaluation separate from normal interactive questions.

  • Scroll to main() in app.py.
  • Find the current inspection branch:
if sys.argv[1:] == ["--inspect"]:
        run_inspection(chunks, chunk_vectors)
    else:
        run_interactive(chunks, chunk_vectors)

What does the current branch do?

The current branch recognizes inspection mode. Every other command starts the interactive assistant.

  • Replace that branch with this updated version:
if sys.argv[1:] == ["--inspect"]:
        run_inspection(chunks, chunk_vectors)
    elif sys.argv[1:] == ["--eval"]:
        evaluate_retrieval(chunks, chunk_vectors)
    else:
        run_interactive(chunks, chunk_vectors)

What does the new branch do?

The new elif sends the evaluation flag to evaluate_retrieval(). Commands without a recognized flag still reach the interactive assistant.

  • Save app.py.

Before you run the evaluation, do you expect every test question to retrieve its matching policy?

  • Run the retrieval evaluation from the local-rag-assistant folder with this command:
python3 app.py --eval

What should you see?

The command runs all three questions through the same embedding and ranking path as the assistant. You should see three result lines that begin with PASS.

The final line should read Retrieval accuracy: 3/3. That score proves each test question selected its expected policy.

You now have a repeatable quality check. Your retrieval pipeline can prove that it selects the right evidence for all three known questions.

Did a case fail?

Check that each expected title in EVAL_CASES exactly matches the corresponding heading in handbook.txt.

If no evaluation results appear, confirm the new --eval branch sits before the final else block.

Help me debug my retrieval evaluation.

✔️ Awesome, I've got everything!

Great. Double check that you saved app.py after seeing Retrieval accuracy: 3/3.

ⓧ I'd like to double check the full code

import json
import sys
from urllib.request import Request, urlopen

BASE_URL = "http://localhost:11434"
GENERATION_MODEL = "gemma3:1b"
EMBEDDING_MODEL = "all-minilm"
HANDBOOK_FILE = "handbook.txt"

EVAL_CASES = [
    ("How long can I work from another country?", "Remote Work Abroad"),
    ("What can I spend on courses each year?", "Learning Budget"),
    ("How quickly must I report a stolen work laptop?", "Security Incidents"),
]


def post_json(path, payload):
    body = json.dumps(payload).encode("utf-8")
    request = Request(BASE_URL + path, data=body, headers={"Content-Type": "application/json"})
    with urlopen(request, timeout=120) as response:
        return json.loads(response.read().decode("utf-8"))


def generate(prompt):
    result = post_json("/api/generate", {"model": GENERATION_MODEL, "prompt": prompt, "stream": False, "options": {"temperature": 0}})
    return result["response"].strip()


def embed(texts):
    return post_json("/api/embed", {"model": EMBEDDING_MODEL, "input": texts})["embeddings"]


def load_chunks():
    with open(HANDBOOK_FILE, "r", encoding="utf-8") as handbook:
        return [chunk.strip() for chunk in handbook.read().split("\n\n") if chunk.strip()]


def cosine_similarity(left, right):
    numerator = sum(a * b for a, b in zip(left, right))
    denominator = (sum(value * value for value in left) ** 0.5) * (sum(value * value for value in right) ** 0.5)
    return numerator / denominator if denominator else 0.0


def retrieve(question, chunks, chunk_vectors, top_k=2):
    question_vector = embed(question)[0]
    scored_chunks = [(chunk, cosine_similarity(question_vector, vector)) for chunk, vector in zip(chunks, chunk_vectors)]
    scored_chunks.sort(key=lambda item: item[1], reverse=True)
    return scored_chunks[:top_k]


def source_title(chunk):
    return chunk.splitlines()[0].lstrip("# ").strip()


def run_baseline():
    prompt = "You are answering without access to Orbit Labs' private handbook. Answer the employee question. If the company-specific policy is unknown, say so.\n\nQuestion: How many days per year can an Orbit Labs employee work outside their home country?"
    print("Baseline answer:\n")
    print(generate(prompt))


def run_inspection(chunks, chunk_vectors):
    question = "How long can I work from another country?"
    print("Question: " + question)
    print("\nRetrieval ranking:")
    for chunk, score in retrieve(question, chunks, chunk_vectors, top_k=3):
        print("- {0}: {1:.3f}".format(source_title(chunk), score))


def answer_question(question, chunks, chunk_vectors):
    retrieved = retrieve(question, chunks, chunk_vectors)
    context = "\n\n---\n\n".join(chunk for chunk, score in retrieved)
    prompt = "You are Orbit Labs' support assistant. Answer only from the CONTEXT. If the answer is not in the context, say: I could not find that in the handbook. Keep the answer concise and name the relevant policy.\n\nCONTEXT:\n" + context + "\n\nQUESTION:\n" + question
    return generate(prompt), retrieved


def run_interactive(chunks, chunk_vectors):
    print("Orbit Labs Handbook Assistant")
    print("Type q to quit.\n")
    while True:
        question = input("Question: ").strip()
        if question.lower() == "q":
            break
        if not question:
            continue
        answer, retrieved = answer_question(question, chunks, chunk_vectors)
        print("\nAnswer:\n" + answer)
        print("\nSources:")
        for chunk, score in retrieved:
            print("- {0} ({1:.3f})".format(source_title(chunk), score))
        print()


def evaluate_retrieval(chunks, chunk_vectors):
    passed = 0
    for question, expected_title in EVAL_CASES:
        top_chunk, score = retrieve(question, chunks, chunk_vectors, top_k=1)[0]
        actual_title = source_title(top_chunk)
        result = "PASS" if actual_title == expected_title else "FAIL"
        if result == "PASS":
            passed += 1
        print("{0}: expected {1}, got {2} ({3:.3f})".format(result, expected_title, actual_title, score))
    print("Retrieval accuracy: {0}/{1}".format(passed, len(EVAL_CASES)))


def main():
    if sys.argv[1:] == ["--baseline"]:
        run_baseline()
        return
    chunks = load_chunks()
    chunk_vectors = embed(chunks)
    if sys.argv[1:] == ["--inspect"]:
        run_inspection(chunks, chunk_vectors)
    elif sys.argv[1:] == ["--eval"]:
        evaluate_retrieval(chunks, chunk_vectors)
    else:
        run_interactive(chunks, chunk_vectors)


if __name__ == "__main__":
    main()

How to use this reference

This version combines the three evaluation cases with the evaluator and its command route. Compare it with your saved app.py to catch any missing lines or indentation differences.

Recheck the interactive assistant

The evaluation flag adds a separate route through main(). The default route should still start the interactive question loop.

Before you run the assistant, do you expect the evaluation branch to change its normal question flow?

  • Start the interactive assistant from the local-rag-assistant folder with this command:
python3 app.py

What does this command check?

Running the script without a flag reaches the final else branch. You should see the assistant title followed by the Question: prompt.

  • Enter How quickly must I report a stolen work laptop? at the Question: prompt.

You should see the one-hour reporting requirement. You should also see Security Incidents under Sources:.

  • Enter q at the next prompt to exit.

That's the final workflow confirmed. Your assistant still answers questions while its retrieval quality now has a repeatable test.

Does the evaluation run instead?

Check that evaluate_retrieval() only appears inside the --eval branch. Keep run_interactive() inside the final else block.

Help me restore the interactive route.

Secret mission

Add a Low-Confidence Guardrail

Every question currently receives a closest policy section, even when the evidence is weak. In this mission, you will add a similarity threshold that refuses unrelated questions before they reach the generation model.

Clean Up Your Resources

Clean Up Your Resources

Everything you created runs on your Mac, so there are no cloud charges or ongoing service costs. Choose whether to keep your resources running, pause them for later, or delete them to recover local disk space.

Resources you used:

  • The gemma3:1b model files stored by Ollama.
  • The all-minilm model files.
  • Active Ollama model processes left from your tests.
  • The local-rag-assistant folder containing app.py and handbook.txt.

Keep everything running

No action needed. Choose this if you want to keep testing your local support assistant.

  • Your local models stay downloaded and ready to use.
  • Your local-rag-assistant folder stays on your Mac.
  • Ollama remains installed on your Mac.
  • Your project continues to cost $0.

Pause - I'll come back to this later

Stop active model processes to free up memory. Your downloaded models and project files remain available for your next session.

  • Stop any model processes left from your tests by running these commands:
ollama stop gemma3:1b
ollama stop all-minilm

What Do These Commands Do?

These commands release memory used by active local inference processes. They keep the downloaded model files on disk.

A model that is already inactive has no memory to release. You can leave it as it is.

  • Keep the local-rag-assistant folder so your Python code and fictional handbook remain available.

Delete - I don't want to use this again

Remove the downloaded models to recover their disk space. The final folder command permanently deletes your Python code and fictional handbook.

  • Stop any remaining model processes by running these commands:
ollama stop gemma3:1b
ollama stop all-minilm

Why Stop the Models First?

Stopping the models releases any memory still assigned to them. Their downloaded files remain available until you remove them in the next action.

  • Remove both downloaded models by running these commands:
ollama rm gemma3:1b
ollama rm all-minilm

What Do These Commands Remove?

These commands remove the downloaded model files from local storage. Ollama remains installed on your Mac.

Models Not Removed?

Make sure the Ollama application is running before you retry the removal commands.

Ask for help with the model cleanup commands.

  • Confirm the model files are removed by running this command:
ollama ls

What Should You See?

The local model list should no longer include gemma3:1b or all-minilm.

Finding a full folder path can be fiddly. The path variable below keeps the deletion command focused on the exact folder you choose.

  • Locate the local-rag-assistant folder in Finder.
  • Record its full path here: full path to your local-rag-assistant folder.

Deleting the folder is the final irreversible part. This command targets only the folder path you recorded.

  • Delete the local-rag-assistant folder by running this command in Terminal:
rm -rf "[[PROJECT_PATH=\"full path to your local-rag-assistant folder\"]]"

What Does This Command Do?

The rm -rf command permanently removes the folder at the recorded path. The quotation marks protect paths that contain spaces.

Folder Still There?

Check that the recorded path ends with local-rag-assistant before you retry the command.

Ask for help checking the folder path.

  • Return to Finder to confirm that the local-rag-assistant folder no longer appears in its previous location.

Nice Work!

Nice Work!

You did it! Your local RAG support assistant now answers Orbit Labs handbook questions with cited evidence through Ollama.

You've learned how to:

  • Build a semantic retrieval pipeline that turns handbook sections into embeddings. Cosine similarity ranks each section against an employee's question.
  • Use grounding to restrict generated answers to retrieved handbook evidence. Source attribution reveals the policy titles behind each response.
  • Create a repeatable retrieval evaluation with known questions. Its pass or fail results verify whether the correct policy ranks first.
  • Secret Mission: Add a low-confidence guardrail that compares the top score with MIN_SIMILARITY. Unrelated questions now receive a refusal before generation.

Ready to quiz yourself?