Build a RAG System with Evaluation
Build a fitness exercise RAG pipeline with BM25 baseline and eval metrics.
Introduction
30 Second Summary
When you ask an AI "what exercises target my glutes without equipment?" the quality of the answer depends entirely on what information gets retrieved first. The difference between a helpful response and a hallucinated one is retrieval quality.
In this project, you will build a Retrieval-Augmented Generation system over 800+ fitness exercises, measure its retrieval quality against a keyword baseline, and deploy an interactive web app for live demos.
What You'll Build
A production-grade Fitness Exercise RAG system that retrieves relevant exercises from an 800+ exercise knowledge base and generates contextual answers using a free LLM, complete with an evaluation harness that measures retrieval quality against a BM25 baseline.
By the end of this project, you'll have:
- A RAG pipeline that ingests 800+ exercises from a public dataset and answers natural language fitness questions using ChromaDB vector search.
- An evaluation framework with a 20-question test set, a BM25 keyword baseline, and precision/recall metrics that prove your system works.
- A Streamlit web app where anyone can interact with your RAG system in under 30 seconds.
- Secret Mission: Add hybrid retrieval using Reciprocal Rank Fusion and prove it outperforms both individual approaches in your eval harness.
Are there any prerequisites?
This project assumes you have Python experience, familiarity with LLM APIs, and a basic understanding of embeddings. You do not need prior experience with ChromaDB, LangChain, or evaluation engineering.
Before We Start
You're about to build a RAG system that retrieves fitness exercises from a knowledge base of 800+ exercises and generates answers using a free LLM.
Before writing any code, you need a project folder, a Python virtual environment with the right packages, and access to a free LLM via OpenRouter. This step gets all of that in place.
In this step, get ready to:
- Create your project folder and Python virtual environment.
- Install all required Python packages for RAG, evaluation, and UI.
- Configure free LLM access through OpenRouter.
Set up your project and install dependencies
Your RAG system depends on several Python libraries for vector search, LLM calls, keyword retrieval, and the web UI. A virtual environment keeps these isolated from your system Python.
- Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
- Type Terminal and press Enter to open it.
- Move to your Desktop by running this command:
cd ~/Desktop- Create your project folder and move into it by running these commands:
mkdir fitrag
cd fitrag- Create a Python virtual environment by running:
python3 -m venv venv- Activate the virtual environment by running the command for your operating system:
macOS
source venv/bin/activateWindows
venv\Scripts\activateYou should see (venv) appear at the start of your terminal prompt. This confirms your virtual environment is active.
Don't see (venv) in your prompt?
On Windows, you may need to enable script execution first. Run Set-ExecutionPolicy -Scope CurrentUser RemoteSigned in PowerShell, then try activating again.
On macOS, make sure you typed source before the path. Without it, the activation script won't run in your current shell.
help me activate my Python virtual environment
- Create a requirements.txt file inside your fitrag folder. This file lists every Python package the project needs. Paste the following content into it:
langchain==1.3.4
langchain-openai==1.2.2
chromadb==1.5.9
streamlit==1.58.0
rank-bm25==0.2.2
requests
numpyWhat are these packages?
- langchain and langchain-openai provide the LLM orchestration layer for building the RAG chain.
- chromadb is the vector database that stores and retrieves exercise embeddings locally.
- streamlit powers the interactive web UI for querying exercises.
- rank-bm25 provides the keyword-based BM25 retriever you will use as a baseline for evaluation.
- Install all the packages by running:
pip install -r requirements.txtThis will download and install all seven packages plus their dependencies. It may take a minute or two.
Once complete, you should see a line ending with Successfully installed followed by a list of package names.
Seeing installation errors?
Make sure (venv) is showing in your terminal prompt. Installing without an active virtual environment can cause permission errors.
If a specific package fails, try running pip install with just that package name to see the full error message.
help me fix my pip install errors
✔️ Awesome, I've got everything!
Great. Double check you have saved your requirements.txt file.
ⓧ I'd like to double check the full code
langchain==1.3.4
langchain-openai==1.2.2
chromadb==1.5.9
streamlit==1.58.0
rank-bm25==0.2.2
requests
numpyConfigure your OpenRouter API key
OpenRouter gives you free access to open-source LLMs through a single API. You will use it to power the answer generation in your RAG pipeline.
- Go to openrouter.ai and create a free account.
- Once signed in, navigate to openrouter.ai/settings/keys.
- Click Create new secret key.
- Copy the key immediately. You will not be able to see it again after leaving the page.
Why OpenRouter?
OpenRouter routes your requests to free open-source models without charging you. The free tier has rate limits but no cost, which makes it ideal for a portfolio project.
- Back in your terminal (with (venv) still active), set your API key as an environment variable:
macOS
export OPENROUTER_API_KEY=your-api-key-hereWindows
set OPENROUTER_API_KEY=your-api-key-here- Replace your-api-key-here with the key you just copied from OpenRouter.
Keep your terminal open
The environment variable only lasts for your current terminal session. If you close your terminal, you will need to set it again before running the project scripts.
Not sure if the key is set?
Run echo $OPENROUTER_API_KEY (macOS) or echo %OPENROUTER_API_KEY% (Windows) in your terminal. You should see your key printed back.
help me verify my OpenRouter API key is set correctly
Open your project in VS Code
- Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
- Type VS Code and press Enter to open it.
- Click File in the top menu bar.
- Click Open Folder.
- Select the fitrag folder on your Desktop, then click Open.
You should see requirements.txt and the venv folder in the Explorer sidebar on the left.
Your environment is ready. Next up, you will build the RAG pipeline that ingests 800+ fitness exercises into a vector database and generates answers.
Set Up the Project and Download Data
Your environment is set up with all the dependencies installed and your API key configured. Now you need data.
A RAG system is only as useful as the knowledge it retrieves from. You'll download a public dataset of 800+ fitness exercises and store it locally so your pipeline can ingest it in the next step.
In this step, get ready to:
- Create a download script that fetches the exercise dataset from GitHub.
- Run the script to download 800+ exercises into a local data/ directory.
Download the exercise dataset
The free-exercise-db is a public domain dataset hosted on GitHub containing 800+ exercises in JSON format. Each exercise includes fields like name, instructions, primary muscles, equipment, and difficulty level.
What is free-exercise-db?
This is an open-source dataset maintained on GitHub under The Unlicense (public domain). Each exercise entry includes structured fields like name, primaryMuscles, equipment, level, and step-by-step instructions.
The rich field structure makes retrieval non-trivial. Exercises with similar names can have very different mechanics, muscles, and equipment requirements.
- In VS Code, create a new file called download_data.py in the root of your project folder.
- Paste the following code into download_data.py:
import requests
import json
import os
DATA_DIR = "data"
EXERCISES_URL = "https://raw.githubusercontent.com/yuhonas/free-exercise-db/main/dist/exercises.json"
def download_exercises():
os.makedirs(DATA_DIR, exist_ok=True)
output_path = os.path.join(DATA_DIR, "exercises.json")
if os.path.exists(output_path):
print(f"Dataset already exists at {output_path}")
return
print("Downloading exercise dataset...")
response = requests.get(EXERCISES_URL)
response.raise_for_status()
exercises = response.json()
with open(output_path, "w", encoding="utf-8") as f:
json.dump(exercises, f, indent=2)
print(f"Downloaded {len(exercises)} exercises to {output_path}")
if __name__ == "__main__":
download_exercises()What does this code do?
- EXERCISES_URL points to the combined JSON file hosted on the free-exercise-db GitHub repository.
- os.makedirs(DATA_DIR, exist_ok=True) creates a data/ folder if it doesn't already exist.
- The if os.path.exists check prevents re-downloading if you've already fetched the data.
- response.raise_for_status() raises an error if the download fails, so you get a clear message instead of a silent failure.
- Save download_data.py.
- Run the download script in your terminal by running this command:
python download_data.py- You should see output similar to: Downloaded 870 exercises to data/exercises.json (the exact number may vary slightly).
- Confirm the file exists by running:
ls data/- You should see exercises.json in the output.
Seeing a connection error?
- Make sure you have an active internet connection. The script fetches data from GitHub.
- If you see a ConnectionError or timeout, wait a moment and try again.
- Check that your virtual environment is still active (you should see the environment name in your terminal prompt).
Why isn't my download script working?
✔️ Awesome, I've got everything!
Great. Double check you've saved download_data.py and that data/exercises.json exists in your project folder.
ⓧ I'd like to double check the full code
import requests
import json
import os
DATA_DIR = "data"
EXERCISES_URL = "https://raw.githubusercontent.com/yuhonas/free-exercise-db/main/dist/exercises.json"
def download_exercises():
os.makedirs(DATA_DIR, exist_ok=True)
output_path = os.path.join(DATA_DIR, "exercises.json")
if os.path.exists(output_path):
print(f"Dataset already exists at {output_path}")
return
print("Downloading exercise dataset...")
response = requests.get(EXERCISES_URL)
response.raise_for_status()
exercises = response.json()
with open(output_path, "w", encoding="utf-8") as f:
json.dump(exercises, f, indent=2)
print(f"Downloaded {len(exercises)} exercises to {output_path}")
if __name__ == "__main__":
download_exercises()You now have 800+ exercises stored locally as your knowledge base. Next up, you'll build the RAG pipeline that embeds these exercises into a vector store and generates answers from them.
Build the RAG Pipeline
In the previous step, you downloaded 800+ exercises from the free-exercise-db dataset. Now it's time to make that data searchable.
A RAG pipeline has two core stages: retrieval (finding the most relevant documents for a question) and generation (feeding those documents to an LLM to produce an answer). In this step, you'll build both.
In this step, get ready to:
- Load and transform exercise data into rich text documents.
- Embed all documents into a ChromaDB vector store for semantic search.
- Wire up the retrieval and LLM generation chain.
Load and chunk the exercise data
Raw JSON isn't useful for semantic search. You need to transform each exercise into a single text document that combines all its fields into a readable format. This gives the embedding model rich context to work with.
- Create a new file called ingest.py in your project folder.
- Add the data loading and document transformation functions by pasting the following code:
import json
import os
import chromadb
DATA_PATH = os.path.join("data", "exercises.json")
CHROMA_PATH = "chroma_db"
def load_exercises():
with open(DATA_PATH, "r", encoding="utf-8") as f:
exercises = json.load(f)
return exercises
def exercise_to_document(exercise):
name = exercise.get("name", "Unknown")
instructions = " ".join(exercise.get("instructions", []))
primary_muscles = ", ".join(exercise.get("primaryMuscles", []))
secondary_muscles = ", ".join(exercise.get("secondaryMuscles", []))
category = exercise.get("category", "")
force = exercise.get("force", "")
level = exercise.get("level", "")
equipment = exercise.get("equipment", "none")
mechanic = exercise.get("mechanic", "")
document = (
f"Exercise: {name}\n"
f"Category: {category}\n"
f"Force: {force}\n"
f"Level: {level}\n"
f"Equipment: {equipment}\n"
f"Mechanic: {mechanic}\n"
f"Primary Muscles: {primary_muscles}\n"
f"Secondary Muscles: {secondary_muscles}\n"
f"Instructions: {instructions}"
)
return documentWhat does this code do?
load_exercises() reads the JSON file you downloaded in Step 1 and returns the full list of exercise objects.
exercise_to_document() transforms a single exercise object into a multi-line string that combines its name, category, force type, level, equipment, mechanic, muscles, and instructions. This rich text format gives the embedding model much more context than just the exercise name alone.
Create the ChromaDB vector store
ChromaDB takes your text documents, converts them into vector embeddings using a local model, and stores them for fast similarity search. You don't need any API calls for this step because ChromaDB runs its embedding model entirely on your machine.
- In ingest.py, add the vector store creation function and main block below the exercise_to_document function:
def create_vector_store():
exercises = load_exercises()
documents = []
metadatas = []
ids = []
for i, exercise in enumerate(exercises):
doc = exercise_to_document(exercise)
documents.append(doc)
metadatas.append({
"name": exercise.get("name", "Unknown"),
"category": exercise.get("category", ""),
"equipment": exercise.get("equipment", "none"),
"level": exercise.get("level", ""),
"primary_muscles": ", ".join(exercise.get("primaryMuscles", []))
})
ids.append(f"exercise_{i}")
client = chromadb.PersistentClient(path=CHROMA_PATH)
collection = client.get_or_create_collection(
name="fitness_exercises",
metadata={"hnsw:space": "cosine"}
)
if collection.count() > 0:
print(f"Collection already has {collection.count()} documents. Skipping ingestion.")
return
batch_size = 100
for start in range(0, len(documents), batch_size):
end = min(start + batch_size, len(documents))
collection.add(
documents=documents[start:end],
metadatas=metadatas[start:end],
ids=ids[start:end]
)
print(f"Ingested {end}/{len(documents)} exercises")
print(f"Vector store created with {collection.count()} documents at {CHROMA_PATH}")
if __name__ == "__main__":
create_vector_store()What does this code do?
create_vector_store() loops through every exercise, converts it to a text document, and stores both the document and its metadata (name, category, equipment, level, muscles) alongside a unique ID.
PersistentClient saves the database to disk in a chroma_db/ folder. The hnsw:space: cosine setting tells ChromaDB to use cosine similarity for comparing vectors.
Documents are ingested in batches of 100 to avoid memory issues. The collection.count() check prevents duplicate ingestion if you run the script again.
- Save ingest.py.
- Run the ingestion script by running this command:
python ingest.py- You should see progress messages like Ingested 100/870 exercises counting up, followed by a confirmation that the vector store was created.
Why is the first batch slower?
The first time ChromaDB embeds documents, it downloads and initializes the all-MiniLM-L6-v2 model locally via ONNX Runtime. This one-time setup can take a few extra seconds. Subsequent runs skip the download because the model is cached on your machine.
Not seeing any output?
Make sure data/exercises.json exists from Step 1. If it's missing, re-run python download_data.py first.
If you see an import error, confirm your virtual environment is activated and all packages from requirements.txt are installed.
help me debug my ingest.py script
✔️ Awesome, I've got everything!
Great. Double check you've saved ingest.py and that a chroma_db/ folder has appeared in your project directory.
ⓧ I'd like to double check the full code
import json
import os
import chromadb
DATA_PATH = os.path.join("data", "exercises.json")
CHROMA_PATH = "chroma_db"
def load_exercises():
with open(DATA_PATH, "r", encoding="utf-8") as f:
exercises = json.load(f)
return exercises
def exercise_to_document(exercise):
name = exercise.get("name", "Unknown")
instructions = " ".join(exercise.get("instructions", []))
primary_muscles = ", ".join(exercise.get("primaryMuscles", []))
secondary_muscles = ", ".join(exercise.get("secondaryMuscles", []))
category = exercise.get("category", "")
force = exercise.get("force", "")
level = exercise.get("level", "")
equipment = exercise.get("equipment", "none")
mechanic = exercise.get("mechanic", "")
document = (
f"Exercise: {name}\n"
f"Category: {category}\n"
f"Force: {force}\n"
f"Level: {level}\n"
f"Equipment: {equipment}\n"
f"Mechanic: {mechanic}\n"
f"Primary Muscles: {primary_muscles}\n"
f"Secondary Muscles: {secondary_muscles}\n"
f"Instructions: {instructions}"
)
return document
def create_vector_store():
exercises = load_exercises()
documents = []
metadatas = []
ids = []
for i, exercise in enumerate(exercises):
doc = exercise_to_document(exercise)
documents.append(doc)
metadatas.append({
"name": exercise.get("name", "Unknown"),
"category": exercise.get("category", ""),
"equipment": exercise.get("equipment", "none"),
"level": exercise.get("level", ""),
"primary_muscles": ", ".join(exercise.get("primaryMuscles", []))
})
ids.append(f"exercise_{i}")
client = chromadb.PersistentClient(path=CHROMA_PATH)
collection = client.get_or_create_collection(
name="fitness_exercises",
metadata={"hnsw:space": "cosine"}
)
if collection.count() > 0:
print(f"Collection already has {collection.count()} documents. Skipping ingestion.")
return
batch_size = 100
for start in range(0, len(documents), batch_size):
end = min(start + batch_size, len(documents))
collection.add(
documents=documents[start:end],
metadatas=metadatas[start:end],
ids=ids[start:end]
)
print(f"Ingested {end}/{len(documents)} exercises")
print(f"Vector store created with {collection.count()} documents at {CHROMA_PATH}")
if __name__ == "__main__":
create_vector_store()Wire up the LLM generation
You now have a searchable vector store. The next piece is the retrieval interface and the LLM chain that takes retrieved documents and generates a natural language answer.
- Create a new file called retrievers.py in your project folder.
- Add the VectorRetriever class by pasting the following code:
import json
import os
import chromadb
from rank_bm25 import BM25Okapi
from ingest import load_exercises, exercise_to_document
CHROMA_PATH = "chroma_db"
class VectorRetriever:
def __init__(self, top_k=5):
self.top_k = top_k
client = chromadb.PersistentClient(path=CHROMA_PATH)
self.collection = client.get_collection("fitness_exercises")
def retrieve(self, query):
results = self.collection.query(
query_texts=[query],
n_results=self.top_k
)
documents = results["documents"][0]
metadatas = results["metadatas"][0]
ids = results["ids"][0]
return [
{"id": doc_id, "document": doc, "metadata": meta}
for doc_id, doc, meta in zip(ids, documents, metadatas)
]What does this code do?
VectorRetriever connects to the ChromaDB collection you just created. When you call retrieve() with a question, it embeds the question using the same model and finds the 5 most similar exercise documents using cosine similarity.
The imports at the top include BM25Okapi and functions from ingest.py because the BM25 retriever (added next) needs them.
- Add the BM25Retriever class below the VectorRetriever class in retrievers.py:
class BM25Retriever:
def __init__(self, top_k=5):
self.top_k = top_k
exercises = load_exercises()
self.documents = [exercise_to_document(ex) for ex in exercises]
self.ids = [f"exercise_{i}" for i in range(len(exercises))]
self.metadatas = [
{
"name": ex.get("name", "Unknown"),
"category": ex.get("category", ""),
"equipment": ex.get("equipment", "none"),
"level": ex.get("level", ""),
"primary_muscles": ", ".join(ex.get("primaryMuscles", []))
}
for ex in exercises
]
tokenized_corpus = [doc.lower().split() for doc in self.documents]
self.bm25 = BM25Okapi(tokenized_corpus)
def retrieve(self, query):
tokenized_query = query.lower().split()
scores = self.bm25.get_scores(tokenized_query)
top_indices = scores.argsort()[-self.top_k:][::-1]
return [
{
"id": self.ids[i],
"document": self.documents[i],
"metadata": self.metadatas[i]
}
for i in top_indices
]What is BM25?
BM25 is a keyword-matching retrieval algorithm. It scores documents based on how often the query terms appear, adjusted for document length. It's the gold standard baseline for information retrieval because it's fast, interpretable, and surprisingly effective.
Having both a vector retriever and a BM25 retriever lets you compare semantic search against keyword search in your evaluation later.
- Save retrievers.py.
Why build both retrievers now?
The RAG chain you're about to create supports switching between retriever types at query time. Building both retrievers here means the chain is immediately testable with either approach. You'll compare their performance head-to-head in the evaluation step.
✔️ Awesome, I've got everything!
Great. Double check you've saved retrievers.py.
ⓧ I'd like to double check the full code
import json
import os
import chromadb
from rank_bm25 import BM25Okapi
from ingest import load_exercises, exercise_to_document
CHROMA_PATH = "chroma_db"
class VectorRetriever:
def __init__(self, top_k=5):
self.top_k = top_k
client = chromadb.PersistentClient(path=CHROMA_PATH)
self.collection = client.get_collection("fitness_exercises")
def retrieve(self, query):
results = self.collection.query(
query_texts=[query],
n_results=self.top_k
)
documents = results["documents"][0]
metadatas = results["metadatas"][0]
ids = results["ids"][0]
return [
{"id": doc_id, "document": doc, "metadata": meta}
for doc_id, doc, meta in zip(ids, documents, metadatas)
]
class BM25Retriever:
def __init__(self, top_k=5):
self.top_k = top_k
exercises = load_exercises()
self.documents = [exercise_to_document(ex) for ex in exercises]
self.ids = [f"exercise_{i}" for i in range(len(exercises))]
self.metadatas = [
{
"name": ex.get("name", "Unknown"),
"category": ex.get("category", ""),
"equipment": ex.get("equipment", "none"),
"level": ex.get("level", ""),
"primary_muscles": ", ".join(ex.get("primaryMuscles", []))
}
for ex in exercises
]
tokenized_corpus = [doc.lower().split() for doc in self.documents]
self.bm25 = BM25Okapi(tokenized_corpus)
def retrieve(self, query):
tokenized_query = query.lower().split()
scores = self.bm25.get_scores(tokenized_query)
top_indices = scores.argsort()[-self.top_k:][::-1]
return [
{
"id": self.ids[i],
"document": self.documents[i],
"metadata": self.metadatas[i]
}
for i in top_indices
]Now you'll create the generation chain that ties retrieval to LLM output.
- Create a new file called rag_chain.py in your project folder.
- Add the LLM configuration and prompt template by pasting the following code:
import os
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from retrievers import VectorRetriever, BM25Retriever
OPENROUTER_API_KEY = os.environ.get("OPENROUTER_API_KEY", "your-api-key-here")
llm = ChatOpenAI(
model="meta-llama/llama-3.2-3b-instruct:free",
openai_api_key=OPENROUTER_API_KEY,
openai_api_base="https://openrouter.ai/api/v1",
temperature=0.3
)
RAG_PROMPT = ChatPromptTemplate.from_messages([
("system", """You are a knowledgeable fitness assistant. Answer the user's question based ONLY on the provided exercise context.
If the context doesn't contain enough information to answer, say so clearly.
Always reference specific exercises from the context in your answer."""),
("human", """Context:
{context}
Question: {question}
Answer based on the exercises above:""")
])
rag_chain = RAG_PROMPT | llm | StrOutputParser()What does this code do?
ChatOpenAI from LangChain connects to OpenRouter's API using the OpenAI-compatible interface. The openai_api_base parameter redirects requests to OpenRouter instead of OpenAI directly.
The meta-llama/llama-3.2-3b-instruct:free model is completely free to use on OpenRouter. The :free suffix ensures no charges are incurred.
The prompt template instructs the LLM to answer only from the provided context and always reference specific exercises. The rag_chain pipes the prompt into the LLM and parses the output as a plain string.
- Add the query function and test block below the rag_chain line in rag_chain.py:
def query_rag(question, retriever_type="vector"):
if retriever_type == "vector":
retriever = VectorRetriever(top_k=5)
else:
retriever = BM25Retriever(top_k=5)
results = retriever.retrieve(question)
context = "\n\n---\n\n".join([r["document"] for r in results])
answer = rag_chain.invoke({"context": context, "question": question})
return {
"answer": answer,
"sources": [r["metadata"] for r in results],
"source_ids": [r["id"] for r in results]
}
if __name__ == "__main__":
result = query_rag("What are the best bodyweight exercises for chest?")
print("Answer:", result["answer"])
print("\nSources:")
for source in result["sources"]:
print(f" - {source['name']} ({source['primary_muscles']})")What does this code do?
query_rag() is the main function your app will call. It accepts a question and a retriever type, retrieves the top 5 matching documents, joins them into a context string, and passes everything to the LLM chain.
The response includes both the generated answer and the source metadata, so users can verify which exercises informed the answer.
- Save rag_chain.py.
- Test your RAG pipeline by running this command:
python rag_chain.py- You should see a generated answer about bodyweight chest exercises, followed by a list of source exercise names with their target muscles.
Seeing an authentication or API error?
Make sure your OPENROUTER_API_KEY environment variable is set in your current terminal session. If you closed the terminal since Step 1, you'll need to export it again.
If you see a rate limit error, wait 30 seconds and try again. The free tier has rate limits but no charges.
help me fix my RAG chain API connection
✔️ Awesome, I've got everything!
Great. Double check you've saved rag_chain.py and confirmed the output looks reasonable.
ⓧ I'd like to double check the full code
import os
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from retrievers import VectorRetriever, BM25Retriever
OPENROUTER_API_KEY = os.environ.get("OPENROUTER_API_KEY", "your-api-key-here")
llm = ChatOpenAI(
model="meta-llama/llama-3.2-3b-instruct:free",
openai_api_key=OPENROUTER_API_KEY,
openai_api_base="https://openrouter.ai/api/v1",
temperature=0.3
)
RAG_PROMPT = ChatPromptTemplate.from_messages([
("system", """You are a knowledgeable fitness assistant. Answer the user's question based ONLY on the provided exercise context.
If the context doesn't contain enough information to answer, say so clearly.
Always reference specific exercises from the context in your answer."""),
("human", """Context:
{context}
Question: {question}
Answer based on the exercises above:""")
])
rag_chain = RAG_PROMPT | llm | StrOutputParser()
def query_rag(question, retriever_type="vector"):
if retriever_type == "vector":
retriever = VectorRetriever(top_k=5)
else:
retriever = BM25Retriever(top_k=5)
results = retriever.retrieve(question)
context = "\n\n---\n\n".join([r["document"] for r in results])
answer = rag_chain.invoke({"context": context, "question": question})
return {
"answer": answer,
"sources": [r["metadata"] for r in results],
"source_ids": [r["id"] for r in results]
}
if __name__ == "__main__":
result = query_rag("What are the best bodyweight exercises for chest?")
print("Answer:", result["answer"])
print("\nSources:")
for source in result["sources"]:
print(f" - {source['name']} ({source['primary_muscles']})")Your RAG pipeline is working end-to-end. You can ask it questions about exercises and get grounded answers backed by source documents. Next up, you'll build a proper evaluation framework to measure how well each retriever actually performs.
Build the BM25 Baseline
You have a working vector search retriever that finds exercises by semantic meaning. But how do you know if it's actually good?
To measure retrieval quality, you need a baseline to compare against. BM25 is the gold standard baseline for information retrieval. In this step, you'll walk through how the BM25 retriever works and verify that both retrievers share a unified interface ready for evaluation.
In this step, get ready to:
- Understand how BM25 keyword retrieval works.
- Verify the unified retriever interface.
- Test both retrievers with a sample query.
Understand how BM25 retrieves exercises
Vector search finds exercises by meaning. If you search for "leg workouts," it understands that squats and lunges are relevant even if the word "leg" never appears in the document. BM25 takes a completely different approach.
What is BM25?
BM25 (Best Matching 25) is a keyword-based ranking algorithm. It scores documents based on how often the query terms appear, adjusted for document length and term rarity.
A term that appears rarely across the corpus (like "kettlebell") gets a higher weight than a common term (like "exercise"). This makes BM25 excellent at finding documents with exact keyword matches.
- In VS Code, switch back to retrievers.py and find the BM25Retriever class.
Here's how the initialization works:
class BM25Retriever:
def __init__(self, top_k=5):
self.top_k = top_k
exercises = load_exercises()
self.documents = [exercise_to_document(ex) for ex in exercises]
self.ids = [f"exercise_{i}" for i in range(len(exercises))]
self.metadatas = [
{
"name": ex.get("name", "Unknown"),
"category": ex.get("category", ""),
"equipment": ex.get("equipment", "none"),
"level": ex.get("level", ""),
"primary_muscles": ", ".join(ex.get("primaryMuscles", []))
}
for ex in exercises
]
tokenized_corpus = [doc.lower().split() for doc in self.documents]
self.bm25 = BM25Okapi(tokenized_corpus)What does this code do?
- load_exercises() reads the same 800+ exercises JSON that the vector store used.
- exercise_to_document(ex) transforms each exercise into the same rich text format (name, muscles, equipment, instructions).
- tokenized_corpus splits every document into lowercase words. BM25 needs individual tokens to count term frequencies.
- BM25Okapi(tokenized_corpus) builds the BM25 index over all tokenized documents. This is where term frequency and inverse document frequency scores are precomputed.
Now look at the retrieve method:
def retrieve(self, query):
tokenized_query = query.lower().split()
scores = self.bm25.get_scores(tokenized_query)
top_indices = scores.argsort()[-self.top_k:][::-1]
return [
{
"id": self.ids[i],
"document": self.documents[i],
"metadata": self.metadatas[i]
}
for i in top_indices
]How does BM25 scoring work?
- The query is tokenized the same way as the corpus (lowercase, split on spaces).
- get_scores() returns a score for every document in the corpus based on keyword overlap.
- argsort()[-self.top_k:][::-1] grabs the indices of the top-5 highest-scoring documents.
- The return format is a list of dicts with id, document, and metadata keys.
Why BM25 as a baseline?
Hiring managers reviewing RAG portfolios look for measured comparisons. BM25 is the standard baseline because it's simple, well-understood, and surprisingly strong at exact-keyword queries.
If your vector search can't beat BM25, it's a signal that semantic embeddings aren't adding value for your use case. Showing this comparison is exactly what differentiates a portfolio project from a tutorial copy-paste.
Verify the unified retriever interface
Both VectorRetriever and BM25Retriever share the exact same interface. Each class exposes a retrieve(query) method that returns a list of dictionaries with three keys: id, document, and metadata.
Why does a unified interface matter?
When both retrievers return results in the same shape, you can swap them into an evaluation harness without changing any evaluation code. The eval script just calls retriever.retrieve(query) and doesn't care which approach generated the results.
- Test the BM25 retriever by running this command in your terminal:
python -c "from retrievers import BM25Retriever; r = BM25Retriever(); results = r.retrieve('bodyweight chest exercises'); print([doc['metadata']['name'] for doc in results])"You should see a list of 5 exercise names related to bodyweight chest work.
- Now test the vector retriever with the same query by running:
python -c "from retrievers import VectorRetriever; r = VectorRetriever(); results = r.retrieve('bodyweight chest exercises'); print([doc['metadata']['name'] for doc in results])"You should see a different set of 5 exercise names. The two retrievers find different exercises because they search differently.
Why are the results different?
BM25 finds exercises that literally contain the words "bodyweight," "chest," or "exercises" in their text. Vector search finds exercises whose meaning is close to the query, even if they use different words (like "push-up" instead of "chest").
Neither approach is universally better. That's exactly why you'll build an evaluation framework in the next step to measure which one retrieves more relevant results for different query types.
Getting an import error?
- Make sure your virtual environment is active. The rank-bm25 package must be installed in the active environment.
- Check that you're running the command from your project folder (the one containing retrievers.py).
- If you see a ChromaDB collection error, confirm that chroma_db/ exists by running ls in your project directory.
Help me debug my retriever import error.
✔️ Awesome, I've got everything!
Great. Both retrievers return results in the same shape, ready for evaluation.
ⓧ I'd like to double check the full code
Here's the complete retrievers.py file:
import json
import os
import chromadb
from rank_bm25 import BM25Okapi
from ingest import load_exercises, exercise_to_document
CHROMA_PATH = "chroma_db"
class VectorRetriever:
def __init__(self, top_k=5):
self.top_k = top_k
client = chromadb.PersistentClient(path=CHROMA_PATH)
self.collection = client.get_collection("fitness_exercises")
def retrieve(self, query):
results = self.collection.query(
query_texts=[query],
n_results=self.top_k
)
documents = results["documents"][0]
metadatas = results["metadatas"][0]
ids = results["ids"][0]
return [
{"id": doc_id, "document": doc, "metadata": meta}
for doc_id, doc, meta in zip(ids, documents, metadatas)
]
class BM25Retriever:
def __init__(self, top_k=5):
self.top_k = top_k
exercises = load_exercises()
self.documents = [exercise_to_document(ex) for ex in exercises]
self.ids = [f"exercise_{i}" for i in range(len(exercises))]
self.metadatas = [
{
"name": ex.get("name", "Unknown"),
"category": ex.get("category", ""),
"equipment": ex.get("equipment", "none"),
"level": ex.get("level", ""),
"primary_muscles": ", ".join(ex.get("primaryMuscles", []))
}
for ex in exercises
]
tokenized_corpus = [doc.lower().split() for doc in self.documents]
self.bm25 = BM25Okapi(tokenized_corpus)
def retrieve(self, query):
tokenized_query = query.lower().split()
scores = self.bm25.get_scores(tokenized_query)
top_indices = scores.argsort()[-self.top_k:][::-1]
return [
{
"id": self.ids[i],
"document": self.documents[i],
"metadata": self.metadatas[i]
}
for i in top_indices
]Your baseline is locked in. Next up, you'll build the evaluation framework that measures exactly how well each retriever performs across 20 different query types.
Create the Evaluation Framework
You have a working RAG pipeline and a BM25 baseline. The obvious question is: which one actually retrieves better exercises for a given query? Without measurement, you are guessing.
In this step, you will build an evaluation framework that answers that question with data. You will write a 20-question eval set, implement a keyword precision metric, and run both retrievers through the same harness to produce a side-by-side comparison.
In this step, get ready to:
- Write a 20-question evaluation set with ground-truth keywords.
- Build an evaluation script with keyword precision@5 metrics.
- Run the evaluation and compare Vector vs. BM25 performance.
Write the eval set
An evaluation set is a collection of questions paired with expected answers. Each question has keywords that the retrieved documents should contain if the retriever is working correctly.
- Create a new file called eval_set.json in your project root (the same folder as retrievers.py).
- Paste the following 20 evaluation questions into the file:
[
{
"question": "What exercises target the glutes without any equipment?",
"expected_exercise_ids": ["exercise_0", "exercise_1", "exercise_2"],
"expected_keywords": ["glutes", "bodyweight"]
},
{
"question": "What are good beginner chest exercises with dumbbells?",
"expected_exercise_ids": ["exercise_3", "exercise_4", "exercise_5"],
"expected_keywords": ["chest", "dumbbell", "beginner"]
},
{
"question": "How do I do a barbell squat properly?",
"expected_exercise_ids": ["exercise_6"],
"expected_keywords": ["squat", "barbell", "form"]
},
{
"question": "What exercises strengthen the lower back?",
"expected_exercise_ids": ["exercise_7", "exercise_8", "exercise_9"],
"expected_keywords": ["lower back", "erector"]
},
{
"question": "What cable exercises work the shoulders?",
"expected_exercise_ids": ["exercise_10", "exercise_11"],
"expected_keywords": ["cable", "shoulders", "deltoid"]
},
{
"question": "What compound exercises work multiple muscle groups?",
"expected_exercise_ids": ["exercise_12", "exercise_13", "exercise_14"],
"expected_keywords": ["compound", "multiple"]
},
{
"question": "What isolation exercises target the biceps?",
"expected_exercise_ids": ["exercise_15", "exercise_16"],
"expected_keywords": ["biceps", "isolation", "curl"]
},
{
"question": "What exercises can I do for abs at home?",
"expected_exercise_ids": ["exercise_17", "exercise_18", "exercise_19"],
"expected_keywords": ["abs", "abdominals", "bodyweight"]
},
{
"question": "What stretching exercises help with flexibility?",
"expected_exercise_ids": ["exercise_20", "exercise_21"],
"expected_keywords": ["stretch", "flexibility"]
},
{
"question": "What advanced exercises target the hamstrings?",
"expected_exercise_ids": ["exercise_22", "exercise_23"],
"expected_keywords": ["hamstrings", "advanced"]
},
{
"question": "What exercises use a barbell for back training?",
"expected_exercise_ids": ["exercise_24", "exercise_25"],
"expected_keywords": ["barbell", "back"]
},
{
"question": "What push exercises are good for triceps?",
"expected_exercise_ids": ["exercise_26", "exercise_27"],
"expected_keywords": ["triceps", "push"]
},
{
"question": "What exercises target the quadriceps with machines?",
"expected_exercise_ids": ["exercise_28", "exercise_29"],
"expected_keywords": ["quadriceps", "machine"]
},
{
"question": "What bodyweight exercises target the upper body?",
"expected_exercise_ids": ["exercise_30", "exercise_31"],
"expected_keywords": ["bodyweight", "upper body"]
},
{
"question": "What exercises help with grip strength?",
"expected_exercise_ids": ["exercise_32", "exercise_33"],
"expected_keywords": ["grip", "forearms"]
},
{
"question": "What plyometric exercises improve explosiveness?",
"expected_exercise_ids": ["exercise_34", "exercise_35"],
"expected_keywords": ["plyometric", "explosive"]
},
{
"question": "What exercises target the calves?",
"expected_exercise_ids": ["exercise_36", "exercise_37"],
"expected_keywords": ["calves", "calf"]
},
{
"question": "What exercises are good for shoulder rehabilitation?",
"expected_exercise_ids": ["exercise_38", "exercise_39"],
"expected_keywords": ["shoulder", "rehabilitation", "rotator"]
},
{
"question": "What exercises use resistance bands?",
"expected_exercise_ids": ["exercise_40", "exercise_41"],
"expected_keywords": ["band", "resistance"]
},
{
"question": "What kettlebell exercises work the full body?",
"expected_exercise_ids": ["exercise_42", "exercise_43"],
"expected_keywords": ["kettlebell", "full body"]
}
]What does this eval set do?
Each question targets a different retrieval challenge. Some test muscle targeting ("glutes", "biceps"), some test equipment constraints ("cable", "barbell"), and some test difficulty levels ("beginner", "advanced").
The expected_keywords field lists terms that should appear in correctly retrieved documents. If you ask about glute exercises without equipment, the retrieved docs should contain "glutes" and "bodyweight" somewhere in their text.
- Save eval_set.json.
- Confirm the file exists by running:
ls eval_set.jsonYou should see eval_set.json listed in the output.
File not found?
Make sure you saved the file in your project root directory (the same folder where retrievers.py lives), not inside the data/ subdirectory.
Also check that the filename is exactly eval_set.json with an underscore, not a hyphen.
Why can't my evaluation script find eval_set.json?
✔️ Awesome, I've got everything!
Great. Make sure you have saved your file before continuing.
ⓧ I'd like to double check the full code
[
{
"question": "What exercises target the glutes without any equipment?",
"expected_exercise_ids": ["exercise_0", "exercise_1", "exercise_2"],
"expected_keywords": ["glutes", "bodyweight"]
},
{
"question": "What are good beginner chest exercises with dumbbells?",
"expected_exercise_ids": ["exercise_3", "exercise_4", "exercise_5"],
"expected_keywords": ["chest", "dumbbell", "beginner"]
},
{
"question": "How do I do a barbell squat properly?",
"expected_exercise_ids": ["exercise_6"],
"expected_keywords": ["squat", "barbell", "form"]
},
{
"question": "What exercises strengthen the lower back?",
"expected_exercise_ids": ["exercise_7", "exercise_8", "exercise_9"],
"expected_keywords": ["lower back", "erector"]
},
{
"question": "What cable exercises work the shoulders?",
"expected_exercise_ids": ["exercise_10", "exercise_11"],
"expected_keywords": ["cable", "shoulders", "deltoid"]
},
{
"question": "What compound exercises work multiple muscle groups?",
"expected_exercise_ids": ["exercise_12", "exercise_13", "exercise_14"],
"expected_keywords": ["compound", "multiple"]
},
{
"question": "What isolation exercises target the biceps?",
"expected_exercise_ids": ["exercise_15", "exercise_16"],
"expected_keywords": ["biceps", "isolation", "curl"]
},
{
"question": "What exercises can I do for abs at home?",
"expected_exercise_ids": ["exercise_17", "exercise_18", "exercise_19"],
"expected_keywords": ["abs", "abdominals", "bodyweight"]
},
{
"question": "What stretching exercises help with flexibility?",
"expected_exercise_ids": ["exercise_20", "exercise_21"],
"expected_keywords": ["stretch", "flexibility"]
},
{
"question": "What advanced exercises target the hamstrings?",
"expected_exercise_ids": ["exercise_22", "exercise_23"],
"expected_keywords": ["hamstrings", "advanced"]
},
{
"question": "What exercises use a barbell for back training?",
"expected_exercise_ids": ["exercise_24", "exercise_25"],
"expected_keywords": ["barbell", "back"]
},
{
"question": "What push exercises are good for triceps?",
"expected_exercise_ids": ["exercise_26", "exercise_27"],
"expected_keywords": ["triceps", "push"]
},
{
"question": "What exercises target the quadriceps with machines?",
"expected_exercise_ids": ["exercise_28", "exercise_29"],
"expected_keywords": ["quadriceps", "machine"]
},
{
"question": "What bodyweight exercises target the upper body?",
"expected_exercise_ids": ["exercise_30", "exercise_31"],
"expected_keywords": ["bodyweight", "upper body"]
},
{
"question": "What exercises help with grip strength?",
"expected_exercise_ids": ["exercise_32", "exercise_33"],
"expected_keywords": ["grip", "forearms"]
},
{
"question": "What plyometric exercises improve explosiveness?",
"expected_exercise_ids": ["exercise_34", "exercise_35"],
"expected_keywords": ["plyometric", "explosive"]
},
{
"question": "What exercises target the calves?",
"expected_exercise_ids": ["exercise_36", "exercise_37"],
"expected_keywords": ["calves", "calf"]
},
{
"question": "What exercises are good for shoulder rehabilitation?",
"expected_exercise_ids": ["exercise_38", "exercise_39"],
"expected_keywords": ["shoulder", "rehabilitation", "rotator"]
},
{
"question": "What exercises use resistance bands?",
"expected_exercise_ids": ["exercise_40", "exercise_41"],
"expected_keywords": ["band", "resistance"]
},
{
"question": "What kettlebell exercises work the full body?",
"expected_exercise_ids": ["exercise_42", "exercise_43"],
"expected_keywords": ["kettlebell", "full body"]
}
]Build the eval harness script
The evaluation harness runs each question through a retriever, checks if the expected keywords appear in the retrieved documents, and reports a precision score. This gives you a single number to compare retrievers against.
- Create a new file called evaluate.py in your project root.
- Add the imports, constants, and metric functions by pasting the following code:
import json
import os
from retrievers import VectorRetriever, BM25Retriever
EVAL_SET_PATH = "eval_set.json"
def load_eval_set():
# Load the 20-question evaluation set from disk
with open(EVAL_SET_PATH, "r", encoding="utf-8") as f:
return json.load(f)
def keyword_precision_at_k(retrieved_docs, expected_keywords, k=5):
# Combine all retrieved document text into one string for keyword matching
retrieved_text = " ".join([doc["document"].lower() for doc in retrieved_docs[:k]])
# Count how many expected keywords appear in the retrieved text
hits = sum(1 for kw in expected_keywords if kw.lower() in retrieved_text)
return hits / len(expected_keywords) if expected_keywords else 0What does keyword_precision_at_k measure?
This metric checks what fraction of expected keywords appear in the top-k retrieved documents. If you ask for glute exercises without equipment and the expected keywords are ["glutes", "bodyweight"], a score of 1.0 means both keywords appeared in the retrieved text.
This is a practical proxy for retrieval quality. If the retrieved documents contain the right keywords, the retriever is pulling relevant exercises.
- Now add the evaluation loop and the main runner function below the code you just pasted:
def evaluate_retriever(retriever, eval_set):
# Run each eval question through the retriever and score it
total_keyword_precision = 0
results = []
for item in eval_set:
question = item["question"]
expected_keywords = item["expected_keywords"]
retrieved = retriever.retrieve(question)
kw_precision = keyword_precision_at_k(retrieved, expected_keywords, k=5)
total_keyword_precision += kw_precision
results.append({
"question": question,
"keyword_precision": kw_precision,
"retrieved_names": [r["metadata"]["name"] for r in retrieved]
})
avg_precision = total_keyword_precision / len(eval_set)
return {
"avg_keyword_precision": avg_precision,
"details": results
}
def run_evaluation():
eval_set = load_eval_set()
print("=" * 60)
print("FITNESS RAG EVALUATION RESULTS")
print("=" * 60)
# Evaluate both retrievers on the same eval set
print("\nEvaluating Vector Retriever (ChromaDB)...")
vector_retriever = VectorRetriever(top_k=5)
vector_results = evaluate_retriever(vector_retriever, eval_set)
print("Evaluating BM25 Retriever...")
bm25_retriever = BM25Retriever(top_k=5)
bm25_results = evaluate_retriever(bm25_retriever, eval_set)
# Print comparison table
print("\n" + "-" * 60)
print(f"{'Metric':<30} {'Vector':<15} {'BM25':<15}")
print("-" * 60)
print(f"{'Keyword Precision@5':<30} {vector_results['avg_keyword_precision']:<15.3f} {bm25_results['avg_keyword_precision']:<15.3f}")
print("-" * 60)
# Save detailed results to file
output = {
"vector": vector_results,
"bm25": bm25_results
}
with open("eval_results.json", "w", encoding="utf-8") as f:
json.dump(output, f, indent=2)
print("\nDetailed results saved to eval_results.json")
if __name__ == "__main__":
run_evaluation()What does this code do?
- evaluate_retriever() loops through all 20 eval questions, retrieves the top 5 documents for each, and computes keyword precision. It returns both the average score and per-question details.
- run_evaluation() orchestrates the full eval. It runs both the Vector and BM25 retrievers, prints a formatted comparison table, and saves the detailed results to eval_results.json.
- Save evaluate.py.
Seeing a syntax error?
Make sure the second code block is pasted directly below the first block, with no extra blank lines between them. The evaluate_retriever function should be at the top level (no indentation), immediately after keyword_precision_at_k.
Why is my evaluate.py throwing a syntax error?
✔️ Awesome, I've got everything!
Great. Double check you have saved your file before continuing.
ⓧ I'd like to double check the full code
import json
import os
from retrievers import VectorRetriever, BM25Retriever
EVAL_SET_PATH = "eval_set.json"
def load_eval_set():
with open(EVAL_SET_PATH, "r", encoding="utf-8") as f:
return json.load(f)
def keyword_precision_at_k(retrieved_docs, expected_keywords, k=5):
retrieved_text = " ".join([doc["document"].lower() for doc in retrieved_docs[:k]])
hits = sum(1 for kw in expected_keywords if kw.lower() in retrieved_text)
return hits / len(expected_keywords) if expected_keywords else 0
def evaluate_retriever(retriever, eval_set):
total_keyword_precision = 0
results = []
for item in eval_set:
question = item["question"]
expected_keywords = item["expected_keywords"]
retrieved = retriever.retrieve(question)
kw_precision = keyword_precision_at_k(retrieved, expected_keywords, k=5)
total_keyword_precision += kw_precision
results.append({
"question": question,
"keyword_precision": kw_precision,
"retrieved_names": [r["metadata"]["name"] for r in retrieved]
})
avg_precision = total_keyword_precision / len(eval_set)
return {
"avg_keyword_precision": avg_precision,
"details": results
}
def run_evaluation():
eval_set = load_eval_set()
print("=" * 60)
print("FITNESS RAG EVALUATION RESULTS")
print("=" * 60)
print("\nEvaluating Vector Retriever (ChromaDB)...")
vector_retriever = VectorRetriever(top_k=5)
vector_results = evaluate_retriever(vector_retriever, eval_set)
print("Evaluating BM25 Retriever...")
bm25_retriever = BM25Retriever(top_k=5)
bm25_results = evaluate_retriever(bm25_retriever, eval_set)
print("\n" + "-" * 60)
print(f"{'Metric':<30} {'Vector':<15} {'BM25':<15}")
print("-" * 60)
print(f"{'Keyword Precision@5':<30} {vector_results['avg_keyword_precision']:<15.3f} {bm25_results['avg_keyword_precision']:<15.3f}")
print("-" * 60)
output = {
"vector": vector_results,
"bm25": bm25_results
}
with open("eval_results.json", "w", encoding="utf-8") as f:
json.dump(output, f, indent=2)
print("\nDetailed results saved to eval_results.json")
if __name__ == "__main__":
run_evaluation()Run evaluations and compare
Now for the moment of truth. You will run both retrievers through the same 20 questions and see which one finds more relevant exercises.
- Run the evaluation script by executing this command in your terminal:
python evaluate.pyYou should see a formatted comparison table printed to your terminal showing the Keyword Precision@5 score for both the Vector and BM25 retrievers.
What should I see?
The output starts with a header line ("FITNESS RAG EVALUATION RESULTS"), followed by progress messages as each retriever runs through 20 questions. At the end, you see a table comparing Keyword Precision@5 for both retrievers.
Your exact numbers will depend on the dataset, but both scores should be between 0 and 1. A score of 0.8 means 80% of expected keywords appeared in the top 5 retrieved documents on average.
- Confirm the results file was saved by running:
ls eval_results.jsonYou should see eval_results.json in the output. This file contains the detailed per-question results for both retrievers.
Getting a ModuleNotFoundError?
Make sure your virtual environment is still active (you should see the environment name in your terminal prompt). If not, reactivate it before running the script.
If the error mentions retrievers or ingest, check that retrievers.py and ingest.py are in the same directory as evaluate.py.
Why is my evaluate.py throwing an import error?
Why is this evaluation important?
Hiring managers screening AI engineer portfolios look for one thing above all: measured baselines. Anyone can call an LLM API. The difference between a junior and a senior approach is having a reproducible eval harness that proves your system works.
Your comparison table is concrete evidence that you understand evaluation engineering, not just prompt engineering.
You now have hard numbers comparing your two retrievers. Next up, you will build a Streamlit UI so anyone can query your RAG system and see the results in real time.
Build the Streamlit UI
Your evaluation framework proved that your RAG pipeline retrieves relevant exercises. Now you need a way for someone to actually interact with the system.
In this step, you'll build a Streamlit web app that lets users type fitness questions, choose a retriever (Vector or BM25), and see both the generated answer and the source exercises used to produce it.
In this step, get ready to:
- Create the Streamlit app layout with a retriever selector and question input.
- Add source transparency with expandable exercise cards beneath the answer.
- Launch the app locally and test it with sample fitness queries.
Create the app layout
Streamlit turns Python scripts into interactive web apps with zero front-end code. Each Streamlit call renders a UI widget in the browser automatically.
- Create a new file called app.py in your project folder.
- Add the imports and page layout by pasting this code into app.py:
import streamlit as st
from rag_chain import query_rag
# Configure the page title and layout
st.set_page_config(page_title="FitRAG - Fitness Exercise Assistant", layout="wide")
# App header and description
st.title("FitRAG - Fitness Exercise Assistant")
st.markdown("Ask questions about exercises, muscle targeting, equipment, and workout programming.")
# Radio buttons to switch between retrieval methods
retriever_type = st.radio(
"Retriever",
options=["vector", "bm25"],
format_func=lambda x: "Vector Search (ChromaDB)" if x == "vector" else "BM25 (Keyword)",
horizontal=True
)
# Text input for the user's fitness question
question = st.text_input(
"Ask a fitness question:",
placeholder="e.g., What exercises target my glutes without equipment?"
)What does this code do?
This sets up the interactive input layer of your app.
- st.set_page_config sets the browser tab title and uses a wide layout for better readability.
- st.radio renders a horizontal toggle so users can compare Vector vs BM25 retrieval live.
- st.text_input creates a text field where users type their fitness questions.
- Save app.py.
You won't see a change yet. The next chunk adds the response logic that makes the app functional.
Add source transparency
A good RAG demo shows its sources. Hiring managers want to see that your system is transparent about which documents informed the answer.
- In app.py, add the following code below the question = st.text_input(...) block:
# Only run the query when the user has typed something
if question:
with st.spinner("Searching exercises and generating answer..."):
result = query_rag(question, retriever_type=retriever_type)
# Display the generated answer
st.markdown("### Answer")
st.write(result["answer"])
# Show each retrieved exercise as an expandable card
st.markdown("### Retrieved Exercises")
for i, source in enumerate(result["sources"], 1):
with st.expander(f"{i}. {source['name']}"):
st.write(f"**Muscles:** {source['primary_muscles']}")
st.write(f"**Equipment:** {source['equipment']}")
st.write(f"**Level:** {source['level']}")
st.write(f"**Category:** {source['category']}")What does this code do?
This block handles the full query-to-display pipeline.
- st.spinner shows a loading message while the retriever and LLM process the question.
- query_rag calls your existing RAG chain from rag_chain.py, passing the selected retriever type.
- st.expander renders each source exercise as a collapsible card showing the muscle group, equipment, level, and category. Users can expand any source to verify the answer.
- Save app.py.
Why show retrieved sources?
Source transparency is what separates a portfolio RAG project from a toy demo. It proves to reviewers that your system grounds its answers in real data, and it lets users spot-check whether the retriever pulled relevant exercises.
✔️ Awesome, I've got everything!
Great. Double check you've saved your file before moving on.
ⓧ I'd like to double check the full code
import streamlit as st
from rag_chain import query_rag
st.set_page_config(page_title="FitRAG - Fitness Exercise Assistant", layout="wide")
st.title("FitRAG - Fitness Exercise Assistant")
st.markdown("Ask questions about exercises, muscle targeting, equipment, and workout programming.")
retriever_type = st.radio(
"Retriever",
options=["vector", "bm25"],
format_func=lambda x: "Vector Search (ChromaDB)" if x == "vector" else "BM25 (Keyword)",
horizontal=True
)
question = st.text_input(
"Ask a fitness question:",
placeholder="e.g., What exercises target my glutes without equipment?"
)
if question:
with st.spinner("Searching exercises and generating answer..."):
result = query_rag(question, retriever_type=retriever_type)
st.markdown("### Answer")
st.write(result["answer"])
st.markdown("### Retrieved Exercises")
for i, source in enumerate(result["sources"], 1):
with st.expander(f"{i}. {source['name']}"):
st.write(f"**Muscles:** {source['primary_muscles']}")
st.write(f"**Equipment:** {source['equipment']}")
st.write(f"**Level:** {source['level']}")
st.write(f"**Category:** {source['category']}")Run and verify the app
- Launch the Streamlit app by running this command in your terminal:
streamlit run app.py- Your browser should open automatically to a local URL (typically http://localhost:8501).
- Type a test question into the input field, such as What exercises target my glutes without equipment?.
- You should see a generated answer appear under the "Answer" heading, followed by numbered exercise cards you can expand.
- Try switching the retriever radio button from **Vector Search (ChromaDB)** to **BM25 (Keyword)** and re-submitting the same question to compare results.
App not loading?
- Make sure your virtual environment is active. If the streamlit command is not found, activate your venv and try again.
- Check that your OPENROUTER_API_KEY environment variable is set in the same terminal session where you run the app.
- If you see an import error for rag_chain, make sure rag_chain.py is in the same folder as app.py.
Help me debug why my Streamlit app won't start.
Your FitRAG system is now fully interactive. Anyone can open the app, ask a question, and see exactly which exercises informed the answer. Next up, you'll take this further in the Secret Mission by adding hybrid retrieval that combines the strengths of both approaches.
Secret mission
Add Hybrid Retrieval with Reciprocal Rank Fusion
Your vector retriever captures semantic meaning. Your BM25 retriever nails exact keyword matches. What if you could combine both strengths into one retriever that outperforms either alone? Build a hybrid retriever using Reciprocal Rank Fusion and prove it wins in your eval harness.
Clean Up Your Resources
Clean Up Your Resources
This project runs entirely locally, so there are no ongoing costs. ChromaDB stores data on your machine, OpenRouter free models have no charges, and the exercise dataset is public domain.
Decide whether to keep your resources running, pause them to come back later, or delete them entirely.
Resources you used:
- Project folder on your Desktop containing all Python files, chroma_db/, and data/.
- Python virtual environment (inside your project folder).
- OpenRouter account (free, no billing).
- OPENROUTER_API_KEY environment variable (session-only).
Keep everything running
No action needed. Choose this if you're still actively building or plan to extend the project soon.
- Your project files, chroma_db/ vector store, and data/exercises.json remain on disk.
- Your OpenRouter account stays active with free-tier access.
- The OPENROUTER_API_KEY environment variable only lasts for your current terminal session. You will need to re-export it if you close and reopen your terminal.
Pause - I'll come back to this later
Shut down any running processes to free up memory. Your files stay on disk for when you return.
- If Streamlit is still running, press Ctrl+C in the terminal to stop it.
- Deactivate your virtual environment by running:
deactivate- When you come back, reactivate your virtual environment and re-export your OPENROUTER_API_KEY before running any scripts.
Delete - I don't want to use this again
Remove all project files and start fresh. This deletes everything you built in this project.
- If Streamlit is still running, press Ctrl+C in the terminal to stop it.
- Deactivate your virtual environment by running:
deactivate- Navigate to your Desktop and delete the project folder by running these commands (macOS/Linux):
cd ~/Desktop
rm -rf fitrag- On Windows, run:
cd ~\Desktop
rmdir /s /q fitrag- This removes all Python files, the chroma_db/ directory, data/exercises.json, and the virtual environment.
- Your OPENROUTER_API_KEY environment variable disappears automatically when you close the terminal.
- Your OpenRouter account remains active. Since it is free, there is no cost to keeping it.
Nice Work!
Nice Work!
You just built a production-grade RAG system with a proper evaluation framework. This is the exact project structure hiring managers look for in AI engineer portfolios.
You've learned how to:
- Build a RAG pipeline over 800+ fitness exercises using ChromaDB vector search and a free LLM via OpenRouter.
- Implement a BM25 baseline retriever and a keyword precision@5 evaluation harness that compares retrieval approaches with measurable metrics.
- Deploy an interactive Streamlit UI that lets anyone query the system and inspect retrieved sources in under 30 seconds.
- Added hybrid retrieval using Reciprocal Rank Fusion that outperforms both individual retrieval approaches in the eval harness.
Ready to quiz yourself?