Chat With Your Documents Using RAG

Build a Python RAG app that answers questions about any document using LangChain.

Introduction

30 Second Summary

Ever wished you could just ask a long document a question instead of reading the whole thing? That is exactly what modern AI-powered apps do behind the scenes.

In this project, you will build a command-line Python app that loads any text document and answers natural-language questions about it. Your app retrieves relevant passages from the actual document before generating an answer, so it stays grounded in real data instead of making things up.

What You'll Build

A Python RAG (Retrieval-Augmented Generation) application that loads a text document, splits it into chunks, stores the chunks in a local vector database, and answers questions grounded in the document's actual content using LangChain and OpenAI.

By the end of this project, you'll have:

  • A working Python script (rag_app.py) that implements a full RAG pipeline with LangChain.
  • A local ChromaDB vector store populated with embedded document chunks.
  • An interactive question-answering loop that retrieves relevant context and generates grounded answers with OpenAI's GPT model.
  • Secret Mission: Add conversation memory so your app handles follow-up questions like "tell me more about that."

Are there any prerequisites?

You need basic Python knowledge (variables, functions, and running scripts). No prior LangChain or AI experience is required.

Set Up Python and Install Dependencies

Before you can build anything with LangChain, you need a working Python environment and the right packages installed.

In this step, you'll install Python, create an isolated project environment, install the LangChain ecosystem packages, and configure your OpenAI API key so your app can talk to GPT models.

In this step, get ready to:
  • Install Python 3.11+ and create a project environment.
  • Install LangChain, OpenAI, and ChromaDB packages.
  • Set up your OpenAI API key as an environment variable.
Install Python 3.11+ and create your project

This project requires Python 3.11 or higher. Let's check whether you already have it installed.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or Command Prompt (Windows) and press Enter to open it.
  • Check your Python version by running this command:

macOS

python3 --version

Windows

python --version

✔️ I see version 3.11 or higher

Nice, Python is ready. You should see something like Python 3.11.x or Python 3.12.x. Move on to creating your project folder below.

ⓧ I see an older version

If you see Python 3.9 or 3.10, you need to install a newer version. LangChain's packages work best with 3.11+.

  • Download Python 3.11+ from the official site: python.org/downloads.
  • Run the installer. On Windows, check the box that says Add python.exe to PATH before clicking Install.
  • After installing, close and reopen your terminal, then run the version check again.

macOS users with an older default Python

macOS often ships with an older Python version that shadows newer installs. After installing Python 3.11+, try running python3.11 --version or python3.12 --version to confirm the new version is available.

If you use a version manager like pyenv, run pyenv install 3.11 and then pyenv local 3.11 to activate it for this project.

ⓧ Command not found

Python is not installed yet. Let's fix that.

  • Download Python 3.11+ from the official site: python.org/downloads.
  • Run the installer. On Windows, check the box that says Add python.exe to PATH before clicking Install.
  • After installing, close and reopen your terminal, then run the version check again to confirm.

Now let's create your project folder and a virtual environment to keep your packages isolated.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir rag-project
cd rag-project
  • Create a virtual environment by running this command:

macOS

python3 -m venv .venv

Windows

python -m venv .venv
  • Activate the virtual environment by running:

macOS

source .venv/bin/activate

Windows

.venv\Scripts\activate
  • You should see (.venv) appear at the start of your terminal prompt. This confirms the virtual environment is active.

What is a virtual environment?

A virtual environment is an isolated Python workspace. Packages you install inside it stay separate from your system Python, so different projects can use different package versions without conflicting.

Don't see (.venv) in your prompt?

Make sure you ran the activate command from inside the rag-project folder. On Windows, if you see an error about script execution, run Set-ExecutionPolicy -Scope CurrentUser RemoteSigned in PowerShell first, then try activating again.

Still stuck? Help me activate my Python virtual environment.

Now let's open this project in your code editor.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type VS Code (or Cursor if you use that) and press Enter to open it.
  • Click File in the top menu bar.
  • Click Open Folder.
  • Select the rag-project folder on your Desktop, then click Open.
Install the LangChain packages

Your RAG app needs several packages from the LangChain ecosystem, plus ChromaDB for the vector database.

  • Install all the required packages by running this command in your terminal (make sure you see (.venv) in your prompt):
pip install langchain langchain-openai langchain-community langchain-chroma langchain-text-splitters chromadb openai

What are all these packages?

LangChain splits its functionality across focused packages. langchain is the core framework. langchain-openai provides the OpenAI model integrations.

langchain-community includes community-built tools like document loaders. langchain-chroma connects LangChain to the ChromaDB vector store. langchain-text-splitters handles breaking documents into chunks.

  • Wait for the installation to finish. You should see Successfully installed followed by a list of package names at the end of the output.

Seeing an error during install?

Make sure your virtual environment is active (you see (.venv) in your prompt). If you closed and reopened the terminal, you need to activate it again.

If you see a permissions error, do NOT use sudo pip install. Instead, make sure you activated the virtual environment first.

Still stuck? Help me fix my pip install error for LangChain packages.

Get and set your OpenAI API key

Your app needs an OpenAI API key to generate embeddings and get answers from GPT. The key tells OpenAI who is making the request.

  • Open your browser and go to platform.openai.com/api-keys.
  • Sign in or create an OpenAI account if you don't have one yet.
  • Click Create new secret key.
  • Give it a name like rag-project and click Create secret key.
  • Copy the key immediately. You will not be able to see it again after closing this dialog.

Keep your API key safe

Your API key is a secret. Do not share it with anyone or commit it to a public repository. Store it in a safe location like a notes app or password manager until you set it as an environment variable below.

  • Back in your terminal, set the API key as an environment variable by running this command:

macOS

export OPENAI_API_KEY="your-api-key-here"

Windows

setx OPENAI_API_KEY "your-api-key-here"

Windows users: reopen your terminal

The setx command sets the variable for future terminal sessions. After running it, close your terminal, reopen it, navigate back to your project folder, and reactivate your virtual environment.

  • Replace your-api-key-here with the actual key you just copied. Keep the quotes around it.

Why an environment variable?

Setting your key as the OPENAI_API_KEY environment variable means LangChain's OpenAI integrations will automatically read it. You never have to paste the key directly into your code.

  • Verify your key is set by running this command:

macOS

echo $OPENAI_API_KEY

Windows

echo %OPENAI_API_KEY%
  • You should see your API key printed in the terminal. This confirms LangChain will be able to find it.

Seeing a blank line instead of your key?

On macOS, the export command only lasts for the current terminal session. If you closed and reopened your terminal, run the export command again. On Windows, make sure you reopened your terminal after running setx.

Need help? Help me set up my OPENAI_API_KEY environment variable.

Your environment is ready. Next up, you'll load a document and split it into chunks that your RAG app can search through.

Load and Chunk a Document

Your environment is set up and ready to go. Now it's time to give your app some real data to work with.

A RAG app answers questions using actual document content instead of relying on an LLM's training data alone. To make that work, you first need to load a document and break it into small, searchable pieces called chunks. In this step, you'll create a text file, load it with LangChain's document loader, and split it into chunks that your app can later search through.

In this step, get ready to:
  • Create a sample text document to query.
  • Load the document and split it into searchable chunks using LangChain.
  • Run the script and verify the chunks are created.
Create a sample text file

Your RAG app needs a source document to retrieve answers from. You'll create a plain text file with content you want to ask questions about.

  • In VS Code, click File in the top menu bar.
  • Click New File.
  • Name the file document.txt and save it in your project folder.
  • Paste in at least 2-3 paragraphs of text content you find interesting. A Wikipedia article, a blog post, or a news article all work well.
  • Save document.txt.

Longer documents work better

Aim for at least 500 words of text. The more content in your document, the more interesting your question-answering results will be later. A full Wikipedia section or a medium-length blog post is ideal.

  • Confirm the file exists by running:
ls

You should see document.txt in the output.

Write code to load and chunk the document

LangChain provides purpose-built tools for loading documents and splitting them into chunks. You'll use TextLoader to read your file and RecursiveCharacterTextSplitter to break it into pieces small enough for a vector search.

  • Create a new file called rag_app.py in VS Code by clicking File then New File.
  • Add these imports at the top of rag_app.py:
import os
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_chroma import Chroma

What are all these imports for?

You're importing more than you need for this step because you'll use them in later steps. Here's what each group does:

  • langchain_openai provides the LLM and embedding models (used in Steps 3-4).
  • langchain_core provides chain-building blocks like prompts, output parsers, and runnables (used in Step 4).
  • langchain_community provides TextLoader to read files from disk.
  • langchain_text_splitters provides RecursiveCharacterTextSplitter to chunk your document.
  • langchain_chroma provides the ChromaDB vector store (used in Step 3).

You won't see a change yet. The next chunk uses these imports to load your document.

  • Below the imports in rag_app.py, add this code to load and split your document:
# Step 2: Load and chunk the document
loader = TextLoader("document.txt", encoding="utf-8")
documents = loader.load()

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50
)
chunks = text_splitter.split_documents(documents)
print(f"Split document into {len(chunks)} chunks")

What does this code do?

  • TextLoader("document.txt", encoding="utf-8") reads your text file from disk into a LangChain Document object.
  • RecursiveCharacterTextSplitter breaks the document into smaller pieces by splitting on natural boundaries like paragraphs and sentences.
  • chunk_size=500 means each chunk contains a maximum of 500 characters.
  • chunk_overlap=50 means each chunk shares 50 characters with the next one. This overlap prevents cutting off a sentence right in the middle of a key idea.
  • split_documents(documents) does the actual splitting and returns a list of chunk objects.
  • Save rag_app.py.

Seeing a red underline on an import?

If VS Code shows a warning on any import line, make sure your virtual environment is selected as the Python interpreter. Press Cmd+Shift+P (macOS) or Ctrl+Shift+P (Windows), type Python: Select Interpreter, and choose the interpreter inside your .venv folder.

help me fix my import errors in rag_app.py

✔️ Awesome, I've got everything!

Great. Double check you've saved rag_app.py before moving on.

ⓧ I'd like to double check the full code

Here's what your complete rag_app.py should look like so far:

import os
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_chroma import Chroma

# Step 2: Load and chunk the document
loader = TextLoader("document.txt", encoding="utf-8")
documents = loader.load()

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50
)
chunks = text_splitter.split_documents(documents)
print(f"Split document into {len(chunks)} chunks")
Run the script and verify your chunks
  • Run your script by entering this command in the terminal:
python rag_app.py

You should see output like:

Split document into 12 chunks

What should I see?

The exact number of chunks depends on how much text you put in document.txt. A typical Wikipedia section (around 1000 words) produces roughly 10-15 chunks with a chunk_size of 500. As long as you see a number greater than 0, your document loaded and split correctly.

Seeing FileNotFoundError?

This means Python can't find document.txt. Make sure the file is saved in the same folder where you're running the script. Run ls in your terminal to confirm both rag_app.py and document.txt appear in the same directory.

help me fix the FileNotFoundError in my RAG app

Your document is loaded and chunked. Next up, you'll turn these chunks into vector embeddings and store them in a database so your app can search through them at lightning speed.

Create Embeddings and Store in ChromaDB

Your document is now split into manageable chunks. The next challenge is making those chunks searchable so you can quickly find the ones most relevant to a user's question.

In this step, you'll convert each chunk into a numerical vector (called an embedding) that captures its meaning. You'll store these embeddings in a local ChromaDB database so your app can perform lightning-fast similarity searches.

In this step, get ready to:
  • Initialize OpenAI embeddings and create a ChromaDB vector store from your document chunks.
  • Test the vector store by running a similarity search query.
Initialize OpenAI embeddings and create the vector store

An embedding is a list of numbers that represents the meaning of a piece of text. Texts with similar meanings produce similar number patterns, which is how your app will find relevant chunks for a given question.

  • In rag_app.py, find the line that says print(f"Split document into {len(chunks)} chunks"), then copy the following code below it:
# Step 3: Create embeddings and store in ChromaDB
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db"
)
print("Vector store created successfully")

What does this code do?

OpenAIEmbeddings(model="text-embedding-3-small") creates an embeddings object that uses OpenAI's small embedding model to convert text into vectors. This model is fast and cost-effective for learning projects.

Chroma.from_documents() takes your document chunks, converts each one into a vector using the embeddings object, and stores everything in a local database at ./chroma_db. This database persists on disk so you don't need to re-embed every time you run the script.

What is a vector store?

A vector store is a database optimized for storing and searching embeddings. When you ask a question, the store converts your question into a vector and finds the stored chunks whose vectors are most similar. This is how RAG finds relevant context without reading the entire document every time.

  • Save rag_app.py.
  • Run the script by entering this command in your terminal:
python rag_app.py

You should see both your chunk count from before and a new line: Vector store created successfully.

  • Confirm the chroma_db directory was created by running:
ls

You should see chroma_db listed alongside rag_app.py and document.txt in the output.

Seeing an authentication error?

If you see an error about an invalid API key, double-check that your OPENAI_API_KEY environment variable is still set in this terminal session. Closing and reopening the terminal clears it.

If you see an error about the model name, make sure you typed text-embedding-3-small exactly as shown (no extra spaces or capitals).

Help me debug my ChromaDB or OpenAI embeddings error

Test the vector store with a similarity search

Before building the full RAG chain, you should verify that the vector store can actually find relevant chunks for a given question. You'll add a quick test at the bottom of your script, run it, then remove the test lines.

  • In rag_app.py, add these temporary test lines at the very bottom of the file (after the print("Vector store created successfully") line):
# Temporary test - remove after verifying
results = vectorstore.similarity_search("test query about your document", k=2)
for i, doc in enumerate(results):
    print(f"\nResult {i+1}:")
    print(doc.page_content[:200])

What does this test do?

similarity_search() converts your query into a vector and returns the k most similar chunks from the database. If the results contain text related to your query, your embeddings and vector store are working correctly.

  • Replace "test query about your document" with a real question about the content of your document.txt file.
  • Save rag_app.py and run it again:
python rag_app.py

Below the vector store confirmation, you should see two results containing text passages from your document that relate to your test question.

  • Once you've confirmed the results look relevant, delete the temporary test lines you just added (the 4 lines starting from # Temporary test).
  • Save rag_app.py.

Results don't seem relevant to your question?

Make sure your test query relates to a topic actually covered in your document.txt. If you pasted a short document, the chunks may all look similar. Try a more specific question targeting a detail you know is in the text.

If you see a chroma_db directory error, delete the chroma_db folder and run the script again. ChromaDB can conflict if you re-run without clearing the old data.

Help me debug my similarity search results

✔️ Awesome, I've got everything!

Great. Double-check you've saved your file and removed the temporary test lines.

ⓧ I'd like to double check the full code

Here's the complete rag_app.py at this point (with the temporary test lines removed):

import os
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_chroma import Chroma

# Step 2: Load and chunk the document
loader = TextLoader("document.txt", encoding="utf-8")
documents = loader.load()

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50
)
chunks = text_splitter.split_documents(documents)
print(f"Split document into {len(chunks)} chunks")

# Step 3: Create embeddings and store in ChromaDB
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db"
)
print("Vector store created successfully")

Your document chunks are now embedded and stored in a searchable vector database. Next up, you'll build the full RAG chain that retrieves relevant context and generates answers using OpenAI.

Build the RAG Chain with LCEL

Your document is chunked, embedded, and stored in ChromaDB.

Now you need a way to actually ask questions and get answers grounded in that data. This is where LangChain Expression Language (LCEL) comes in. LCEL lets you compose multiple components into a single chain using the | pipe operator, wiring together a retriever, a prompt template, an LLM, and an output parser into one pipeline that flows from question to answer.

In this step, get ready to:
  • Create a retriever and prompt template that grounds answers in your document.
  • Assemble a full LCEL chain using the pipe operator.
  • Test the chain with a question and verify it returns a context-grounded answer.
Create a retriever and define the prompt template

Before you can wire up a chain, you need its individual components. A retriever fetches the most relevant chunks from your vector store for any given question. A prompt template tells the LLM how to use those chunks to form an answer.

  • In rag_app.py, find the line that says print("Vector store created successfully").
  • Below that line, add the retriever and helper function by copying this code:
# Step 4: Build the RAG chain with LCEL
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})


def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

What does this code do?

  • .as_retriever(search_kwargs={"k": 3}) converts your vector store into a retriever that returns the top 3 most similar chunks for any query.
  • format_docs() is a helper that joins those retrieved chunks into a single string separated by blank lines. The chain will use this to pass context into the prompt.
  • Now add the prompt template and LLM initialization directly below the format_docs function:
prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a helpful assistant. Answer the question based only on the following context. If you cannot find the answer in the context, say so.\n\nContext:\n{context}"),
    ("human", "{question}")
])

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

What does this code do?

  • ChatPromptTemplate.from_messages() defines a two-part prompt: a system message that instructs the LLM to answer only from the provided {context}, and a human message that passes through the user's {question}.
  • temperature=0 makes the model's responses deterministic. This is ideal for factual Q&A where you want consistent, grounded answers.
  • Save rag_app.py.

Seeing a red underline on the imports?

If your editor shows import errors for ChatPromptTemplate or ChatOpenAI, make sure your virtual environment is activated. Your imports at the top of the file already include these classes from the previous steps.

Help me debug import errors in my LangChain RAG app

Assemble the LCEL chain

Now you have all the components: a retriever, a formatter, a prompt, and an LLM. LCEL lets you connect them with the | pipe operator so data flows through each stage automatically.

  • Add the chain assembly below the llm line:
rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

What does this chain do?

This single expression defines the entire RAG pipeline. When you invoke it with a question, here is what happens:

  • The dictionary at the start creates two parallel inputs. retriever | format_docs fetches the top 3 relevant chunks and joins them into a string. RunnablePassthrough() passes the original question through unchanged.
  • Those two values (context and question) fill into the prompt template's placeholders.
  • The filled prompt flows into ChatOpenAI, which generates an answer grounded in the retrieved context.
  • StrOutputParser() extracts the plain text from the LLM's response object.
  • Save rag_app.py.

Seeing a syntax error on the chain?

Make sure the opening parenthesis ( is on the same line as rag_chain =. Each | pipe must be at the start of a new line, indented inside the parentheses. Double-check that the closing ) is on its own line at the end.

Help me fix a syntax error in my LCEL chain

Test the chain with a sample question

The chain is assembled but you have not run it yet. You will add temporary test lines to invoke it with a question, verify the answer is grounded in your document, and then remove those lines before moving on.

  • Add these temporary test lines at the very end of rag_app.py:
# Temporary test - remove after verifying
response = rag_chain.invoke("What is this document about?")
print(f"\nAnswer: {response}")

What does this code do?

rag_chain.invoke() sends your question through the entire pipeline: retrieval, prompt filling, LLM generation, and output parsing. The printed answer should reference specific details from your document.txt rather than generic knowledge.

  • Save rag_app.py.
  • Run the script by entering this command in your terminal:
python rag_app.py
  • You should see the chunk count, the vector store success message, and then an answer that references specific content from your document.

Getting an authentication error?

This means your OPENAI_API_KEY environment variable is not set in the current terminal session. Re-export it in the same terminal where you are running the script.

If the answer looks generic and does not mention anything specific from your document, check that document.txt has enough content. Short documents (under a few sentences) may not produce meaningful retrieval results.

Help me troubleshoot my RAG chain test

  • Once you have confirmed the answer is grounded in your document, delete the three temporary test lines (the comment, the response line, and the print line) from the end of your file.
  • Save rag_app.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file with the test lines removed.

ⓧ I'd like to double check the full code

Here is what your complete rag_app.py should look like at this point:

import os
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_chroma import Chroma

# Step 2: Load and chunk the document
loader = TextLoader("document.txt", encoding="utf-8")
documents = loader.load()

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50
)
chunks = text_splitter.split_documents(documents)
print(f"Split document into {len(chunks)} chunks")

# Step 3: Create embeddings and store in ChromaDB
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db"
)
print("Vector store created successfully")

# Step 4: Build the RAG chain with LCEL
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})


def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)


prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a helpful assistant. Answer the question based only on the following context. If you cannot find the answer in the context, say so.\n\nContext:\n{context}"),
    ("human", "{question}")
])

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

Your RAG chain is working. It retrieves relevant context from your document and generates grounded answers. Next up, you will wrap this chain in an interactive loop so you can ask as many questions as you want from the terminal.

Add a Question-Answering Loop

Your RAG chain can retrieve context and generate answers. Right now, though, it only handles a single question before the script ends.

In this step, you'll wrap the chain in an interactive loop so you can keep asking questions about your document until you choose to stop. This is what turns a one-shot script into a usable Q&A tool.

In this step, get ready to:
  • Add an interactive question-answering loop to your script.
  • Run the complete app and test it with multiple questions.
Wrap the chain in an interactive loop

A Q&A app needs to stay open and accept new questions. A while True loop keeps the script running until the user explicitly types quit.

  • In rag_app.py, scroll to the bottom of the file (below the rag_chain definition).
  • Add the interactive Q&A loop by copying this code at the end of the file:
# Step 5: Interactive Q&A loop
print("\n--- Document Q&A Ready ---")
print("Ask questions about your document. Type 'quit' to exit.\n")

while True:
    question = input("You: ")
    if question.lower() == "quit":
        print("Goodbye!")
        break
    answer = rag_chain.invoke(question)
    print(f"\nAssistant: {answer}\n")

What does this code do?

This loop keeps your app alive and listening for questions.

  • The two print() statements display a welcome message so the user knows the app is ready.
  • input("You: ") pauses the script and waits for the user to type a question.
  • If the user types quit (in any case), the loop breaks and the script ends.
  • rag_chain.invoke(question) sends the question through the full RAG pipeline: retrieve context, format it, fill the prompt, call the LLM, and parse the output.
  • Save rag_app.py.

Seeing a syntax error?

  • Make sure the new code is at the same indentation level as the rest of the top-level code (no extra indentation around the while True line).
  • Check that the f"\nAssistant: {answer}\n" line uses an f prefix before the opening quote.

Why is my Q&A loop not working?

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before running it.

ⓧ I'd like to double check the full code

Here is the complete rag_app.py file with all steps included:

import os
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_chroma import Chroma

# Step 2: Load and chunk the document
loader = TextLoader("document.txt", encoding="utf-8")
documents = loader.load()

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50
)
chunks = text_splitter.split_documents(documents)
print(f"Split document into {len(chunks)} chunks")

# Step 3: Create embeddings and store in ChromaDB
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db"
)
print("Vector store created successfully")

# Step 4: Build the RAG chain with LCEL
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})


def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)


prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a helpful assistant. Answer the question based only on the following context. If you cannot find the answer in the context, say so.\n\nContext:\n{context}"),
    ("human", "{question}")
])

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

# Step 5: Interactive Q&A loop
print("\n--- Document Q&A Ready ---")
print("Ask questions about your document. Type 'quit' to exit.\n")

while True:
    question = input("You: ")
    if question.lower() == "quit":
        print("Goodbye!")
        break
    answer = rag_chain.invoke(question)
    print(f"\nAssistant: {answer}\n")
Run the complete app and test your questions

Time to see the full pipeline in action. When you run the script, it will load your document, create embeddings, build the chain, and drop you into an interactive session.

  • Run your app by entering this command in the terminal:
python rag_app.py

You should see the chunk count, a confirmation that the vector store was created, and then the prompt You: waiting for your input.

  • Ask 3-4 questions about the content of your document.txt file. Try questions that require the app to find specific information from the text.
  • When you're done testing, type quit and press Enter to exit.

What should I see?

The answers should be grounded in your document's content. If you ask something that isn't covered in the document, the assistant should tell you it cannot find the answer in the context.

This is the key difference between RAG and a plain chatbot. The LLM only answers based on the retrieved chunks, not its general training data.

Getting empty or irrelevant answers?

  • Make sure your document.txt has enough content. Very short documents (a few sentences) may not provide enough context for meaningful answers.
  • Check that your OPENAI_API_KEY environment variable is still set in the terminal session where you're running the script.
  • Try asking questions that use specific keywords from your document rather than very broad questions.

Why are my RAG answers not matching my document content?

Your document Q&A app is fully functional. You've built a complete RAG pipeline from scratch. Next up, the Secret Mission challenges you to add conversation memory so your app can handle follow-up questions.

Secret mission

Add Conversation Memory

Your app answers questions, but it has no memory. Ask "tell me more about that" and it won't know what "that" refers to. In this secret mission, you'll add conversation memory so your RAG app can handle follow-up questions across multiple turns.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs locally with a pay-per-use API, so there are no ongoing costs unless you run the script again.

Resources you used:

  • Your project folder (containing rag_app.py, document.txt, chroma_db/, and your virtual environment).
  • Your OpenAI API key (set as the OPENAI_API_KEY environment variable and stored on the OpenAI platform).

Keep everything running

No action needed. Choose this if you plan to keep experimenting with different documents or extending your RAG app.

  • Your project folder stays on your Desktop ready to use.
  • Your virtual environment and installed packages remain in place.
  • Your OpenAI API key stays active. You are only charged when you run the script and make API calls.

Pause - I'll come back to this later

Shut down running processes to free up memory. Your files stay intact for when you return.

  • If your script is still running in the terminal, type quit to exit the Q&A loop.
  • Deactivate your virtual environment by running:
deactivate
  • Close VS Code. Your project folder remains on your Desktop untouched.
  • No need to revoke your API key. You are only charged when the script makes calls.

Delete - I don't want to use this again

Remove all project resources and start fresh. This cannot be undone.

  • Close VS Code if it is still open.
  • Delete your project folder by running the command for your operating system:

macOS

cd ~/Desktop
rm -rf rag-langchain

Windows

cd ~\Desktop
Remove-Item -Recurse -Force rag-langchain

Use your actual folder name

Replace rag-langchain with whatever you named your project folder in Step 1.

  • Revoke your OpenAI API key so it cannot be used again.
  • Go to platform.openai.com/api-keys.
  • Find the key you created for this project and click the trashcan icon next to it.
  • Confirm deletion. The key will no longer work for any application.

Nice Work!

Nice Work!

Nice work! You just built a fully functional document Q&A app powered by LangChain's RAG pipeline. Your app answers questions grounded in real data rather than hallucinating.

Here's what you accomplished:

  • Loaded and chunked a text document using LangChain's document loaders and text splitters.
  • Created vector embeddings and stored them in a local ChromaDB vector store.
  • Built an LCEL chain using the | pipe operator that retrieves relevant context and generates grounded answers with OpenAI through an interactive Q&A loop.
  • Added conversation memory for multi-turn follow-up questions.

Ready to quiz yourself?