RAG Document Q&A with Docker and Qdrant

Build a document Q&A system using RAG, Qdrant, and LlamaIndex in Docker.

Introduction

30 Second Summary

Ever needed to find a specific answer buried in a pile of documents? Instead of reading through everything manually, imagine asking a question in plain English and getting an accurate answer instantly.

In this project, you will build a document Q&A system that answers natural-language questions about your own text files. The system uses RAG (retrieval-augmented generation) to combine vector search with LLM-powered answers.

What You'll Build

A Python application backed by Qdrant (a vector database running in Docker) and LlamaIndex that ingests your documents, stores them as vector embeddings, and generates answers grounded in your content.

By the end of this project, you'll have:

  • A Qdrant vector database running in a Docker container with persistent storage via Docker Compose.
  • A Python ingestion pipeline that loads documents, chunks them, generates embeddings with OpenAI, and stores vectors in Qdrant.
  • A query engine that retrieves relevant context and generates grounded answers using GPT-5.4 mini.
  • Secret Mission: A Streamlit web interface for browser-based document Q&A.

Are there any prerequisites?

You'll need basic Python knowledge and familiarity with the command line. No prior Docker or AI experience is required.

Before We Start

Every AI application that answers questions about documents needs two things: a place to store document knowledge and a way to search it intelligently. In this project, you will build both using Docker containers and Python.

Before you can write any code, you need three tools in place: Docker Desktop to run your vector database, Python 3.12 to run your scripts, and an OpenAI API key to generate embeddings and LLM responses.

In this step, get ready to:
  • Install Docker Desktop and verify it is running.
  • Set up Python 3.12 with a virtual environment.
  • Get an OpenAI API key and set it as an environment variable.
Install Docker Desktop

Docker Desktop lets you run containers on your machine. You will use it to run Qdrant (your vector database) as a container in the next step.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or PowerShell (Windows) and press Enter to open it.
  • Check if Docker is already installed by running this command:
docker --version

✔️ I see a version number

You should see something like Docker version 28.x.x. Docker is ready to go.

  • Make sure Docker Desktop is running by opening it from your applications. You should see the Docker whale icon in your menu bar (macOS) or system tray (Windows).

ⓧ Command not found

  • Download Docker Desktop from docker.com/products/docker-desktop.
  • Run the installer and follow the on-screen prompts.
  • Once installation finishes, open Docker Desktop. It may take a minute to start for the first time.
  • Return to your terminal and run docker --version again. You should now see a version number.

Still not seeing a version number?

Close your terminal and reopen it. New installations sometimes require a fresh terminal session to register the docker command.

help me troubleshoot Docker Desktop installation

Install Python 3.12 and create a virtual environment

Python 3.12 is required because the LlamaIndex packages you will install later depend on features introduced in this version.

  • Check your Python version by running this command:
python3 --version

✔️ I see version 3.12 or higher

You should see something like Python 3.12.x or Python 3.13.x. You are good to go.

ⓧ I see an older version

If you see Python 3.11 or lower, you need to install Python 3.12.

  • Download Python 3.12 from python.org/downloads.
  • Run the installer. On Windows, check the box that says Add python.exe to PATH before clicking Install Now.
  • Close and reopen your terminal after installation.

macOS shipping an older Python?

macOS often ships with an older system Python. After installing 3.12 from python.org, you may need to use python3.12 instead of python3 to invoke the correct version.

  • Run python3 --version again (or python3.12 --version on macOS) to confirm you now see 3.12.

ⓧ Command not found

  • Download Python 3.12 from python.org/downloads.
  • Run the installer. On Windows, check the box that says Add python.exe to PATH before clicking Install Now.
  • Close and reopen your terminal after installation.
  • Run python3 --version to confirm you see Python 3.12.x.

Still getting command not found?

On Windows, try python --version (without the 3). The Windows installer sometimes registers the command as python rather than python3.

help me get Python 3.12 working

Now create a project folder and a virtual environment inside it.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir rag-qa-system
cd rag-qa-system
  • Create a virtual environment by running this command:
python3 -m venv venv

What is a virtual environment?

A virtual environment is an isolated Python installation inside your project folder. Packages you install here stay separate from your system Python, so different projects can use different package versions without conflicts.

  • Activate the virtual environment by running the command for your operating system:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate

You should see (venv) appear at the beginning of your terminal prompt. This confirms the virtual environment is active.

Don't see (venv) in your prompt?

On Windows, if you see an error about execution policies, run Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned in PowerShell first, then try the activate command again.

help me activate my Python virtual environment

Get an OpenAI API key

Your RAG system uses OpenAI to generate vector embeddings and LLM answers. You need an API key to authenticate these requests.

Will this cost money?

Yes, but very little. The text-embedding-3-small model costs $0.02 per 1M tokens, and gpt-5.4-mini costs $0.75 per 1M input tokens. For a small document set, your total spend will be under $0.05.

You need a paid OpenAI account (Tier 1 or above). The free tier does not support these models.

  • Go to platform.openai.com/api-keys and sign in to your OpenAI account.
  • Click Create new secret key.
  • Give it a name like rag-qa-project and click Create secret key.
  • Copy the key immediately. You will not be able to see it again after closing this dialog.

Keep your key safe

Never share your API key or commit it to version control. You will paste it into a terminal command next, which keeps it out of your project files.

  • Back in your terminal (still inside the rag-qa-system folder with (venv) active), set the environment variable by running the command for your operating system:

macOS

export OPENAI_API_KEY=your-api-key-here

Windows

$env:OPENAI_API_KEY="your-api-key-here"
  • Replace your-api-key-here with the actual key you just copied from OpenAI.

What does this command do?

This sets OPENAI_API_KEY as an environment variable in your current terminal session. The Python libraries you install later will read this variable automatically to authenticate with the OpenAI API.

This variable lasts only for the current terminal session. If you close and reopen your terminal, you will need to set it again.

  • Verify the variable is set by running:

macOS

echo $OPENAI_API_KEY

Windows

echo $env:OPENAI_API_KEY

You should see your API key printed back in the terminal. If it shows a blank line, re-run the export command above.

Seeing a blank line?

Make sure you did not accidentally include extra spaces around the = sign in the export command. The format must be OPENAI_API_KEY=sk-... with no spaces.

help me set my OpenAI API key as an environment variable

Your environment is ready. Next up, you will launch Qdrant as a Docker container and see its dashboard in your browser.

Launch Qdrant in Docker

Your environment is ready with Python, Docker, and your API key in place. Now it's time to launch the infrastructure that powers your RAG system's memory.

Your documents need to be stored as searchable vectors somewhere. Qdrant is an open-source vector database purpose-built for this job. You'll run it as a Docker container using Docker Compose, which defines your infrastructure in a single configuration file.

In this step, get ready to:
  • Create your project directory and docker-compose.yml file.
  • Start Qdrant with Docker Compose and verify it's running.
  • Visit the Qdrant dashboard in your browser.
Create the project directory and Docker Compose file

Every Docker Compose project needs a folder to house its configuration. You'll create a project directory on your Desktop so you can find it easily.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir rag-document-qa
cd rag-document-qa
  • Confirm you're in the right place by running:
pwd

You should see a path ending in rag-document-qa.

Don't see rag-document-qa in the path?

  • Make sure you ran both mkdir and cd commands. If you only ran mkdir, you're still on the Desktop.
  • Check for typos in the folder name.

Still stuck? Help me create and navigate to my project folder.

  • Open the rag-document-qa folder in Cursor (or VS Code) by running:
cursor .

Using VS Code instead?

Run code . instead of cursor . to open the folder in VS Code.

  • Create a new file called docker-compose.yml in the rag-document-qa folder.
  • Paste the following configuration into docker-compose.yml:
services:
  qdrant:
    image: qdrant/qdrant:v1.18.0
    container_name: qdrant
    restart: unless-stopped
    ports:
      - "6333:6333"
      - "6334:6334"
    volumes:
      - qdrant_data:/qdrant/storage

volumes:
  qdrant_data:

What does this configuration do?

  • image: qdrant/qdrant:v1.18.0 pins Qdrant to a specific version so your setup is reproducible.
  • container_name: qdrant gives your container a predictable name you can reference later.
  • restart: unless-stopped means Qdrant will auto-restart if it crashes, unless you explicitly stop it.
  • Port 6333 exposes the REST API and the dashboard. Port 6334 exposes the gRPC interface for faster programmatic access.
  • qdrant_data:/qdrant/storage maps a Docker named volume to Qdrant's internal storage path. Your vectors persist even if the container is removed and recreated.
  • Save docker-compose.yml.

File not saving?

  • Make sure you created the file in the rag-document-qa folder, not a subfolder or a different location.
  • Check that the filename is exactly docker-compose.yml with no extra spaces or extensions.

Need help? Why can't I save my docker-compose.yml file?

✔️ Awesome, I've got everything!

Great. Make sure you've saved the file before moving on.

ⓧ I'd like to double check the full code

services:
  qdrant:
    image: qdrant/qdrant:v1.18.0
    container_name: qdrant
    restart: unless-stopped
    ports:
      - "6333:6333"
      - "6334:6334"
    volumes:
      - qdrant_data:/qdrant/storage

volumes:
  qdrant_data:
Start Qdrant with Docker Compose

With the configuration in place, a single command will pull the Qdrant image and start the container in the background.

  • In your terminal, make sure you're inside the rag-document-qa folder (where docker-compose.yml lives).
  • Start Qdrant by running this command:
docker compose up -d

What does this command do?

The -d flag stands for "detached" mode. It starts the container in the background so your terminal stays free for other commands.

The first time you run this, Docker will download the Qdrant image. This may take 30-60 seconds depending on your internet speed.

  • Verify that Qdrant is running by running:
docker compose ps

You should see a qdrant container listed with a status of Up.

Container not starting?

  • Make sure Docker Desktop is running. You should see the Docker icon in your system tray or menu bar.
  • If you see a port conflict error, another service is already using port 6333. Run docker ps to find it, then docker stop <container_id> to free the port.
  • If the image fails to pull, check your internet connection and try running docker compose up -d again.

Still stuck? Help me troubleshoot my Qdrant Docker container.

Visit the Qdrant dashboard

Qdrant includes a built-in web dashboard served from the same port as its REST API. This gives you a visual way to inspect your vector collections.

  • Open your web browser.
  • Navigate to http://localhost:6333/dashboard.

You should see the Qdrant dashboard loaded in your browser. It will be empty because you haven't created any vector collections yet.

Why is the dashboard empty?

The dashboard shows no collections because you haven't ingested any documents yet. In the next step, you'll build the ingestion pipeline and the dashboard will display a populated collection.

Can't reach the dashboard?

  • Run docker compose ps to confirm the container is still running. If it exited, check the logs with docker compose logs qdrant.
  • Make sure you're using http:// not https://. Qdrant does not use HTTPS by default.

Still stuck? Why can't I access the Qdrant dashboard at localhost:6333?

Your vector database is live and ready to store embeddings. Next up, you'll build the Python pipeline that loads your documents and pushes them into Qdrant.

Build the Document Ingestion Pipeline

Your Qdrant vector database is running and ready to store embeddings. Now you need a way to feed it documents.

In this step, you'll build a Python script that reads text files from a folder, splits them into smaller chunks, generates vector embeddings for each chunk using OpenAI, and stores those vectors in Qdrant. This is the ingestion half of a RAG pipeline.

In this step, get ready to:
  • Install Python dependencies for LlamaIndex, Qdrant, and OpenAI.
  • Create an ingestion script that loads, chunks, and embeds documents.
  • Run the script and verify the collection appears in the Qdrant dashboard.
Install Python dependencies

The ingestion pipeline uses LlamaIndex to orchestrate document loading, chunking, and embedding. LlamaIndex has a modular architecture where each integration is a separate package.

  • Create a file called requirements.txt in your project folder by running:
touch requirements.txt
  • Open requirements.txt in your editor and paste the following:
llama-index-core==0.14.22
llama-index-vector-stores-qdrant==0.10.1
llama-index-embeddings-openai==0.3.1
llama-index-llms-openai==0.3.1
qdrant-client==1.14.2

What are these packages?

  • llama-index-core is the orchestration framework that ties everything together.
  • llama-index-vector-stores-qdrant lets LlamaIndex talk to your Qdrant instance.
  • llama-index-embeddings-openai generates vector embeddings using OpenAI's embedding model.
  • llama-index-llms-openai connects to OpenAI's language models for generating answers.
  • qdrant-client is the Python client library for communicating with the Qdrant server.
  • Install all the dependencies by running:
pip install -r requirements.txt
  • Wait for the installation to finish. You should see Successfully installed followed by a list of packages at the end of the output.

Seeing installation errors?

  • Make sure your virtual environment is activated. You should see the environment name in parentheses at the start of your terminal prompt.
  • If you see a permissions error, confirm you're using the pip from your virtual environment, not the system Python.

Help me fix my pip install error

Create sample documents and the ingestion script

Before writing the ingestion script, you need some documents for it to process. You'll create a data/ folder with a sample text file.

  • Create the data/ folder inside your project directory by running:
mkdir data
  • Create a sample document inside the data/ folder by running:
cat << 'EOF' > data/sample.txt
Retrieval-Augmented Generation (RAG) is an AI framework that combines information
retrieval with text generation. Instead of relying solely on training data, RAG
systems retrieve relevant documents from a knowledge base and use them as context
for generating responses.

The RAG pipeline has two main phases: ingestion and querying. During ingestion,
documents are split into chunks, converted to vector embeddings, and stored in a
vector database. During querying, the user's question is also converted to an
embedding, similar chunks are retrieved via semantic search, and those chunks are
passed as context to a large language model that generates the final answer.

Vector databases like Qdrant are optimized for storing and searching high-dimensional
vectors. They use approximate nearest neighbor algorithms to find the most similar
vectors efficiently, even across millions of entries.
EOF

Use your own documents

You can add any .txt files to the data/ folder. The ingestion script reads everything in that folder. Try dropping in meeting notes, documentation, or articles you want to query later.

Now you'll create the ingestion script. This is the core of the pipeline.

  • Create a file called ingest.py in your project folder and paste the following imports and settings configuration:
import os
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, StorageContext, Settings
from llama_index.core.node_parser import SentenceSplitter
from llama_index.vector_stores.qdrant import QdrantVectorStore
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI
from qdrant_client import QdrantClient

# Configure settings
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.llm = OpenAI(model="gpt-5.4-mini")

What does this code do?

  • Settings.embed_model tells LlamaIndex to use OpenAI's text-embedding-3-small model for converting text into vectors. This model costs $0.02 per 1M tokens.
  • Settings.llm configures gpt-5.4-mini as the language model for generating answers later in the query step.
  • LlamaIndex reads your OPENAI_API_KEY environment variable automatically. No need to pass it explicitly.
  • Below the settings configuration, add the Qdrant connection and vector store setup:
# Connect to Qdrant
client = QdrantClient(host="localhost", port=6333)

# Create vector store
vector_store = QdrantVectorStore(
    client=client,
    collection_name="documents",
)

What does this code do?

  • QdrantClient(host="localhost", port=6333) connects to the Qdrant instance you started with Docker Compose in the previous step.
  • QdrantVectorStore creates a LlamaIndex-compatible wrapper around Qdrant. The collection_name is like a table name in a traditional database.
  • Below the vector store setup, add the document loading and index building code:
# Load documents from the data folder
documents = SimpleDirectoryReader("data").load_data()
print(f"Loaded {len(documents)} documents")

# Create storage context with Qdrant
storage_context = StorageContext.from_defaults(vector_store=vector_store)

# Build index with custom chunking
index = VectorStoreIndex.from_documents(
    documents,
    storage_context=storage_context,
    transformations=[SentenceSplitter(chunk_size=512, chunk_overlap=50)],
    show_progress=True,
)

print("Documents ingested successfully!")
print(f"Collection 'documents' created in Qdrant")

What does this code do?

  • SimpleDirectoryReader("data") reads every file in the data/ folder.
  • SentenceSplitter(chunk_size=512, chunk_overlap=50) breaks documents into 512-token chunks with 50 tokens of overlap between consecutive chunks. The overlap ensures sentences at chunk boundaries aren't cut off mid-thought.
  • VectorStoreIndex.from_documents does the heavy lifting. It chunks your documents, sends each chunk to OpenAI for embedding, and stores the resulting vectors in Qdrant.
  • Save ingest.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

import os
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, StorageContext, Settings
from llama_index.core.node_parser import SentenceSplitter
from llama_index.vector_stores.qdrant import QdrantVectorStore
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI
from qdrant_client import QdrantClient

# Configure settings
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.llm = OpenAI(model="gpt-5.4-mini")

# Connect to Qdrant
client = QdrantClient(host="localhost", port=6333)

# Create vector store
vector_store = QdrantVectorStore(
    client=client,
    collection_name="documents",
)

# Load documents from the data folder
documents = SimpleDirectoryReader("data").load_data()
print(f"Loaded {len(documents)} documents")

# Create storage context with Qdrant
storage_context = StorageContext.from_defaults(vector_store=vector_store)

# Build index with custom chunking
index = VectorStoreIndex.from_documents(
    documents,
    storage_context=storage_context,
    transformations=[SentenceSplitter(chunk_size=512, chunk_overlap=50)],
    show_progress=True,
)

print("Documents ingested successfully!")
print(f"Collection 'documents' created in Qdrant")
Run the ingestion script and verify in the dashboard

Time to feed your documents into Qdrant. The script will chunk your text, call OpenAI's embedding API, and store the vectors.

  • Run the ingestion script by executing:
python ingest.py
  • You should see output similar to this:
Loaded 1 documents
Parsing nodes: 100%|██████████| 1/1
Generating embeddings: 100%|██████████| 1/1
Documents ingested successfully!
Collection 'documents' created in Qdrant

What just happened?

Your script read the text file from data/, split it into chunks of 512 tokens, sent each chunk to OpenAI's text-embedding-3-small model to generate vector embeddings, and stored those vectors in Qdrant under the documents collection.

  • Open the Qdrant dashboard in your browser at http://localhost:6333/dashboard.
  • Click on the documents collection in the list. You should see it now contains points (vectors) with metadata.

Don't see the documents collection?

  • Check that your OPENAI_API_KEY environment variable is set in the same terminal where you ran the script. Run echo $OPENAI_API_KEY to verify.
  • Make sure the Qdrant container is still running. Run docker ps and look for the qdrant container.
  • Confirm you have at least one file in your data/ folder. An empty folder causes the script to fail.

Help me fix my ingestion script error

Your documents are now embedded and stored in Qdrant. Next up, you'll build a query engine that searches these vectors and generates answers using GPT-5.4 mini.

Build the Query Engine

Your documents are now stored as vectors inside Qdrant. The ingestion pipeline did the heavy lifting of chunking, embedding, and storing. Now you need a way to actually ask questions and get answers grounded in those documents.

In this step, you'll build a query script that connects to the existing Qdrant collection, finds the most relevant chunks using semantic search, and passes them to GPT-5.4 mini to generate an answer. You'll also see exactly which source documents contributed to each response.

In this step, get ready to:
  • Create query.py to query your document collection.
  • Test the system with questions about your documents.
  • Observe how different questions pull different source chunks.
Create the query script

The query script reuses the same connection pattern as ingest.py but instead of writing vectors, it reads them. It connects to Qdrant, loads your existing collection, and creates a query engine that retrieves the top matching chunks for any question you ask.

  • Create a new file called query.py in your project folder (the same folder where ingest.py lives).
  • Add the imports and settings configuration by pasting this code:
import os
from llama_index.core import VectorStoreIndex, Settings
from llama_index.vector_stores.qdrant import QdrantVectorStore
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI
from qdrant_client import QdrantClient

# Configure settings
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.llm = OpenAI(model="gpt-5.4-mini")

What does this code do?

This is the same configuration pattern you used in ingest.py. The imports bring in LlamaIndex's vector store index, the Qdrant connector, and the OpenAI embedding and LLM models.

The Settings object tells LlamaIndex which embedding model to use for converting your question into a vector, and which LLM to use for generating answers from retrieved chunks.

You won't see a change yet. The next chunk connects to your existing Qdrant data.

  • Below the settings configuration, add the Qdrant connection and query engine setup:
# Connect to Qdrant
client = QdrantClient(host="localhost", port=6333)

# Load existing vector store
vector_store = QdrantVectorStore(
    client=client,
    collection_name="documents",
)

# Create index from existing vector store
index = VectorStoreIndex.from_vector_store(vector_store=vector_store)

# Create query engine
query_engine = index.as_query_engine(similarity_top_k=3)

What does this code do?

  • QdrantClient(host="localhost", port=6333) connects to the same Qdrant instance you launched in Step 2.
  • QdrantVectorStore loads the documents collection that your ingestion script created.
  • VectorStoreIndex.from_vector_store() rebuilds the index from the existing vectors. No re-ingestion needed.
  • similarity_top_k=3 tells the query engine to retrieve the 3 most relevant chunks for each question.

You won't see a change yet. The next chunk adds the interactive loop that lets you ask questions.

  • Below the query engine setup, add the interactive query loop:
# Interactive query loop
print("RAG Document Q&A System")
print("=" * 40)
print("Ask questions about your documents. Type 'quit' to exit.\n")

while True:
    question = input("Question: ")
    if question.lower() in ("quit", "exit", "q"):
        break

    response = query_engine.query(question)
    print(f"\nAnswer: {response}\n")

    # Show source documents
    print("Sources:")
    for i, node in enumerate(response.source_nodes, 1):
        print(f"  {i}. {node.node.metadata.get('file_name', 'Unknown')} (score: {node.score:.3f})")
    print()

What does this code do?

This creates a loop that repeatedly asks for a question, queries the RAG engine, and prints the answer. The loop exits when you type quit, exit, or q.

After each answer, the script prints which source files contributed to the response and their relevance scores. A score closer to 1.0 means a stronger semantic match between your question and that chunk.

  • Save query.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before running it.

ⓧ I'd like to double check the full code

import os
from llama_index.core import VectorStoreIndex, Settings
from llama_index.vector_stores.qdrant import QdrantVectorStore
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI
from qdrant_client import QdrantClient

# Configure settings
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.llm = OpenAI(model="gpt-5.4-mini")

# Connect to Qdrant
client = QdrantClient(host="localhost", port=6333)

# Load existing vector store
vector_store = QdrantVectorStore(
    client=client,
    collection_name="documents",
)

# Create index from existing vector store
index = VectorStoreIndex.from_vector_store(vector_store=vector_store)

# Create query engine
query_engine = index.as_query_engine(similarity_top_k=3)

# Interactive query loop
print("RAG Document Q&A System")
print("=" * 40)
print("Ask questions about your documents. Type 'quit' to exit.\n")

while True:
    question = input("Question: ")
    if question.lower() in ("quit", "exit", "q"):
        break

    response = query_engine.query(question)
    print(f"\nAnswer: {response}\n")

    # Show source documents
    print("Sources:")
    for i, node in enumerate(response.source_nodes, 1):
        print(f"  {i}. {node.node.metadata.get('file_name', 'Unknown')} (score: {node.score:.3f})")
    print()
Test with sample questions

Your query engine is ready. Time to put it to work against the documents you ingested earlier.

  • Run the query script by entering this command in your terminal:
python query.py
  • When you see the Question: prompt, type a question related to the content of your documents and press Enter.
  • You should see an answer generated from your document content, followed by a Sources: section listing the file names and relevance scores.

What should I see?

The output starts with a header saying RAG Document Q&A System followed by a separator line. After you type a question, you'll see an Answer: line with text grounded in your documents, then numbered sources with filenames and scores like (score: 0.842).

Seeing a connection error?

  • Make sure your Qdrant container is still running. Return to your terminal and run docker ps to check.
  • If the container stopped, restart it with docker compose up -d from your project folder.
  • If you see an OpenAI authentication error, confirm your OPENAI_API_KEY environment variable is still set in this terminal session.

Why is my query script failing to connect?

Observe how different questions retrieve different context chunks

Each question you ask gets converted into a vector. Qdrant then finds the 3 stored chunks whose vectors are closest to your question's vector. Different questions activate different parts of your documents.

  • Ask 2-3 different questions about your documents. Pay attention to how the source filenames and scores change between questions.
  • Notice that relevance scores vary. A chunk with a score of 0.9 is a much stronger match than one with 0.6.
  • When you're done experimenting, type quit and press Enter to exit the script.

Why do scores matter?

The relevance score tells you how confident the system is that a chunk contains information relevant to your question. When all scores are low (below 0.5), the system is essentially guessing. High scores mean the answer is well-grounded in your actual documents.

In production RAG systems, you can set a minimum score threshold to avoid returning answers when no chunk is a good match.

Your query engine is working. Next up, you'll experiment with retrieval settings to see how changing chunk sizes and top-k values affects answer quality.

Experiment with Retrieval Settings

Your RAG system is working, but how well it answers depends heavily on two settings: how many chunks you retrieve and how those chunks were created in the first place.

In this step, you will tune both the retrieval side (how many results to pull) and the ingestion side (how text gets split into chunks). Small changes here can dramatically improve or degrade answer quality.

In this step, get ready to:
  • Adjust similarity_top_k to control how many chunks are retrieved.
  • Try different chunk sizes in the ingestion pipeline and observe quality changes.
  • Add a second document and query across both.
Adjust similarity_top_k to control retrieval

The similarity_top_k parameter controls how many document chunks the query engine retrieves before sending them to the LLM. More chunks means more context, but also more noise and higher token usage.

  • In query.py, find this line:
query_engine = index.as_query_engine(similarity_top_k=3)
  • Change 3 to 1, so the line looks like this:
query_engine = index.as_query_engine(similarity_top_k=1)
  • Save query.py and run it by entering this command:
python query.py
  • Ask the same question you tested earlier and notice how only one source appears in the output.
  • Type quit to exit.

What does similarity_top_k control?

With similarity_top_k=1, the LLM only sees the single most relevant chunk. Answers are concise but may miss important context spread across multiple sections.

With higher values (like 5 or more), the LLM has more context to work with but may produce longer, less focused answers. It also uses more tokens per query.

  • Now try a higher value. In query.py, change similarity_top_k=1 to similarity_top_k=5:
query_engine = index.as_query_engine(similarity_top_k=5)
  • Save query.py and run it again:
python query.py
  • Ask the same question. Notice how more sources appear and the answer may be more detailed.
  • Type quit to exit.
  • Return similarity_top_k to its original value of 3. Find this line in query.py:
query_engine = index.as_query_engine(similarity_top_k=5)
  • Change it back to:
query_engine = index.as_query_engine(similarity_top_k=3)
  • Save query.py.

Finding your sweet spot

For most document Q&A use cases, similarity_top_k=3 is a good default. It balances context coverage against noise. Increase it when your documents contain scattered information about a single topic.

Try different chunk sizes in the ingestion pipeline

Chunk size determines how much text fits in each vector. Smaller chunks are more precise in retrieval but may lose surrounding context. Larger chunks preserve context but may dilute the relevance signal.

  • Before re-ingesting, you need to delete the existing collection. Delete it by running this command:
python -c "from qdrant_client import QdrantClient; QdrantClient(host='localhost', port=6333).delete_collection(collection_name='documents')"

Why delete the collection first?

When you re-run ingest.py, LlamaIndex adds new vectors to the existing collection rather than replacing them. Deleting the collection ensures you get a clean set of vectors with your new chunk settings.

  • In ingest.py, find this line:
    transformations=[SentenceSplitter(chunk_size=512, chunk_overlap=50)],
  • Change chunk_size=512 to chunk_size=256 and chunk_overlap=50 to chunk_overlap=25, so the line looks like:
    transformations=[SentenceSplitter(chunk_size=256, chunk_overlap=25)],
  • Save ingest.py and re-run the ingestion by entering this command:
python ingest.py
  • You should see a higher number of chunks processed compared to the first time you ingested, because smaller chunks means more of them.
  • Run query.py and ask the same questions you tested before:
python query.py
  • Compare the answers and relevance scores to what you saw with the larger chunk size. Notice whether answers are more focused but potentially missing broader context.
  • Type quit to exit.

How chunk size affects retrieval quality

With chunk_size=256, each chunk holds about a paragraph. Retrieval becomes more precise because each vector represents a tighter idea. However, the LLM receives shorter fragments and may lack context to form a complete answer.

With chunk_size=512 (the original), each chunk holds roughly a full section. The trade-off is that retrieval may pull in some irrelevant sentences alongside the relevant ones.

  • Now restore the original chunk settings. In ingest.py, change the line back to:
    transformations=[SentenceSplitter(chunk_size=512, chunk_overlap=50)],
  • Save ingest.py.
  • Delete the collection again to prepare for a clean re-ingestion:
python -c "from qdrant_client import QdrantClient; QdrantClient(host='localhost', port=6333).delete_collection(collection_name='documents')"

Seeing an error when deleting the collection?

Make sure your Qdrant container is still running. Check with docker ps to confirm the qdrant container is listed. If it stopped, start it again with docker compose up -d.

help me troubleshoot a Qdrant connection error when trying to delete a collection

Add a second document and query across both

A real RAG system usually ingests many documents. Adding a second file lets you see how the query engine retrieves context from multiple sources and shows source attribution in action.

  • Create a second text file in your data/ folder. You can use any topic you like. Create the file by running this command:
echo "Docker is a platform for building, shipping, and running applications in containers. Containers package an application with all its dependencies so it runs consistently across environments. Docker Compose lets you define multi-container applications in a single YAML file. Volumes persist data across container restarts." > data/docker_basics.txt
  • Confirm the file was created by running:
ls data/
  • You should see both your original file and docker_basics.txt listed.
  • Re-run the ingestion to index both documents:
python ingest.py
  • The output should show Loaded 2 documents (or more if you had multiple files in your data/ folder).
  • Run query.py and try questions that span both documents:
python query.py
  • Try asking a question about Docker (the new document). Notice how the sources section now shows docker_basics.txt as the source with a relevance score.
  • Try a question about your original document. The sources should still point to the correct file.
  • Type quit to exit.

What should I see?

Each answer now shows which file the retrieved chunks came from. When you ask about Docker, you should see docker_basics.txt listed with a high relevance score (close to 1.0). When you ask about your original document's topic, that file appears instead.

This source attribution is critical in production RAG systems. It lets users verify where answers come from and builds trust in the system's output.

  • Open the Qdrant dashboard at http://localhost:6333/dashboard in your browser.
  • Click on the documents collection. You should see the total number of points has increased now that both files are indexed.

✔️ Awesome, I've got everything!

Great. Double check that you have saved both ingest.py and query.py with their original settings restored.

ⓧ I'd like to double check the full code

Here is the final state of both files. They should match the original code with the default chunk and retrieval settings.

import os
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, StorageContext, Settings
from llama_index.core.node_parser import SentenceSplitter
from llama_index.vector_stores.qdrant import QdrantVectorStore
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI
from qdrant_client import QdrantClient

# Configure settings
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.llm = OpenAI(model="gpt-5.4-mini")

# Connect to Qdrant
client = QdrantClient(host="localhost", port=6333)

# Create vector store
vector_store = QdrantVectorStore(
    client=client,
    collection_name="documents",
)

# Load documents from the data folder
documents = SimpleDirectoryReader("data").load_data()
print(f"Loaded {len(documents)} documents")

# Create storage context with Qdrant
storage_context = StorageContext.from_defaults(vector_store=vector_store)

# Build index with custom chunking
index = VectorStoreIndex.from_documents(
    documents,
    storage_context=storage_context,
    transformations=[SentenceSplitter(chunk_size=512, chunk_overlap=50)],
    show_progress=True,
)

print("Documents ingested successfully!")
print(f"Collection 'documents' created in Qdrant")
import os
from llama_index.core import VectorStoreIndex, Settings
from llama_index.vector_stores.qdrant import QdrantVectorStore
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI
from qdrant_client import QdrantClient

# Configure settings
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.llm = OpenAI(model="gpt-5.4-mini")

# Connect to Qdrant
client = QdrantClient(host="localhost", port=6333)

# Load existing vector store
vector_store = QdrantVectorStore(
    client=client,
    collection_name="documents",
)

# Create index from existing vector store
index = VectorStoreIndex.from_vector_store(vector_store=vector_store)

# Create query engine
query_engine = index.as_query_engine(similarity_top_k=3)

# Interactive query loop
print("RAG Document Q&A System")
print("=" * 40)
print("Ask questions about your documents. Type 'quit' to exit.\n")

while True:
    question = input("Question: ")
    if question.lower() in ("quit", "exit", "q"):
        break

    response = query_engine.query(question)
    print(f"\nAnswer: {response}\n")

    # Show source documents
    print("Sources:")
    for i, node in enumerate(response.source_nodes, 1):
        print(f"  {i}. {node.node.metadata.get('file_name', 'Unknown')} (score: {node.score:.3f})")
    print()

You now have hands-on experience tuning a RAG pipeline. You understand how retrieval settings directly affect answer quality, and your system handles multiple documents with clear source attribution. Next up, you will take this further by adding a browser-based interface to your Q&A system.

Secret mission

Add a Streamlit Web Interface

Your RAG system works great in the terminal, but what if you could ask questions through a proper web interface? In this secret mission, you'll build a browser-based UI with Streamlit that connects to your existing Qdrant query engine. You'll also add a sidebar that shows exactly which source documents powered each answer.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs locally using Docker, so there are no ongoing cloud costs. The only charges came from OpenAI API calls during ingestion and querying.

Resources you used:

  • Qdrant Docker container running on ports 6333 and 6334.
  • Docker named volume qdrant_data storing your vector data.
  • Project folder on your Desktop containing docker-compose.yml, ingest.py, query.py, app.py, requirements.txt, and the data/ folder.
  • Python virtual environment with installed dependencies.

Keep everything running

No action needed. Choose this if you're still actively experimenting with your RAG system or want to add more documents.

  • Your Qdrant container persists data across restarts thanks to the named volume.
  • You can keep querying or ingesting new documents at any time.
  • Docker Desktop uses some memory while running, but Qdrant's footprint is minimal for small collections.

Pause - I'll come back to this later

Stop the running containers to free up memory without losing any data. Your vectors and documents stay safe in the named volume.

  • Stop the Streamlit app by pressing Ctrl+C in the terminal where it's running.
  • Stop the Qdrant container by running this command from your project folder:
docker compose stop
  • When you're ready to resume, run docker compose up -d from your project folder to start Qdrant again. Your data will still be there.

Delete - I don't want to use this again

Remove all project resources and start fresh. This permanently deletes your vector data and project files.

  • Stop the Streamlit app by pressing Ctrl+C in the terminal where it's running.
  • Remove the Qdrant container and its named volume by running this command from your project folder:
docker compose down -v
  • Deactivate your Python virtual environment by running:
deactivate
  • Delete your project folder from the Desktop. On macOS or Linux, run:
cd ~/Desktop
rm -rf [[PROJECT_FOLDER="your-project-folder-name"]]
  • On Windows, run:
cd ~\Desktop
Remove-Item -Recurse -Force [[PROJECT_FOLDER="your-project-folder-name"]]
  • Verify the container and volume are gone by running:
docker ps -a
docker volume ls
  • You should no longer see the qdrant container or the qdrant_data volume in the output.

Nice Work!

Nice Work!

You just built a production-style RAG document Q&A system from the ground up. You now have a working pipeline that ingests your own documents, stores them as vectors, and generates grounded answers using semantic search.

You've learned how to:

  • Launch Qdrant as a containerized vector database using Docker Compose with persistent storage.
  • Build a document ingestion pipeline with LlamaIndex that chunks, embeds, and stores vectors in Qdrant.
  • Create a query engine that retrieves relevant context and generates grounded answers with GPT-5.4 mini.
  • Built a Streamlit web interface for browser-based document Q&A (Secret Mission).

Ready to quiz yourself?