Build a RAG System with LlamaIndex

Build a production-grade RAG pipeline with vector search and reranking.

Introduction

30 Second Summary

Every time you ask an AI assistant a question about your company's internal documents, it can only guess. Without a retrieval system connecting the AI to your actual data, the answers are unreliable.

In this project, you will build a production-grade RAG (Retrieval-Augmented Generation) pipeline that ingests documents, stores them as searchable vectors in a managed database, and retrieves relevant context to generate grounded answers with citations.

What You'll Build

A document Q&A system powered by LlamaIndex and Pinecone that retrieves relevant chunks from your documents and generates cited answers using OpenAI.

By the end of this project, you'll have:

  • A Python RAG pipeline with document ingestion, intelligent chunking, and vector embeddings stored in a Pinecone serverless index.
  • A query engine with reranking that retrieves the best context and generates grounded, cited answers.
  • An evaluation harness that measures retrieval precision and answer faithfulness to catch regressions before they reach users.
  • Secret Mission: Add hybrid search combining BM25 keyword matching with dense vectors to handle queries that pure semantic search misses.

Are there any prerequisites?

You should be comfortable with basic Python (variables, functions, running scripts from the terminal) and familiar with how API keys and environment variables work. No prior experience with RAG, vector databases, or LlamaIndex is required.

Set Up Your Environment

A RAG pipeline needs several tools working together. LlamaIndex orchestrates the pipeline, Pinecone stores your vectors, and OpenAI generates embeddings.

Before writing any code, you need a clean Python environment with the right versions of each dependency. In this step, you'll install Python, create a project folder with a virtual environment, and install the three core packages.

In this step, get ready to:
  • Install and verify Python 3.10 or higher.
  • Create your project directory and a virtual environment.
  • Install LlamaIndex, Pinecone, and OpenAI packages.
Verify Python 3.10 or higher is installed

LlamaIndex and the Pinecone SDK both require Python 3.10 or higher. Let's confirm you have the right version before going further.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or Command Prompt (Windows) and press Enter to open it.
  • Check your Python version by running this command:
python3 --version

Look at the version number your terminal prints, then pick the tab below that matches what you see.

✔️ I see version 3.10 or higher

Your Python version is ready. Move on to the next substep.

ⓧ I see an older version

Your system has an older Python that won't work with LlamaIndex or Pinecone. You need to install Python 3.10 or higher alongside your current version.

  • Visit python.org/downloads and download the latest Python 3.10+ installer for your operating system.
  • Run the installer. On Windows, check the box labeled Add Python to PATH before clicking Install Now.
  • After installation, close and reopen your terminal.
  • Run python3 --version again to confirm the new version appears.

macOS still showing the old version?

macOS often ships with an older Python that shadows new installs. If python3 --version still shows the old version after installing, try running the full path to the new binary (e.g. /Library/Frameworks/Python.framework/Versions/3.12/bin/python3) or use a version manager like pyenv to switch your active Python.

ⓧ Command not found

Python isn't installed on your system yet.

  • Visit python.org/downloads and download the latest Python 3.10+ installer for your operating system.
  • Run the installer. On Windows, check the box labeled Add Python to PATH before clicking Install Now.
  • After installation, close and reopen your terminal.
  • Run python3 --version to confirm installation. You should see a version number of 3.10 or higher.

Windows says python3 is not recognized?

On Windows, try python --version instead (without the 3). If that shows 3.10 or higher, use python instead of python3 for all commands in this project.

If neither works, reinstall Python from python.org and make sure Add Python to PATH is checked during installation.

help me fix Python not being recognized in my terminal

Create your project directory and virtual environment

A virtual environment isolates your project's packages from everything else on your system. This prevents version conflicts between projects.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir rag-pipeline
cd rag-pipeline
  • Confirm you're in the right folder by running:
pwd

You should see a path ending in /Desktop/rag-pipeline.

  • Create a virtual environment by running this command:
python3 -m venv venv

What does this command do?

python3 -m venv venv tells Python to create a virtual environment in a folder called venv inside your project directory. This folder contains a copy of the Python interpreter and a fresh pip installation that are completely separate from your system Python.

  • Activate the virtual environment by running the command for your operating system:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate

You should see (venv) appear at the start of your terminal prompt. This confirms the virtual environment is active.

Don't see (venv) in your prompt?

Make sure you ran the activation command for your specific operating system. On Windows PowerShell, you may need to run Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser first to allow script execution.

help me activate my Python virtual environment

Install project dependencies

Your virtual environment is empty right now. You need to install the three packages that power your RAG pipeline.

  • Create a requirements.txt file in your rag-pipeline folder. This file pins the exact versions of each dependency so your project is reproducible.
  • Paste the following contents into requirements.txt:
llama-index==0.14.22
pinecone==9.0.1
openai>=1.0.0

What does each package do?

llama-index is the RAG framework that handles document loading, chunking, embedding, indexing, and querying.

pinecone is the Python SDK for the Pinecone vector database where your embeddings will be stored and searched.

openai is the client library for OpenAI's APIs, which LlamaIndex uses to generate embeddings and LLM responses.

  • Save requirements.txt.
  • Install all dependencies by running this command:
pip install -r requirements.txt

This will download and install all three packages along with their sub-dependencies. It may take a minute or two.

  • Once the installation finishes, verify the packages are installed by running:
pip list | grep -E "llama-index|pinecone|openai"

You should see llama-index, pinecone, and openai listed with their version numbers.

Seeing installation errors?

Make sure your virtual environment is active (you should see (venv) in your prompt). If you run pip install without an active venv, it installs to your system Python and may hit permission errors.

If you see a message about pip being outdated, run pip install --upgrade pip first, then retry the install command.

help me fix my pip install errors

✔️ Awesome, I've got everything!

Great. Double check you've saved requirements.txt and that your virtual environment is still active.

ⓧ I'd like to double check the full code

Here is what your requirements.txt should look like:

llama-index==0.14.22
pinecone==9.0.1
openai>=1.0.0

Your environment is ready. Next up, you'll configure your API keys and create a Pinecone index to store your document embeddings.

Configure API Keys and Create Your Pinecone Index

Your dependencies are installed and ready to go. Now your RAG pipeline needs to connect to two external services: OpenAI for generating vector embeddings from text, and Pinecone for storing and searching those vectors.

In this step, you'll create accounts on both platforms, set your API keys as environment variables, and provision a Pinecone serverless index that will hold your document vectors.

In this step, get ready to:
  • Sign up for OpenAI and Pinecone accounts.
  • Set API keys as environment variables and create a config file.
  • Create a serverless Pinecone index.
Sign up for OpenAI and Pinecone

Your pipeline uses OpenAI's embedding model to convert text into vectors and Pinecone to store them. Both services require API keys to authenticate requests.

  • Open platform.openai.com in your browser.
  • Sign up for a new account (or log in if you already have one).
  • Navigate to the API keys page in the Developer Platform.
  • Click Create new secret key.
  • Copy the key and paste it somewhere safe (a text file or password manager). You will not be able to see the full key again after you close the dialog.

Why can't I see my key again?

For security, OpenAI only shows the full secret key at creation time. If you lose it, you'll need to generate a new one. Save it now before closing the dialog.

  • Open app.pinecone.io in your browser.
  • Sign up for a free account and choose the Starter plan.
  • After signing up, go to API keys in your project.
  • Copy the default API key that Pinecone generated for your project. Save it alongside your OpenAI key.

What is the Pinecone Starter plan?

The Starter plan is completely free ($0/month). It includes 2 GB of storage, 1,000,000 reads per month, and up to 5 serverless indexes. That is far more than this project needs.

Set your API keys as environment variables

Storing API keys in environment variables keeps them out of your source code. Your Python scripts will read these variables at runtime.

  • Set both keys in your terminal by running these commands (replace the placeholder values with your actual keys):

macOS

export OPENAI_API_KEY=your-api-key-here
export PINECONE_API_KEY=your-api-key-here

Windows

set OPENAI_API_KEY=your-api-key-here
set PINECONE_API_KEY=your-api-key-here

How long do these variables last?

These environment variables only persist for the current terminal session. If you close and reopen your terminal, you'll need to set them again. For a permanent solution, add them to your shell profile (.bashrc, .zshrc, or Windows system environment variables).

Now create a config file that centralizes all your project settings. This file reads the environment variables you just set and defines constants for the embedding model, index name, and cloud region.

  • Create a new file called config.py in your project folder by running:
touch config.py
  • Open config.py and add the following code:
import os

OPENAI_API_KEY = os.environ.get("OPENAI_API_KEY", "your-api-key-here")
PINECONE_API_KEY = os.environ.get("PINECONE_API_KEY", "your-api-key-here")
PINECONE_INDEX_NAME = "rag-production"
PINECONE_CLOUD = "aws"
PINECONE_REGION = "us-east-1"
EMBEDDING_MODEL = "text-embedding-3-small"
EMBEDDING_DIMENSION = 1536
LLM_MODEL = "gpt-4o-mini"

What does this config do?

  • `os.environ.get()` reads your API keys from the environment variables you just set. The second argument is a fallback for when the variable is missing.
  • `PINECONE_INDEX_NAME` is set to `"rag-production"`. This is the name your index will use in Pinecone.
  • `EMBEDDING_DIMENSION = 1536` matches the output size of OpenAI's `text-embedding-3-small` model.
  • `PINECONE_REGION` is set to `"us-east-1"` because the Starter plan only supports this AWS region.
  • Save config.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

import os

OPENAI_API_KEY = os.environ.get("OPENAI_API_KEY", "your-api-key-here")
PINECONE_API_KEY = os.environ.get("PINECONE_API_KEY", "your-api-key-here")
PINECONE_INDEX_NAME = "rag-production"
PINECONE_CLOUD = "aws"
PINECONE_REGION = "us-east-1"
EMBEDDING_MODEL = "text-embedding-3-small"
EMBEDDING_DIMENSION = 1536
LLM_MODEL = "gpt-4o-mini"
Create and run the Pinecone index script

A vector database index is the structure that holds your embedded document chunks. You need to create one in Pinecone before you can store anything.

  • Create a new file called create_index.py in your project folder by running:
touch create_index.py
  • Open create_index.py and add the following code:
"""Create and configure the Pinecone serverless index."""
from pinecone import Pinecone, ServerlessSpec
from config import (
    PINECONE_API_KEY,
    PINECONE_INDEX_NAME,
    PINECONE_CLOUD,
    PINECONE_REGION,
    EMBEDDING_DIMENSION,
)


def create_pinecone_index():
    pc = Pinecone(api_key=PINECONE_API_KEY)

    # Check if index already exists
    existing_indexes = [idx.name for idx in pc.list_indexes()]
    if PINECONE_INDEX_NAME in existing_indexes:
        print(f"Index '{PINECONE_INDEX_NAME}' already exists.")
        return

    # Create a new serverless index
    pc.create_index(
        name=PINECONE_INDEX_NAME,
        dimension=EMBEDDING_DIMENSION,
        metric="cosine",
        spec=ServerlessSpec(cloud=PINECONE_CLOUD, region=PINECONE_REGION),
    )
    print(f"Index '{PINECONE_INDEX_NAME}' created successfully.")


if __name__ == "__main__":
    create_pinecone_index()

What does this script do?

  • `Pinecone(api_key=PINECONE_API_KEY)` creates an authenticated client using the API key from your config.
  • The script checks whether an index named `rag-production` already exists. If it does, it skips creation.
  • `pc.create_index()` provisions a new serverless index with 1536 dimensions (matching your embedding model), cosine similarity as the distance metric, and AWS us-east-1 as the deployment region.
  • `ServerlessSpec` tells Pinecone to use their serverless infrastructure. No pods to manage or scale.
  • Save create_index.py.
  • Run the script to create your Pinecone index:
python create_index.py

You should see this output in your terminal:

Index 'rag-production' created successfully.

Pinecone is eventually consistent

After creation, there may be a brief delay (a few seconds) before your index is fully ready to accept vectors. This is normal for serverless infrastructure.

Seeing an authentication error?

  • Make sure your PINECONE_API_KEY environment variable is set in the same terminal session where you're running the script. Run echo $PINECONE_API_KEY (macOS) or echo %PINECONE_API_KEY% (Windows) to verify.
  • Check that you copied the full API key with no leading or trailing spaces.
  • Confirm you selected the Starter plan during Pinecone sign-up. The free plan provisions a project with a default API key automatically.

Help me debug my Pinecone authentication error.

✔️ Awesome, I've got everything!

Great. Double check you've saved both files before moving on.

ⓧ I'd like to double check the full code

"""Create and configure the Pinecone serverless index."""
from pinecone import Pinecone, ServerlessSpec
from config import (
    PINECONE_API_KEY,
    PINECONE_INDEX_NAME,
    PINECONE_CLOUD,
    PINECONE_REGION,
    EMBEDDING_DIMENSION,
)


def create_pinecone_index():
    pc = Pinecone(api_key=PINECONE_API_KEY)

    # Check if index already exists
    existing_indexes = [idx.name for idx in pc.list_indexes()]
    if PINECONE_INDEX_NAME in existing_indexes:
        print(f"Index '{PINECONE_INDEX_NAME}' already exists.")
        return

    # Create a new serverless index
    pc.create_index(
        name=PINECONE_INDEX_NAME,
        dimension=EMBEDDING_DIMENSION,
        metric="cosine",
        spec=ServerlessSpec(cloud=PINECONE_CLOUD, region=PINECONE_REGION),
    )
    print(f"Index '{PINECONE_INDEX_NAME}' created successfully.")


if __name__ == "__main__":
    create_pinecone_index()

Your Pinecone index is live and waiting for vectors. Next up, you'll create sample documents, chunk them intelligently, and embed them into this index.

Ingest and Chunk Documents

Your Pinecone index is created and waiting for data. The next challenge is getting your documents into that index in a way that makes them easy to search.

Raw documents are too large to embed in one piece. In this step, you will create sample technical documents, split them into smaller chunks that preserve meaning, then embed and store those chunks in Pinecone.

In this step, get ready to:
  • Create sample technical documents about a fictional product.
  • Load and chunk those documents using LlamaIndex's SentenceSplitter.
  • Embed and store all chunks in your Pinecone index.
Create sample technical documents

A RAG system needs a knowledge base to search through. You will create two markdown documents about a fictional product called AcmeTech CloudSync. These documents simulate the kind of internal technical documentation companies feed into RAG pipelines.

  • Create a data directory inside your project folder by running this command:
mkdir data
  • Confirm the directory exists by running:
ls

You should see data listed alongside your existing files like config.py and create_index.py.

  • Create a new file called product_guide.md inside the data directory, then paste the following content:
# AcmeTech CloudSync Product Guide

## Overview
AcmeTech CloudSync is an enterprise file synchronization platform that enables teams to collaborate on documents in real-time across multiple devices and operating systems.

## Key Features

### Real-Time Collaboration
CloudSync supports up to 50 simultaneous editors on a single document. Changes are propagated within 200ms using WebSocket connections with conflict resolution based on operational transforms.

### Version History
Every document maintains a complete version history for 90 days on the Standard plan and unlimited history on the Enterprise plan. Versions can be compared side-by-side and restored with one click.

### Security
All data is encrypted at rest using AES-256 and in transit using TLS 1.3. CloudSync is SOC 2 Type II certified and HIPAA compliant on Enterprise plans.

## Pricing

| Plan | Storage | Users | Price |
|------|---------|-------|-------|
| Starter | 100 GB | Up to 10 | $12/user/month |
| Standard | 1 TB | Up to 100 | $24/user/month |
| Enterprise | Unlimited | Unlimited | Custom pricing |

## Troubleshooting

### Error Code E-4012
This error occurs when the sync agent cannot reach the CloudSync servers. Common causes:
1. Firewall blocking outbound connections on port 443
2. Proxy configuration not set in the sync agent settings
3. DNS resolution failure for sync.acmetech.io

Resolution: Check network connectivity, verify proxy settings in Settings > Network > Proxy, and ensure sync.acmetech.io resolves correctly.

### Error Code E-5003
This error indicates a file conflict that could not be automatically resolved. This happens when two users edit the same section of a document while one is offline.

Resolution: Open the conflict resolution dialog via File > Resolve Conflicts. Choose which version to keep or merge changes manually.

## API Reference

### Authentication
All API requests require a Bearer token in the Authorization header:
```
Authorization: Bearer <your-api-token>
```

### Endpoints

#### GET /api/v2/files
Returns a list of files in the authenticated user's workspace.

Query parameters:
- `limit` (integer, default 50): Maximum number of results
- `offset` (integer, default 0): Pagination offset
- `sort` (string): Sort by "name", "modified", or "size"

#### POST /api/v2/files/sync
Triggers a manual sync for the specified file IDs.

Request body:
```json
{
  "file_ids": ["file_abc123", "file_def456"],
  "priority": "high"
}
```

Why these documents?

This product guide covers features, pricing, troubleshooting codes, and API endpoints. It gives your RAG pipeline a variety of content types to search through, from structured tables to step-by-step instructions.

  • Create a second file called architecture.md inside the data directory, then paste the following content:
# AcmeTech CloudSync Architecture

## System Design

CloudSync uses a microservices architecture deployed on Kubernetes across three AWS regions (us-east-1, eu-west-1, ap-southeast-1) for global availability.

### Core Services

1. **Sync Engine**: Handles real-time document synchronization using CRDTs (Conflict-free Replicated Data Types). Written in Rust for performance.
2. **Storage Service**: Manages file storage on S3 with intelligent tiering. Hot files in S3 Standard, cold files automatically moved to S3 Glacier after 30 days.
3. **Auth Service**: OAuth 2.0 and SAML 2.0 authentication. Supports MFA via TOTP and hardware keys (FIDO2).
4. **Search Service**: Full-text search powered by Elasticsearch with custom analyzers for code files.

### Data Flow

1. Client uploads file change via WebSocket connection
2. Sync Engine validates the change and applies CRDT merge
3. Merged document is stored in Storage Service
4. Change event published to Kafka for downstream consumers
5. Search Service indexes the updated content
6. Connected clients receive the update via WebSocket push

### Performance SLAs

| Metric | Target | Current |
|--------|--------|--------|
| Sync latency (p99) | < 500ms | 320ms |
| API response time (p95) | < 200ms | 145ms |
| Uptime | 99.99% | 99.995% |
| File upload throughput | 100 MB/s per user | 150 MB/s |

### Disaster Recovery

- RPO (Recovery Point Objective): 0 seconds (synchronous replication)
- RTO (Recovery Time Objective): < 5 minutes (automated failover)
- Backups: Continuous to S3 Cross-Region Replication + daily snapshots to Glacier

What does this document cover?

This architecture document has system design details, performance SLAs, and disaster recovery specs. It gives your RAG pipeline a second source to retrieve from, which lets you test whether the system correctly identifies which document answered a question.

  • Save both files.
Build the ingestion script

Now you need a script that loads these documents, splits them into chunks small enough to embed, and upserts those embeddings into Pinecone. LlamaIndex handles this entire pipeline with a few configuration lines.

  • Create a new file called ingest.py in your project folder.
  • Add the imports and environment setup at the top of the file:
"""Ingest documents: load, chunk, embed, and store in Pinecone."""
import os
from llama_index.core import (
    SimpleDirectoryReader,
    StorageContext,
    VectorStoreIndex,
    Settings,
)
from llama_index.core.node_parser import SentenceSplitter
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI
from llama_index.vector_stores.pinecone import PineconeVectorStore
from pinecone import Pinecone
from config import (
    OPENAI_API_KEY,
    PINECONE_API_KEY,
    PINECONE_INDEX_NAME,
    EMBEDDING_MODEL,
    LLM_MODEL,
)

os.environ["OPENAI_API_KEY"] = OPENAI_API_KEY

What does this setup do?

The imports pull in LlamaIndex's core components: SimpleDirectoryReader loads files from a folder, SentenceSplitter handles chunking, and VectorStoreIndex orchestrates embedding and storage.

The os.environ line sets the OpenAI API key as an environment variable so LlamaIndex can access it when making embedding calls.

  • Below the imports, add the ingest_documents() function:

def ingest_documents():
    # Configure LlamaIndex settings
    Settings.embed_model = OpenAIEmbedding(model=EMBEDDING_MODEL)
    Settings.llm = OpenAI(model=LLM_MODEL)
    Settings.node_parser = SentenceSplitter(chunk_size=512, chunk_overlap=50)

    # Load documents from the data directory
    documents = SimpleDirectoryReader("data").load_data()
    print(f"Loaded {len(documents)} documents.")

    # Connect to Pinecone
    pc = Pinecone(api_key=PINECONE_API_KEY)
    pinecone_index = pc.Index(PINECONE_INDEX_NAME)

    # Create vector store and storage context
    vector_store = PineconeVectorStore(pinecone_index=pinecone_index)
    storage_context = StorageContext.from_defaults(vector_store=vector_store)

    # Build index (embeds and upserts all chunks)
    index = VectorStoreIndex.from_documents(
        documents,
        storage_context=storage_context,
    )
    print("Documents ingested and indexed in Pinecone.")

    # Verify ingestion
    stats = pinecone_index.describe_index_stats()
    print(f"Pinecone index stats: {stats.total_vector_count} vectors stored.")

    return index


if __name__ == "__main__":
    ingest_documents()

What does this function do?

  • Settings.node_parser = SentenceSplitter(chunk_size=512, chunk_overlap=50) splits each document into 512-token chunks with 50 tokens of overlap between consecutive chunks. The overlap prevents cutting mid-sentence at boundaries.
  • SimpleDirectoryReader("data").load_data() reads every file in the data/ folder and attaches metadata like the file name to each document.
  • VectorStoreIndex.from_documents() runs the full pipeline: chunk the documents, embed each chunk with OpenAI's text-embedding-3-small model, and upsert the vectors into Pinecone.
  • describe_index_stats() queries Pinecone for the total number of stored vectors so you can confirm everything was saved.

Why 512-token chunks with 50-token overlap?

Chunks that are too large dilute the embedding with irrelevant context. Chunks that are too small lose meaning. 512 tokens is the production default that balances precision and context. The 50-token overlap ensures a sentence that spans a chunk boundary still appears in both chunks, so no information is lost at the seam.

  • Save ingest.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved ingest.py before running it in the next substep.

ⓧ I'd like to double check the full code

"""Ingest documents: load, chunk, embed, and store in Pinecone."""
import os
from llama_index.core import (
    SimpleDirectoryReader,
    StorageContext,
    VectorStoreIndex,
    Settings,
)
from llama_index.core.node_parser import SentenceSplitter
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI
from llama_index.vector_stores.pinecone import PineconeVectorStore
from pinecone import Pinecone
from config import (
    OPENAI_API_KEY,
    PINECONE_API_KEY,
    PINECONE_INDEX_NAME,
    EMBEDDING_MODEL,
    LLM_MODEL,
)

os.environ["OPENAI_API_KEY"] = OPENAI_API_KEY


def ingest_documents():
    # Configure LlamaIndex settings
    Settings.embed_model = OpenAIEmbedding(model=EMBEDDING_MODEL)
    Settings.llm = OpenAI(model=LLM_MODEL)
    Settings.node_parser = SentenceSplitter(chunk_size=512, chunk_overlap=50)

    # Load documents from the data directory
    documents = SimpleDirectoryReader("data").load_data()
    print(f"Loaded {len(documents)} documents.")

    # Connect to Pinecone
    pc = Pinecone(api_key=PINECONE_API_KEY)
    pinecone_index = pc.Index(PINECONE_INDEX_NAME)

    # Create vector store and storage context
    vector_store = PineconeVectorStore(pinecone_index=pinecone_index)
    storage_context = StorageContext.from_defaults(vector_store=vector_store)

    # Build index (embeds and upserts all chunks)
    index = VectorStoreIndex.from_documents(
        documents,
        storage_context=storage_context,
    )
    print("Documents ingested and indexed in Pinecone.")

    # Verify ingestion
    stats = pinecone_index.describe_index_stats()
    print(f"Pinecone index stats: {stats.total_vector_count} vectors stored.")

    return index


if __name__ == "__main__":
    ingest_documents()
Run the ingestion pipeline

With the script and documents in place, you can now run the full pipeline. This will chunk your documents, send each chunk to OpenAI for embedding, and store the resulting vectors in Pinecone.

  • Run the ingestion script by executing this command:
python ingest.py

The script takes 10-20 seconds to run. You should see output similar to this:

Loaded 2 documents.
Documents ingested and indexed in Pinecone.
Pinecone index stats: 14 vectors stored.

What should I see?

The exact number of vectors depends on how SentenceSplitter divides your documents. With 512-token chunks and 50-token overlap across both files, you should see roughly 10-20 vectors. The important thing is that the count is greater than zero.

Seeing an error?

  • If you see an authentication error mentioning OPENAI_API_KEY, make sure your environment variable is set correctly. Run echo $OPENAI_API_KEY in your terminal to verify.
  • If you see a Pinecone connection error, confirm your PINECONE_API_KEY environment variable is set and that the rag-production index exists in your Pinecone dashboard.
  • If the script cannot find the data directory, make sure you are running the command from your project folder (the one containing config.py).

Help me debug my ingest.py script

Your documents are now embedded and stored in Pinecone, ready to be searched. Next up, you will build a query engine that retrieves the most relevant chunks and generates grounded answers with citations.

Build the Query Engine with Reranking

Your documents are now embedded and stored in Pinecone. The next challenge is turning that data into useful answers.

A naive approach would retrieve chunks by vector similarity and pass them straight to an LLM. The problem is that vector search alone often returns chunks that are semantically similar but not truly relevant. A reranking step solves this by scoring each retrieved chunk against the actual question, keeping only the best matches.

In this step, get ready to:
  • Define a system prompt that enforces grounded, cited answers.
  • Build a query engine with reranking that retrieves top-10 candidates and compresses to top-3.
  • Test the engine with sample questions about your documents.
Create the query engine file with imports and system prompt

The query engine needs to connect to the same Pinecone index, configure the LLM to only answer from retrieved context, and apply reranking. You will put all of this in a new file called query_engine.py.

  • Create a new file called query_engine.py in your project folder (the same folder that contains config.py).
  • Add the imports and system prompt by pasting this code:
"""Build the query engine with reranking and citation support."""
import os
from llama_index.core import VectorStoreIndex, Settings
from llama_index.core.postprocessor import SentenceTransformerRerank
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI
from llama_index.vector_stores.pinecone import PineconeVectorStore
from pinecone import Pinecone
from config import (
    OPENAI_API_KEY,
    PINECONE_API_KEY,
    PINECONE_INDEX_NAME,
    EMBEDDING_MODEL,
    LLM_MODEL,
)

os.environ["OPENAI_API_KEY"] = OPENAI_API_KEY

SYSTEM_PROMPT = """You are a helpful assistant that answers questions based ONLY on the provided context.
If the context does not contain enough information to answer, say "I don't have enough information to answer that."
Always cite the source document for each claim using [Source: filename] format."""

What does this code do?

  • The imports bring in SentenceTransformerRerank from LlamaIndex's core postprocessor module. This is the reranker that will score retrieved chunks against your query.
  • SYSTEM_PROMPT constrains the LLM to answer only from retrieved context. It also forces citation in [Source: filename] format so you can trace every claim back to its source document.
  • Save query_engine.py.

You won't see a change yet. The next chunk adds the function that builds the query engine.

Add the query engine builder with reranking

The build_query_engine() function connects to your existing Pinecone index, wraps it in a LlamaIndex VectorStoreIndex, attaches a reranker, and returns a ready-to-use query engine.

  • In query_engine.py, add the following function below the SYSTEM_PROMPT definition:
def build_query_engine():
    # Configure settings
    Settings.embed_model = OpenAIEmbedding(model=EMBEDDING_MODEL)
    Settings.llm = OpenAI(model=LLM_MODEL, system_prompt=SYSTEM_PROMPT)

    # Connect to existing Pinecone index
    pc = Pinecone(api_key=PINECONE_API_KEY)
    pinecone_index = pc.Index(PINECONE_INDEX_NAME)

    # Create vector store and index
    vector_store = PineconeVectorStore(pinecone_index=pinecone_index)
    index = VectorStoreIndex.from_vector_store(vector_store)

    # Create reranker (reranks top-10 results down to top-3)
    reranker = SentenceTransformerRerank(
        model="cross-encoder/ms-marco-MiniLM-L-6-v2",
        top_n=3,
    )

    # Build query engine with reranking
    query_engine = index.as_query_engine(
        similarity_top_k=10,
        node_postprocessors=[reranker],
    )

    return query_engine

What does this code do?

  • VectorStoreIndex.from_vector_store(vector_store) creates an index backed by the vectors you already stored in Pinecone. No re-ingestion needed.
  • similarity_top_k=10 tells the retriever to pull the top 10 chunks from Pinecone based on embedding similarity.
  • The SentenceTransformerRerank postprocessor then re-scores all 10 chunks using a cross-encoder model (cross-encoder/ms-marco-MiniLM-L-6-v2) and keeps only the top 3. This is a free, local model that runs on your CPU.
  • The cross-encoder evaluates each (query, chunk) pair jointly, which is much more accurate than the initial embedding-based similarity. This typically boosts answer precision by 10-30%.

Why rerank instead of just retrieving fewer chunks?

Retrieving only 3 chunks upfront misses relevant results that scored slightly lower in embedding space. Retrieving 10 gives you a wider net, then the cross-encoder applies a smarter, slower scoring pass to pick the truly best 3. You get both recall (wide net) and precision (smart filter).

  • Save query_engine.py.

You won't see a change yet. The next chunk adds the function that actually runs queries and prints results.

Add the query function and test the engine

The query() function takes a question, builds the engine, runs the query, and prints the answer along with source documents and their relevance scores.

  • In query_engine.py, add the following code below the build_query_engine() function:
def query(question: str):
    engine = build_query_engine()
    response = engine.query(question)

    print(f"\nQuestion: {question}")
    print(f"\nAnswer: {response.response}")
    print(f"\nSources:")
    for node in response.source_nodes:
        print(f"  - {node.node.metadata.get('file_name', 'unknown')} (score: {node.score:.3f})")

    return response


if __name__ == "__main__":
    query("What happens when error code E-4012 occurs and how do I fix it?")
    print("\n" + "=" * 60 + "\n")
    query("What is the sync latency SLA for CloudSync?")

What does this code do?

  • query() builds the engine fresh each time it's called. For a tutorial this keeps things simple. In production you would build once and reuse.
  • response.source_nodes gives you the chunks the LLM actually used to generate its answer, along with their reranking scores.
  • The if __name__ block runs two test questions. The first targets troubleshooting content in product_guide.md. The second targets performance data in architecture.md.
  • Save query_engine.py.
  • Run the script by executing this command in your terminal:
python query_engine.py

The first time you run this, the reranker model will download automatically (about 90 MB). After that, you should see two answers with cited sources and relevance scores.

What should I see?

For the first question about error code E-4012, the answer should reference firewall issues, proxy settings, and DNS resolution. The source should show product_guide.md with a high reranking score.

For the sync latency SLA question, the answer should mention the 500ms p99 target. The source should show architecture.md.

Not seeing results or getting an error?

  • If you see a connection error, confirm your OPENAI_API_KEY and PINECONE_API_KEY environment variables are still set in this terminal session.
  • If you see an empty response, make sure you ran ingest.py successfully in the previous step and that your Pinecone index has vectors stored.
  • If the reranker model download fails, check your internet connection. The model downloads from Hugging Face on first use.

Why isn't my query engine returning results?

✔️ Awesome, I've got everything!

Great. Double check you have saved query_engine.py and that both test queries returned grounded answers with source citations.

ⓧ I'd like to double check the full code

Here is the complete query_engine.py file:

"""Build the query engine with reranking and citation support."""
import os
from llama_index.core import VectorStoreIndex, Settings
from llama_index.core.postprocessor import SentenceTransformerRerank
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI
from llama_index.vector_stores.pinecone import PineconeVectorStore
from pinecone import Pinecone
from config import (
    OPENAI_API_KEY,
    PINECONE_API_KEY,
    PINECONE_INDEX_NAME,
    EMBEDDING_MODEL,
    LLM_MODEL,
)

os.environ["OPENAI_API_KEY"] = OPENAI_API_KEY

SYSTEM_PROMPT = """You are a helpful assistant that answers questions based ONLY on the provided context.
If the context does not contain enough information to answer, say "I don't have enough information to answer that."
Always cite the source document for each claim using [Source: filename] format."""


def build_query_engine():
    # Configure settings
    Settings.embed_model = OpenAIEmbedding(model=EMBEDDING_MODEL)
    Settings.llm = OpenAI(model=LLM_MODEL, system_prompt=SYSTEM_PROMPT)

    # Connect to existing Pinecone index
    pc = Pinecone(api_key=PINECONE_API_KEY)
    pinecone_index = pc.Index(PINECONE_INDEX_NAME)

    # Create vector store and index
    vector_store = PineconeVectorStore(pinecone_index=pinecone_index)
    index = VectorStoreIndex.from_vector_store(vector_store)

    # Create reranker (reranks top-10 results down to top-3)
    reranker = SentenceTransformerRerank(
        model="cross-encoder/ms-marco-MiniLM-L-6-v2",
        top_n=3,
    )

    # Build query engine with reranking
    query_engine = index.as_query_engine(
        similarity_top_k=10,
        node_postprocessors=[reranker],
    )

    return query_engine


def query(question: str):
    engine = build_query_engine()
    response = engine.query(question)

    print(f"\nQuestion: {question}")
    print(f"\nAnswer: {response.response}")
    print(f"\nSources:")
    for node in response.source_nodes:
        print(f"  - {node.node.metadata.get('file_name', 'unknown')} (score: {node.score:.3f})")

    return response


if __name__ == "__main__":
    query("What happens when error code E-4012 occurs and how do I fix it?")
    print("\n" + "=" * 60 + "\n")
    query("What is the sync latency SLA for CloudSync?")

Your query engine is working. Next up, you will build an evaluation harness that measures how well your pipeline retrieves the right chunks and generates faithful answers.

Evaluate Your RAG Pipeline

Your query engine retrieves context and generates answers. But how do you know it's working well enough for production?

In this step, you'll build an evaluation harness that measures two critical metrics: retrieval precision (are the right chunks retrieved?) and answer faithfulness (does the answer stay grounded in context?). This is how production teams catch regressions before they reach users.

In this step, get ready to:
  • Create an evaluation dataset with 10 question-answer pairs.
  • Measure retrieval precision and answer faithfulness.
  • Run the evaluation and interpret results.
Create the evaluation dataset

A RAG evaluation dataset is a set of questions where you already know the correct answer and which source document it comes from. By comparing the pipeline's output against these known-good answers, you can measure quality objectively.

  • Create a new file called evaluate.py in your project root (the same folder as query_engine.py).
  • Add the imports and evaluation dataset by pasting this code:
"""Evaluate the RAG pipeline with precision and faithfulness metrics."""
import os
from query_engine import build_query_engine
from config import OPENAI_API_KEY

os.environ["OPENAI_API_KEY"] = OPENAI_API_KEY

# Evaluation dataset: question, expected answer keywords, expected source file
EVAL_DATASET = [
    {
        "question": "What encryption does CloudSync use for data at rest?",
        "expected_keywords": ["AES-256"],
        "expected_source": "product_guide.md",
    },
    {
        "question": "How many simultaneous editors does CloudSync support?",
        "expected_keywords": ["50"],
        "expected_source": "product_guide.md",
    },
    {
        "question": "What is the sync latency p99 target?",
        "expected_keywords": ["500ms"],
        "expected_source": "architecture.md",
    },
    {
        "question": "What technology does the Sync Engine use for conflict resolution?",
        "expected_keywords": ["CRDT", "Conflict-free Replicated Data Types"],
        "expected_source": "architecture.md",
    },
    {
        "question": "What causes error code E-4012?",
        "expected_keywords": ["sync agent", "cannot reach", "servers"],
        "expected_source": "product_guide.md",
    },
    {
        "question": "What is the RPO for disaster recovery?",
        "expected_keywords": ["0 seconds"],
        "expected_source": "architecture.md",
    },
    {
        "question": "What programming language is the Sync Engine written in?",
        "expected_keywords": ["Rust"],
        "expected_source": "architecture.md",
    },
    {
        "question": "How long is version history retained on the Standard plan?",
        "expected_keywords": ["90 days"],
        "expected_source": "product_guide.md",
    },
    {
        "question": "What message broker does CloudSync use?",
        "expected_keywords": ["Kafka"],
        "expected_source": "architecture.md",
    },
    {
        "question": "What is the price of the Standard plan?",
        "expected_keywords": ["$24", "user/month"],
        "expected_source": "product_guide.md",
    },
]

What does this code do?

The script imports build_query_engine from your existing query engine module. This lets the evaluation run the same pipeline you built in the previous step.

Each entry in EVAL_DATASET has three fields: a question, keywords the answer must contain to be considered faithful, and the source document the retriever should find. These 10 pairs cover both of your documents and a range of question types (facts, numbers, troubleshooting, architecture).

  • Save evaluate.py.

Why only 10 pairs?

Production teams typically start with 50-100 labeled pairs. For this project, 10 pairs are enough to learn the evaluation pattern and catch obvious failures. You can always add more pairs later to increase confidence in your metrics.

Build the evaluation functions

You need two measurement functions. Retrieval precision checks if the correct source document appears in the retrieved results. Answer faithfulness checks if the generated answer contains the expected keywords from the source material.

  • In evaluate.py, add the retrieval precision function below the EVAL_DATASET list:
def evaluate_retrieval_precision(engine):
    """Measure whether the correct source document is retrieved."""
    correct = 0
    total = len(EVAL_DATASET)

    for item in EVAL_DATASET:
        response = engine.query(item["question"])
        source_files = [
            node.node.metadata.get("file_name", "")
            for node in response.source_nodes
        ]
        if item["expected_source"] in source_files:
            correct += 1
            status = "\u2713"
        else:
            status = "\u2717"
        print(f"  {status} '{item['question'][:50]}...' -> sources: {source_files}")

    precision = correct / total
    print(f"\nRetrieval Precision: {correct}/{total} = {precision:.1%}")
    return precision

What does this function measure?

For each question, the function queries your pipeline and checks if the expected_source file appears among the returned source nodes. If your retriever consistently pulls from the wrong document, your precision drops and you know the chunking or embedding needs work.

  • Below the precision function, add the answer faithfulness function:
def evaluate_answer_faithfulness(engine):
    """Check if answers contain expected keywords from the source documents."""
    faithful = 0
    total = len(EVAL_DATASET)

    for item in EVAL_DATASET:
        response = engine.query(item["question"])
        answer = response.response.lower()
        has_keyword = any(
            kw.lower() in answer for kw in item["expected_keywords"]
        )
        if has_keyword:
            faithful += 1
            status = "\u2713"
        else:
            status = "\u2717"
        print(f"  {status} '{item['question'][:50]}...'")
        if not has_keyword:
            print(f"      Expected one of: {item['expected_keywords']}")
            print(f"      Got: {answer[:100]}...")

    faithfulness = faithful / total
    print(f"\nAnswer Faithfulness: {faithful}/{total} = {faithfulness:.1%}")
    return faithfulness

What does this function measure?

This function checks whether the LLM's generated answer actually contains the key facts from the source documents. If the answer hallucinates or drifts away from the retrieved context, the expected keywords won't appear and the faithfulness score drops.

When a question fails, the function prints what it expected versus what it got. This makes debugging straightforward.

  • Save evaluate.py.

Seeing a syntax error?

Make sure the evaluate_answer_faithfulness function is at the same indentation level as evaluate_retrieval_precision (no indentation, top-level). Both functions should start at column 0.

Help me debug my evaluate.py file

Run the evaluation and interpret results

The final piece ties both metrics together into a single evaluation run with a summary that tells you if your pipeline meets a production quality bar.

  • Add the run_evaluation function and the entry point below your two evaluation functions in evaluate.py:
def run_evaluation():
    print("Building query engine...")
    engine = build_query_engine()

    print("\n" + "=" * 60)
    print("RETRIEVAL PRECISION")
    print("=" * 60)
    precision = evaluate_retrieval_precision(engine)

    print("\n" + "=" * 60)
    print("ANSWER FAITHFULNESS")
    print("=" * 60)
    faithfulness = evaluate_answer_faithfulness(engine)

    print("\n" + "=" * 60)
    print("SUMMARY")
    print("=" * 60)
    print(f"  Retrieval Precision: {precision:.1%}")
    print(f"  Answer Faithfulness: {faithfulness:.1%}")

    if precision >= 0.8 and faithfulness >= 0.8:
        print("\n  \u2705 Pipeline meets production quality bar (>80% on both metrics)")
    else:
        print("\n  \u26a0\ufe0f Pipeline needs improvement. Consider:")
        if precision < 0.8:
            print("    - Adjusting chunk size or overlap")
            print("    - Adding metadata filters")
        if faithfulness < 0.8:
            print("    - Strengthening the system prompt")
            print("    - Increasing similarity_top_k")


if __name__ == "__main__":
    run_evaluation()

What does this code do?

The run_evaluation function builds a single query engine instance and passes it to both metric functions. This avoids rebuilding the engine 20 times (once per question per metric).

The 80% threshold is the quality bar. If both metrics exceed 80%, your pipeline is production-ready for this dataset. If either falls below, the script prints specific suggestions for improvement.

  • Save evaluate.py.
  • Run the evaluation by executing this command in your terminal:
python evaluate.py

The evaluation will take 1-2 minutes to complete because it runs 20 queries (10 questions for each metric). You should see check marks and crosses printed for each question, followed by a summary with your precision and faithfulness percentages.

How do I interpret the results?

  • Retrieval Precision tells you how often the retriever pulls from the correct source document. If this is low, your chunking or embedding configuration needs tuning.
  • Answer Faithfulness tells you how often the LLM's answer contains the expected facts. If this is low but precision is high, your system prompt needs strengthening.
  • Both above 80% means your pipeline meets the production quality bar for this dataset.

Evaluation scores below 80%?

Low retrieval precision often means chunks are too large (relevant info gets diluted) or too small (context is split across chunks). Your current 512-token chunks with 50-token overlap should work well for this dataset.

Low faithfulness usually means the LLM is paraphrasing too aggressively. The keyword check is case-insensitive, so exact wording isn't required. If a specific question consistently fails, check that the answer contains at least one of the listed keywords.

Help me improve my RAG evaluation scores

✔️ Awesome, I've got everything!

Great. Double check you've saved your file.

ⓧ I'd like to double check the full code

Here's the complete evaluate.py file for comparison:

"""Evaluate the RAG pipeline with precision and faithfulness metrics."""
import os
from query_engine import build_query_engine
from config import OPENAI_API_KEY

os.environ["OPENAI_API_KEY"] = OPENAI_API_KEY

# Evaluation dataset: question, expected answer keywords, expected source file
EVAL_DATASET = [
    {
        "question": "What encryption does CloudSync use for data at rest?",
        "expected_keywords": ["AES-256"],
        "expected_source": "product_guide.md",
    },
    {
        "question": "How many simultaneous editors does CloudSync support?",
        "expected_keywords": ["50"],
        "expected_source": "product_guide.md",
    },
    {
        "question": "What is the sync latency p99 target?",
        "expected_keywords": ["500ms"],
        "expected_source": "architecture.md",
    },
    {
        "question": "What technology does the Sync Engine use for conflict resolution?",
        "expected_keywords": ["CRDT", "Conflict-free Replicated Data Types"],
        "expected_source": "architecture.md",
    },
    {
        "question": "What causes error code E-4012?",
        "expected_keywords": ["sync agent", "cannot reach", "servers"],
        "expected_source": "product_guide.md",
    },
    {
        "question": "What is the RPO for disaster recovery?",
        "expected_keywords": ["0 seconds"],
        "expected_source": "architecture.md",
    },
    {
        "question": "What programming language is the Sync Engine written in?",
        "expected_keywords": ["Rust"],
        "expected_source": "architecture.md",
    },
    {
        "question": "How long is version history retained on the Standard plan?",
        "expected_keywords": ["90 days"],
        "expected_source": "product_guide.md",
    },
    {
        "question": "What message broker does CloudSync use?",
        "expected_keywords": ["Kafka"],
        "expected_source": "architecture.md",
    },
    {
        "question": "What is the price of the Standard plan?",
        "expected_keywords": ["$24", "user/month"],
        "expected_source": "product_guide.md",
    },
]


def evaluate_retrieval_precision(engine):
    """Measure whether the correct source document is retrieved."""
    correct = 0
    total = len(EVAL_DATASET)

    for item in EVAL_DATASET:
        response = engine.query(item["question"])
        source_files = [
            node.node.metadata.get("file_name", "")
            for node in response.source_nodes
        ]
        if item["expected_source"] in source_files:
            correct += 1
            status = "\u2713"
        else:
            status = "\u2717"
        print(f"  {status} '{item['question'][:50]}...' -> sources: {source_files}")

    precision = correct / total
    print(f"\nRetrieval Precision: {correct}/{total} = {precision:.1%}")
    return precision


def evaluate_answer_faithfulness(engine):
    """Check if answers contain expected keywords from the source documents."""
    faithful = 0
    total = len(EVAL_DATASET)

    for item in EVAL_DATASET:
        response = engine.query(item["question"])
        answer = response.response.lower()
        has_keyword = any(
            kw.lower() in answer for kw in item["expected_keywords"]
        )
        if has_keyword:
            faithful += 1
            status = "\u2713"
        else:
            status = "\u2717"
        print(f"  {status} '{item['question'][:50]}...'")
        if not has_keyword:
            print(f"      Expected one of: {item['expected_keywords']}")
            print(f"      Got: {answer[:100]}...")

    faithfulness = faithful / total
    print(f"\nAnswer Faithfulness: {faithful}/{total} = {faithfulness:.1%}")
    return faithfulness


def run_evaluation():
    print("Building query engine...")
    engine = build_query_engine()

    print("\n" + "=" * 60)
    print("RETRIEVAL PRECISION")
    print("=" * 60)
    precision = evaluate_retrieval_precision(engine)

    print("\n" + "=" * 60)
    print("ANSWER FAITHFULNESS")
    print("=" * 60)
    faithfulness = evaluate_answer_faithfulness(engine)

    print("\n" + "=" * 60)
    print("SUMMARY")
    print("=" * 60)
    print(f"  Retrieval Precision: {precision:.1%}")
    print(f"  Answer Faithfulness: {faithfulness:.1%}")

    if precision >= 0.8 and faithfulness >= 0.8:
        print("\n  \u2705 Pipeline meets production quality bar (>80% on both metrics)")
    else:
        print("\n  \u26a0\ufe0f Pipeline needs improvement. Consider:")
        if precision < 0.8:
            print("    - Adjusting chunk size or overlap")
            print("    - Adding metadata filters")
        if faithfulness < 0.8:
            print("    - Strengthening the system prompt")
            print("    - Increasing similarity_top_k")


if __name__ == "__main__":
    run_evaluation()

Secret mission

Add Hybrid Search with BM25

Pure semantic search fails on keyword-heavy queries like exact error codes, product names, and version numbers. In this extension, you'll add BM25 keyword search alongside your dense vector retriever, fuse the results with Reciprocal Rank Fusion (RRF), and prove the improvement with your evaluation dataset.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. Your Pinecone index is on the free Starter plan ($0/month), so there are no ongoing costs for leaving it running.

Resources you used:

  • Pinecone serverless index (rag-production) on the free Starter plan.
  • Local project folder with Python files (config.py, ingest.py, query_engine.py, evaluate.py, query_engine_hybrid.py) and sample data.
  • Python virtual environment with installed packages.

Keep everything running

No action needed. Choose this if you're still actively experimenting with your RAG pipeline or want to query your documents later.

  • Your Pinecone index stays on the free Starter plan with no charges.
  • Your local files remain on your machine for future use.
  • Your virtual environment stays ready if you want to run queries again.

Pause - I'll come back to this later

Serverless indexes on the Pinecone Starter plan have no idle cost, so there is nothing to pause. Your index stays free whether you use it or not.

  • Deactivate your Python virtual environment by running:
  • No other action is needed. Your Pinecone index remains available at $0/month whenever you return.
deactivate

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow each step below to fully clean up.

  • Delete your Pinecone index. Open the Pinecone console, click the three dots to the right of the rag-production index name, then click Delete.
  • Alternatively, delete the index with Python by running this script:
from pinecone import Pinecone
from config import PINECONE_API_KEY

pc = Pinecone(api_key=PINECONE_API_KEY)
pc.delete_index(name="rag-production")
  • Delete your local project folder. On macOS/Linux, navigate to your Desktop and run:
cd ~/Desktop
rm -rf rag-production
  • On Windows, open PowerShell and run:
cd ~\Desktop
Remove-Item -Recurse -Force rag-production
  • Unset your environment variables for the current terminal session:
unset OPENAI_API_KEY
unset PINECONE_API_KEY

What about my OpenAI and Pinecone accounts?

Your accounts are free to keep. The Pinecone Starter plan costs $0/month, and OpenAI only charges for API calls you make. No ongoing charges accumulate if you stop using them.

Nice Work!

Nice Work!

You just built a production-ready RAG system that demonstrates the exact architecture companies use to give LLMs access to private data. This is not a toy demo. It is the real pipeline pattern behind AI-powered search at scale.

You've learned how to:

  • Ingest documents with intelligent chunking and embed them into a managed Pinecone vector database that persists between sessions.
  • Build a query engine with reranking that retrieves relevant context and generates grounded answers with source citations.
  • Create an evaluation harness measuring retrieval precision and answer faithfulness to catch regressions before they reach users.
  • Added hybrid search combining BM25 keyword matching with dense vectors using Reciprocal Rank Fusion for keyword-sensitive queries.

Ready to quiz yourself?