AI Docker Support Agent on Azure

Introduction

30 Second Summary

You ask an AI chatbot a technical question. It sounds confident. But the answer is wrong. Without a way to verify against real documentation, you have no idea which parts to trust.

In this project, you will build a command-line AI agent that answers Docker questions using real documentation instead of guessing. It retrieves relevant passages, cites its sources, and refuses to answer when it does not have the information.

What You'll Build

You will run your Docker support agent, ask it a tricky version-specific question, and watch it pull the exact paragraph from official docs, synthesize a grounded answer, and print the citation right there in your terminal.

By the end of this project, you'll have:

• A working RAG pipeline that retrieves real Docker documentation from Azure AI Search and generates cited answers via Gemini.
• A side-by-side comparison showing the AI hallucinating without docs versus giving accurate, sourced answers with your knowledge base connected.
• A production guardrail using a tuned relevance score threshold, validated against a 20-question eval set that proves which questions the agent answers and which it correctly refuses.
• Secret Mission: Deploy the agent as a live web API on Azure App Service so anyone can query it over HTTP.

Are there any prerequisites?

You need Python installed, VS Code installed, an active Azure for Students subscription, and basic Python knowledge. Step 1 walks you through everything else.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters for your goals.

Set Up Your Environment and Azure Resources

Before you can build a RAG-powered agent, you need three foundations in place: a Python environment with the right packages, Azure cloud infrastructure to host your search index, and a Gemini API key for embeddings and generation.

Without these pieces, nothing else in the project works. This step gets you from zero to a fully wired development environment with live cloud resources.

In this step, get ready to:

• Install Azure CLI and Python dependencies.
• Create Azure resources (Resource Group, AI Search, Storage Account).
• Get a Gemini API key and configure your environment variables.

Install Azure CLI and Python dependencies

The Azure CLI lets you create and manage Azure resources directly from your terminal. You also need Python packages for talking to the Gemini API and Azure AI Search.

• Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
• Type Terminal (macOS) or PowerShell (Windows) and press Enter to open it.

• Check whether the Azure CLI is already installed by running:

az version

✔️ I see a version number

The Azure CLI is ready. Move on to creating your project folder below.

ⓧ Command not found

Install the Azure CLI for your operating system.

macOS

• Install Azure CLI with Homebrew by running:

brew update && brew install azure-cli

Windows

• Install Azure CLI with WinGet by running:

winget install --exact --id Microsoft.AzureCLI

• Close and reopen your terminal after installation.
• Run az version again to confirm it works.

• Move to your Desktop by running this command:

cd ~/Desktop

• Create your project folder and move into it by running these commands:

mkdir docker-agent
cd docker-agent

• Confirm the folder exists by running:

ls

You should see an empty listing (no files yet) confirming you are inside your new docker-agent folder.

• Create and activate a Python virtual environment by running:

macOS

python3 -m venv venv
source venv/bin/activate

Windows

python -m venv venv
venv\Scripts\activate

Your terminal prompt now shows (venv) at the beginning, confirming the virtual environment is active.

• Open this folder in VS Code by running:

code .

VS Code opens with the docker-agent folder loaded as your workspace.

code command not found?

On macOS, open VS Code manually, press Cmd+Shift+P, type shell command, and run Shell Command: Install 'code' command in PATH. Then restart your terminal and try again.

On Windows, reinstall VS Code and ensure Add to PATH is checked during installation.

help me get the code command working in my terminal

• In VS Code, create a new file called requirements.txt by clicking File then New File.
• When prompted for a filename, type requirements.txt and press Enter.
• Paste the following content into the file:

google-genai==2.16.0
azure-search-documents==12.0.0
azure-identity==1.21.0
flask==3.1.1
python-dotenv==1.1.0

What are these packages?

google-genai is the official Gemini SDK for embeddings and text generation. azure-search-documents talks to Azure AI Search. python-dotenv loads your credentials from a .env file so you never hardcode secrets.

flask is included for the Secret Mission later. azure-identity supports Azure authentication patterns.

• Save the file with Cmd+S (macOS) or Ctrl+S (Windows).
• Back in your terminal (make sure you still see (venv)), install the packages by running:

pip install -r requirements.txt

You should see output showing each package downloading and installing. The final line reads something like Successfully installed followed by the package names.

Seeing install errors?

Make sure your virtual environment is active (you see (venv) in your prompt). If not, re-run the activation command from the platform tabs above.

help me fix a pip install error in my Python virtual environment

Create Azure resources

You need three Azure resources: a Resource Group to organize everything, an Azure AI Search service for vector search, and a Storage Account for blob storage.

• Log into Azure CLI by running:

az login

A browser window opens for you to sign in with your Azure for Students account. After signing in, the terminal displays your subscription details.

What is az login doing?

This command authenticates your local terminal with Azure so you can create and manage cloud resources from the command line. Your Azure for Students subscription gives you $100 in credit with no credit card required.

• Create a Resource Group in Central India by running:

az group create --name docker-agent-rg --location centralindia

You should see JSON output with "provisioningState": "Succeeded" confirming the resource group was created.

What is a Resource Group?

A Resource Group is a container that keeps all your related Azure resources together. When you are done with this project, you can delete the entire group in one command to clean up everything at once.

• Create the Azure AI Search service on the Free tier by running:

az search service create --name [[SEARCH_SERVICE_NAME="your-search-service-name"]] --resource-group docker-agent-rg --sku free --location centralindia

Why do I need a unique service name?

The search service name becomes part of its public URL (https://your-name.search.windows.net). It must be globally unique across all of Azure, use only lowercase letters, digits, and dashes, and be between 2 and 60 characters.

Try something like dockeragent-yourname where you substitute your actual name.

This command takes 1-2 minutes to complete. You should see JSON output with "provisioningState": "succeeded" when done.

• Create a Storage Account by running:

az storage account create --name [[STORAGE_ACCOUNT_NAME="your-storage-account-name"]] --resource-group docker-agent-rg --location centralindia --sku Standard_LRS

You should see JSON output confirming the storage account was provisioned successfully.

What are these resources for?

The AI Search service stores and queries your vector embeddings. The Storage Account holds the raw document files. Both are on free or minimal-cost tiers, so they cost effectively $0 within your student credit.

• Retrieve your AI Search admin key by running:

az search admin-key show --resource-group docker-agent-rg --service-name [[SEARCH_SERVICE_NAME="your-search-service-name"]]

Copy the primaryKey value from the output. You need it for your environment file in the next substep.

Get Gemini API key and configure environment variables

The Gemini API handles both embeddings and text generation for your agent. The free tier does not require a credit card.

• Open your browser and go to Google AI Studio.
• Sign in with your Google account.
• Click Get API Key in the left sidebar.
• Click Create API Key.
• Copy the key that appears. You only see it once, so paste it somewhere safe temporarily.

Is this free?

Yes. The Gemini API free tier does not require a credit card. Charges only apply if you explicitly click Set up billing in the Google Cloud console, which you will not do in this project.

• Switch back to VS Code.
• Create a new file called .env by clicking File then New File.
• Type .env as the filename and press Enter.
• Paste the following content into the file:

GEMINI_API_KEY=your-gemini-api-key-here
AZURE_SEARCH_ENDPOINT=https://your-search-service.search.windows.net
AZURE_SEARCH_KEY=your-search-admin-key-here
AZURE_SEARCH_INDEX=docker-docs-index

• Replace your-gemini-api-key-here with the API key you just copied from Google AI Studio.
• Replace your-search-service in the endpoint URL with the name you chose for your search service.
• Replace your-search-admin-key-here with the primaryKey value from the admin-key command.
• Save the file with Cmd+S (macOS) or Ctrl+S (Windows).

Never commit .env to version control

Your .env file contains credentials. If you use Git later, add .env to your .gitignore immediately.

Before you run the verification, what do you expect to see if the Gemini SDK imported correctly?

• Verify your Python setup by running this in your terminal:

python -c "from google import genai; print('Gemini SDK ready')"

You should see Gemini SDK ready printed in your terminal.

• Verify your Azure resources by running:

az group show --name docker-agent-rg

You should see JSON output showing your resource group details including "location": "centralindia" and "provisioningState": "Succeeded".

Gemini SDK not importing?

Make sure you are running from within the virtual environment (check for (venv) in your prompt). Also confirm you are in the docker-agent folder where you ran pip install.

help me fix my Gemini SDK import error

✔️ Awesome, I've got everything!

Great. Double check both files are saved in your docker-agent folder.

ⓧ I'd like to double check the full code

Here is what your requirements.txt should look like:

google-genai==2.16.0
azure-search-documents==12.0.0
azure-identity==1.21.0
flask==3.1.1
python-dotenv==1.1.0

And your .env file should look like this (with your real values substituted):

GEMINI_API_KEY=your-gemini-api-key-here
AZURE_SEARCH_ENDPOINT=https://your-search-service.search.windows.net
AZURE_SEARCH_KEY=your-search-admin-key-here
AZURE_SEARCH_INDEX=docker-docs-index

Your environment is locked and loaded. Next up, you will ask Gemini raw Docker questions without any knowledge base and see firsthand why RAG is necessary.

Expose the Hallucination Problem

Your Azure infrastructure is ready and your Gemini API key is configured. Before building anything complex, you need to answer a critical question: why bother with a RAG pipeline at all?

Large language models are trained on massive datasets, but they don't always get the details right. Version-specific Docker behavior, deprecated flags, and subtle platform differences are exactly the territory where hallucinations sound plausible. In this step, you'll ask Gemini targeted questions and see it produce confident answers you can't verify without a source.

In this step, get ready to:

• Write a script that asks Gemini targeted Docker questions without any knowledge base.
• Run the script and identify confidently wrong or unverifiable answers.
• Save the output as your "before" baseline for comparison.

Write the naive chatbot script

You'll create a script that sends Docker questions directly to Gemini with no grounding, no context, and no knowledge base. The questions target areas where hallucination is most likely: Compose V1-to-V2 migration details, deprecated flags, and Docker Desktop vs Engine networking.

• Create a new file called naive_chatbot.py in your project folder by running:

touch naive_chatbot.py

• Open naive_chatbot.py in VS Code and paste the following setup code:

import os
from dotenv import load_dotenv
from google import genai

# Load environment variables from .env
load_dotenv()

# Initialize Gemini client with your API key
client = genai.Client(api_key=os.getenv("GEMINI_API_KEY"))

# Targeted questions where hallucination is plausible
questions = [
    "In Docker Compose V2, what happened to the 'links' directive? Is it still supported and what changed?",
    "What is the difference between Docker Desktop networking and Docker Engine networking on Linux? Can containers access host services the same way?",
    "What does the --memory-swap flag do when set to -1 versus when set to 0 in Docker run?",
    "In Compose V2, what is the correct syntax for depends_on with a health check condition? Did the format change from V1?",
    "Is the docker-compose command (hyphenated) still supported or was it removed in Compose V2?",
]

What does this code do?

This sets up the Gemini client and defines 5 Docker questions chosen specifically because they target version-specific nuances. These are "fair fight" questions where a wrong answer sounds completely plausible.

For example, the links directive question targets Compose V2 migration details that changed across versions. A model that hasn't been grounded in the actual docs might confidently state the wrong migration path.

• Now add the main loop that sends each question to Gemini and saves the results. Paste this below the questions list:

print("=" * 60)
print("NAIVE CHATBOT - No knowledge base, no grounding")
print("=" * 60)

# Store results for saving to file
results = []
for question in questions:
    print(f"\nQ: {question}")
    # Ask Gemini directly with no context or grounding
    response = client.models.generate_content(
        model="gemini-2.0-flash",
        contents=question,
    )
    answer = response.text
    # Show first 600 chars of each answer
    print(f"A: {answer[:600]}")
    print("-" * 60)
    results.append(f"Q: {question}\nA: {answer}\n{'=' * 60}\n")

from pathlib import Path

# Save all responses as the "before" baseline
Path("baseline_responses.txt").write_text("\n".join(results), encoding="utf-8")
print("\nBaseline saved to baseline_responses.txt")

What does this code do?

The loop iterates through each question, sends it directly to gemini-2.0-flash with no system prompt or context documents, and prints the first 600 characters of each response.

After all 5 questions are answered, the script writes the full (untruncated) responses to baseline_responses.txt. This file becomes your "before" comparison when you build the RAG pipeline later.

• Save naive_chatbot.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before running it.

ⓧ I'd like to double check the full code

Your naive_chatbot.py should contain two sections in order:

• The import and setup block: import os, load_dotenv(), client initialization, and the questions list with all 5 Docker questions.
• The main loop block: the header print statements, the for question in questions loop calling client.models.generate_content, and the final Path("baseline_responses.txt").write_text(...) save.

Scroll up and confirm both sections are present and in order.

Seeing an import error?

Make sure your virtual environment is still active. You should see (venv) or similar at the start of your terminal prompt. If not, reactivate it.

help me fix an import error in my naive_chatbot.py script

Run the script and spot the hallucinations

This is the moment that justifies the entire RAG pipeline you'll build. You're about to see Gemini answer Docker questions with zero grounding in actual documentation.

• Before you run the script, think about this: do you expect Gemini to get these Docker version-specific details right, or will it produce plausible-sounding answers that might be wrong?
• Run the naive chatbot by executing:

python naive_chatbot.py

You'll see Gemini produce detailed, confident-sounding answers to all 5 questions. The answers read like they come from an expert.

Here's the problem: some of those answers are confidently wrong. Look for invented flag behaviors, incorrect version numbers, or outdated syntax presented as current.

What should I look for?

Some answers may be partially correct. That is exactly the point. The problem is not that the model is always wrong. The problem is that you cannot tell when it is wrong without checking against a source.

• Does it claim the links directive was "removed" when it was actually just deprecated and still functional?
• Does it invent specific version numbers for when changes happened?
• Does it describe --memory-swap behavior that doesn't match the actual Docker documentation?

• Confirm baseline_responses.txt was created by running:

ls baseline_responses.txt

You should see baseline_responses.txt listed. This file contains the full untruncated responses and serves as your "before" baseline for the demo later.

Script hanging or timing out?

The Gemini free tier has rate limits. If the script hangs on a question, wait 30 seconds and try again. Each question makes one API call, so all 5 should complete within a minute.

help me debug my naive_chatbot.py script that is hanging or timing out

You've just proven the hallucination problem firsthand. Gemini produces fluent, confident answers, but without a source you can't trust them on version-specific Docker details. Next up, you'll build the knowledge base that grounds these answers in real documentation.

Build the Knowledge Base

You saw firsthand that Gemini confidently produces wrong answers about version-specific Docker behavior. The root cause is simple: the model has no access to authoritative documentation at query time.

This step builds the retrieval layer that fixes that problem. You will chunk real Docker documentation into searchable passages, convert them into vector embeddings, push them into Azure AI Search, and verify that retrieval actually works before wiring it to generation.

In this step, get ready to:

• Chunk Docker documentation into ~500-token passages with metadata.
• Embed and index all chunks in Azure AI Search using gemini-embedding-2.
• Verify retrieval quality independently of generation.

Prepare Docker documentation

The retrieval layer needs real documentation to search through. You will collect content from Docker's official docs and save it as plain text files that your chunking script can process.

• Create a docs/ directory inside your project folder by running this command:

mkdir docs

• Confirm the folder exists by running:

ls

You should see docs listed alongside your existing project files.

Now you need to populate that folder with real Docker documentation content. You will create 4 text files, each containing content copied from Docker's official docs.

• Visit each of the following Docker documentation pages in your browser:
• Dockerfile reference: https://docs.docker.com/reference/dockerfile
• Compose networking: https://docs.docker.com/compose/how-tos/networking
• Multi-stage builds: https://docs.docker.com/build/building/multi-stage/
• Docker Engine troubleshooting: https://docs.docker.com/engine/daemon/troubleshoot/
• For each page, select all the main text content (skip navigation menus and sidebars) and copy it.
• Paste each page's content into a new text file inside the docs/ folder with these exact names:
• dockerfile-reference.txt
• compose-networking.txt
• multi-stage-builds.txt
• common-errors.txt

• Verify all 4 files are in place by running:

ls docs/

You should see common-errors.txt, compose-networking.txt, dockerfile-reference.txt, and multi-stage-builds.txt listed.

How much content is enough?

Aim for at least 2-3 full sections per file. The more substantive text you include, the better the retrieval results will be. A few paragraphs won't give the chunking algorithm enough material to work with.

Now you need a script to split these documents into searchable passages. Large documents are too big to feed directly into an embedding model, so you will chunk them into overlapping ~500-word passages with metadata tracking which file each chunk came from.

• Create a new file called prepare_docs.py in your project root and add the imports and chunking function:

import json
from pathlib import Path


def chunk_document(file_path, chunk_size=500, overlap=50):
    """Split a document into overlapping chunks with metadata."""
    text = Path(file_path).read_text(encoding="utf-8")
    file_name = Path(file_path).name
    chunks = []
    words = text.split()

    for i in range(0, len(words), chunk_size - overlap):
        chunk_words = words[i : i + chunk_size]
        if len(chunk_words) < 50:
            continue
        chunk_text = " ".join(chunk_words)
        chunks.append(
            {
                "id": f"{file_name}-chunk-{len(chunks)}",
                "content": chunk_text,
                "source_file": file_name,
                "chunk_index": len(chunks),
            }
        )

    return chunks

What does this code do?

• chunk_document() reads a text file and splits it into overlapping passages of ~500 words each.
• The overlap=50 parameter means each chunk shares 50 words with the previous one, preventing ideas from being cut off mid-sentence at chunk boundaries.
• Chunks smaller than 50 words are discarded to avoid indexing fragments that are too short to be useful.
• Each chunk stores its source_file so the agent can cite where an answer came from.

• Below the chunk_document function, add the orchestration function and main block:

def prepare_all_docs(docs_dir="docs"):
    """Process all text files in the docs directory."""
    all_chunks = []
    docs_path = Path(docs_dir)

    for file_path in docs_path.glob("*.txt"):
        print(f"Processing: {file_path.name}")
        chunks = chunk_document(file_path)
        all_chunks.extend(chunks)
        print(f"  Generated {len(chunks)} chunks")

    output_path = Path("chunks.json")
    output_path.write_text(json.dumps(all_chunks, indent=2), encoding="utf-8")
    print(f"\nTotal chunks: {len(all_chunks)}")
    print(f"Saved to: {output_path}")
    return all_chunks


if __name__ == "__main__":
    prepare_all_docs()

What does this code do?

• prepare_all_docs() finds every .txt file in the docs/ directory and chunks each one.
• All chunks across all files are collected into a single list and written to chunks.json for the indexing script to consume.
• The progress output tells you how many chunks each document produced, so you can spot files that might be too short.

• Save prepare_docs.py.
• Run the chunking script:

python prepare_docs.py

You should see output listing each file processed and the number of chunks generated, followed by a total count and a confirmation that chunks.json was saved.

• Confirm chunks.json was created by running:

ls chunks.json

You should see chunks.json listed in the output.

Seeing 'Total chunks: 0'?

• Make sure your .txt files are inside the docs/ folder, not in the project root.
• Verify each file has substantial text content (at least 50 words) by opening them in VS Code.
• Check the file extension is .txt and not .md or another format.

help me debug why prepare_docs.py is producing zero chunks

✔️ Awesome, I've got everything!

Great. Double check you have saved prepare_docs.py and that chunks.json exists in your project root.

ⓧ I'd like to double check the full code

Your prepare_docs.py should have two sections built above, in this order:

• Imports (json, Path) plus the chunk_document() function.
• The prepare_all_docs() function plus the if __name__ == "__main__" block at the bottom.

Scroll up and confirm each chunk is present and in order.

Generate embeddings and index

You have chunked documents sitting in chunks.json. Now you need to convert each chunk into a vector embedding and upload them to Azure AI Search so the agent can perform semantic search at query time.

What are vector embeddings?

A vector embedding is a list of numbers (768 numbers in this project) that represents the meaning of a text passage. Passages about similar topics produce similar vectors.

When a user asks a question, you embed the question too, then find the document chunks whose vectors are closest to the question's vector. This is how semantic search works.

• Create a new file called index_docs.py and add the imports and client configuration:

import os
import json
import time
from pathlib import Path
from dotenv import load_dotenv
from google import genai
from google.genai import types
from azure.search.documents import SearchClient
from azure.search.documents.indexes import SearchIndexClient
from azure.search.documents.indexes.models import (
    SearchIndex,
    SearchField,
    SearchFieldDataType,
    VectorSearch,
    HnswAlgorithmConfiguration,
    VectorSearchProfile,
    SearchableField,
    SimpleField,
)
from azure.core.credentials import AzureKeyCredential

load_dotenv()

gemini_client = genai.Client(api_key=os.getenv("GEMINI_API_KEY"))

search_endpoint = os.getenv("AZURE_SEARCH_ENDPOINT")
search_key = os.getenv("AZURE_SEARCH_KEY")
index_name = os.getenv("AZURE_SEARCH_INDEX")
credential = AzureKeyCredential(search_key)

What does this code do?

• This sets up two clients: one for Gemini (to generate embeddings) and one for Azure AI Search (to store and query them).
• All credentials are loaded from your .env file via load_dotenv(), keeping secrets out of your code.
• The SearchIndexClient manages index schemas, while SearchClient handles document uploads and queries.

• Below the client setup, add the function that defines and creates the search index schema:

def create_search_index():
    """Create the Azure AI Search index with vector fields."""
    index_client = SearchIndexClient(search_endpoint, credential)
    fields = [
        SimpleField(name="id", type=SearchFieldDataType.String, key=True),
        SearchableField(name="content", type=SearchFieldDataType.String),
        SimpleField(
            name="source_file", type=SearchFieldDataType.String, filterable=True
        ),
        SimpleField(
            name="chunk_index", type=SearchFieldDataType.Int32, filterable=True
        ),
        SearchField(
            name="content_vector",
            type=SearchFieldDataType.Collection(SearchFieldDataType.Single),
            searchable=True,
            vector_search_dimensions=768,
            vector_search_profile_name="default-profile",
        ),
    ]
    vector_search = VectorSearch(
        algorithms=[HnswAlgorithmConfiguration(name="default-algorithm")],
        profiles=[VectorSearchProfile(
            name="default-profile", algorithm_configuration_name="default-algorithm"
        )],
    )
    index = SearchIndex(name=index_name, fields=fields, vector_search=vector_search)
    index_client.create_or_update_index(index)
    print(f"Index '{index_name}' created successfully.")

What does this code do?

• This defines the schema for your Azure AI Search index with five fields: a unique id, the content text, source_file metadata, a chunk_index, and a 768-dimensional content_vector for semantic search.
• The HnswAlgorithmConfiguration sets up an approximate nearest-neighbor algorithm that makes vector search fast even with many documents.
• create_or_update_index() creates the index if it does not exist, or updates the schema if it does.

• Save index_docs.py.
• Test that the index creation works by running:

python -c "import index_docs; index_docs.create_search_index()"

You should see: Index 'docker-docs-index' created successfully.

Seeing an authentication error?

• Verify your AZURE_SEARCH_ENDPOINT in .env starts with https:// and ends with .search.windows.net.
• Verify your AZURE_SEARCH_KEY is the admin key (not the query key) from the Azure portal under your Search service's **Keys** section.
• Make sure there are no extra spaces or quotes around the values in your .env file.

help me fix an authentication error when connecting to Azure AI Search

• Now add the embedding helper function below create_search_index:

def generate_embedding(text):
    """Generate embedding for a text chunk using Gemini."""
    result = gemini_client.models.embed_content(
        model="gemini-embedding-2",
        contents=text,
        config=types.EmbedContentConfig(output_dimensionality=768),
    )
    return result.embeddings[0].values

What does this code do?

This sends a text string to gemini-embedding-2 and gets back a list of 768 numbers representing that text's meaning. The output_dimensionality=768 matches the 768-dimension vector field you defined in the index schema.

• Save index_docs.py.
• Verify the embedding function works by running:

python -c "import index_docs; print(len(index_docs.generate_embedding('test')))"

You should see 768 printed, confirming the embedding model returns vectors of the correct dimension.

• Add the indexing function and main block below generate_embedding:

def index_chunks():
    """Embed all chunks and upload to Azure AI Search."""
    chunks = json.loads(Path("chunks.json").read_text(encoding="utf-8"))
    search_client = SearchClient(search_endpoint, index_name, credential)
    documents = []
    for i, chunk in enumerate(chunks):
        print(f"Embedding chunk {i + 1}/{len(chunks)}: {chunk['id']}")
        embedding = generate_embedding(chunk["content"])
        time.sleep(0.5)
        documents.append(
            {
                "id": chunk["id"],
                "content": chunk["content"],
                "source_file": chunk["source_file"],
                "chunk_index": chunk["chunk_index"],
                "content_vector": embedding,
            }
        )
        if len(documents) >= 10:
            search_client.upload_documents(documents)
            print(f"  Uploaded batch of {len(documents)} documents")
            documents = []
    if documents:
        search_client.upload_documents(documents)
        print(f"  Uploaded final batch of {len(documents)} documents")
    print(f"\nIndexing complete. Total documents indexed: {len(chunks)}")

if __name__ == "__main__":
    create_search_index()
    index_chunks()

What does this code do?

• index_chunks() reads every chunk from chunks.json, generates an embedding for each one, and uploads them in batches of 10 to Azure AI Search.
• time.sleep(0.5) adds a half-second pause between API calls to respect Gemini's free tier rate limits.
• The if __name__ block runs both functions in sequence: first creating the index schema, then embedding and uploading all chunks.

• Save index_docs.py.
• Run the full indexing script:

python index_docs.py

You should see the index creation message followed by progress lines for each chunk being embedded and uploaded. The final line reports the total number of documents indexed.

How long does this take?

With the 0.5-second delay between embeddings, indexing takes roughly 0.5 seconds per chunk. If you have 30 chunks, expect about 15-20 seconds. This is normal for the free tier rate limit.

Seeing a rate limit error from Gemini?

• Wait 60 seconds and run the script again. The free tier rate limit resets quickly.
• If errors persist, increase time.sleep(0.5) to time.sleep(1.0) in index_docs.py.
• The script is idempotent. Running it again uploads the same documents without creating duplicates.

help me fix a Gemini rate limit error during document indexing

✔️ Awesome, I've got everything!

Great. Confirm you saw the 'Indexing complete' message with a document count matching your total chunks.

ⓧ I'd like to double check the full code

Your index_docs.py should have four sections built above, in this order:

• Imports and client configuration (genai, SearchClient, SearchIndexClient, AzureKeyCredential, etc.).
• The create_search_index() function defining the index schema with 5 fields.
• The generate_embedding() helper function calling gemini-embedding-2.
• The index_chunks() function plus the if __name__ == "__main__" block calling both create_search_index() and index_chunks().

Scroll up and confirm each chunk is present and in order.

Retrieval quality check

You have embedded documents sitting in Azure AI Search, but you have not proven they actually come back when asked the right question. This verification step is critical: you test retrieval independently of generation so you know the search layer works on its own.

Why test retrieval separately?

If you wire retrieval to generation without checking, a bad answer could mean the retrieval failed OR the generation prompt was wrong. By testing retrieval first, you isolate one variable at a time.

In interviews, this lets you say: "I verified retrieval quality independently before connecting it to the LLM."

• Create a new file called test_retrieval.py and add the imports, client setup, and test queries:

import os
from dotenv import load_dotenv
from google import genai
from google.genai import types
from azure.search.documents import SearchClient
from azure.search.documents.models import VectorizedQuery
from azure.core.credentials import AzureKeyCredential

load_dotenv()

gemini_client = genai.Client(api_key=os.getenv("GEMINI_API_KEY"))

search_endpoint = os.getenv("AZURE_SEARCH_ENDPOINT")
search_key = os.getenv("AZURE_SEARCH_KEY")
index_name = os.getenv("AZURE_SEARCH_INDEX")
search_client = SearchClient(
    search_endpoint, index_name, AzureKeyCredential(search_key)
)

test_queries = [
    "How do I expose a port in a Dockerfile?",
    "What is Docker Compose networking and how do containers communicate?",
    "What changed with the links directive in Compose V2?",
]

print("RETRIEVAL QUALITY CHECK")
print("Verify: correct doc section appears in top-3 for each query\n")

What does this code do?

• This sets up the same Azure AI Search client and Gemini client used for indexing.
• The three test queries cover different difficulty levels: an easy factual question, a medium conceptual question, and one from the hallucination targets in Step 2.
• VectorizedQuery from the Azure SDK lets you pass a pre-computed vector for similarity search.

• Below the setup, add the search loop that runs each query and prints the top-3 results:

for query in test_queries:
    print(f"Query: {query}")
    print("-" * 50)

    embedding_result = gemini_client.models.embed_content(
        model="gemini-embedding-2",
        contents=query,
        config=types.EmbedContentConfig(output_dimensionality=768),
    )
    vector = embedding_result.embeddings[0].values

    vector_query = VectorizedQuery(
        vector=vector, k_nearest_neighbors=3, fields="content_vector"
    )

    results = search_client.search(
        search_text=None,
        vector_queries=[vector_query],
        select=["content", "source_file", "chunk_index"],
    )

    for i, result in enumerate(results):
        print(f"  #{i + 1} [score: {result['@search.score']:.4f}] "
              f"source: {result['source_file']}, chunk: {result['chunk_index']}")
        print(f"       preview: {result['content'][:120]}...")
        print()

    print("=" * 60)
    print()

What does this code do?

• For each test query, it generates an embedding and performs a vector search to find the 3 nearest document chunks.
• k_nearest_neighbors=3 tells Azure to return only the top 3 most similar chunks.
• Each result prints its similarity score, source file, chunk index, and a 120-character preview so you can eyeball whether the right content surfaced.

• Save test_retrieval.py.

Before you run this, what do you expect to see for the query about exposing ports? Which source file should appear in the top results?

• Run the retrieval test:

python test_retrieval.py

For each query, check that the correct documentation file appears in the top-3 results:

• "How do I expose a port in a Dockerfile?" should surface chunks from dockerfile-reference.txt.
• "What is Docker Compose networking..." should surface chunks from compose-networking.txt.
• "What changed with the links directive in Compose V2?" should surface chunks from compose-networking.txt.

Wrong source file showing up in the results?

• If irrelevant chunks rank highest, your documentation files may not have enough content about that topic. Go back and add more text from the relevant Docker doc page.
• If scores are very low (below 0.5), your chunks may be too short. Check that your doc files have substantial content, not just a paragraph or two.
• After updating docs, re-run python prepare_docs.py and then python index_docs.py to rebuild the index.

help me improve my retrieval results for Docker documentation queries

✔️ Awesome, I've got everything!

Great. Confirm you have saved test_retrieval.py and that the correct source files appear in the top results for each query.

ⓧ I'd like to double check the full code

Your test_retrieval.py should have two sections built above, in this order:

• Imports, client setup, test_queries list, and the two print() header lines.
• The for query in test_queries loop that embeds each query, searches the index, and prints results.

Scroll up and confirm each chunk is present and in order.

Your retrieval layer is proven. The correct document sections surface for relevant queries, and you have verified this independently of any generation step. Next up, you will wire this retrieval to Gemini so the agent can answer questions grounded in real documentation.

Wire Up the RAG Pipeline

In the previous step, you confirmed that retrieval returns the correct document chunks for your queries. The search index works.

Now you need a way to query the system naturally: a user types a question, relevant chunks come back from Azure AI Search, and Gemini synthesizes a grounded answer with source citations. This is the full RAG pipeline, and the "before vs after" comparison against your hallucinated baseline is your interview demo moment.

In this step, get ready to:

• Build a retrieval function that embeds a user question and searches the index.
• Build a generation function that prompts Gemini with retrieved context and requires source citations.
• Test the full pipeline end-to-end against the hallucination questions from Step 2.

Build the retrieval function

The retrieval function takes a user question, converts it into a vector embedding, and searches your Azure AI Search index for the closest matching chunks. This is the same pattern you tested in test_retrieval.py, now packaged as a reusable function.

• Create a new file called agent.py in your project folder and add the imports, client setup, and relevance threshold:

import os
from dotenv import load_dotenv
from google import genai
from google.genai import types
from azure.search.documents import SearchClient
from azure.search.documents.models import VectorizedQuery
from azure.core.credentials import AzureKeyCredential

load_dotenv()

gemini_client = genai.Client(api_key=os.getenv("GEMINI_API_KEY"))

search_endpoint = os.getenv("AZURE_SEARCH_ENDPOINT")
search_key = os.getenv("AZURE_SEARCH_KEY")
index_name = os.getenv("AZURE_SEARCH_INDEX")
search_client = SearchClient(
    search_endpoint, index_name, AzureKeyCredential(search_key)
)

RELEVANCE_THRESHOLD = 0.7

What does this code do?

• The imports pull in the Gemini SDK for embeddings and generation, plus the Azure Search SDK for vector queries.
• load_dotenv() loads your .env credentials so they are available as environment variables.
• RELEVANCE_THRESHOLD = 0.7 sets the minimum search score a chunk must have before the agent trusts it as relevant context. You will tune this value in the next step.

• Below the threshold line, add the retrieve_context function:

def retrieve_context(question, top_k=3):
    """Retrieve relevant document chunks from Azure AI Search."""
    question_embedding = gemini_client.models.embed_content(
        model="gemini-embedding-2",
        contents=question,
        config=types.EmbedContentConfig(output_dimensionality=768),
    )
    vector = question_embedding.embeddings[0].values

    vector_query = VectorizedQuery(
        vector=vector, k_nearest_neighbors=top_k, fields="content_vector"
    )

    results = search_client.search(
        search_text=None,
        vector_queries=[vector_query],
        select=["content", "source_file", "chunk_index"],
    )

    chunks = []
    for result in results:
        chunks.append(
            {
                "content": result["content"],
                "source_file": result["source_file"],
                "score": result["@search.score"],
            }
        )

    return chunks

What does this function do?

• It embeds the user's question into a 768-dimension vector using gemini-embedding-2.
• It performs a vector search with VectorizedQuery, returning the 3 nearest document chunks.
• Each returned chunk includes its text content, source filename, and similarity score. The score tells you how confident the search is that this chunk is relevant.

• Save agent.py.
• Verify the file loads without errors by running this command:

python -c "import agent; print('agent.py loaded successfully')"

You should see agent.py loaded successfully printed with no errors.

Seeing an ImportError or ModuleNotFoundError?

• Make sure your virtual environment is active. If not, activate it again.
• Check that your .env file is in the same directory as agent.py.
• If you see a package not found error, confirm you installed google-genai and azure-search-documents in your virtual environment.

Help me debug my agent.py import error.

Build the generation function

The generation function takes the retrieved chunks, formats them as context, and sends them to Gemini with a strict system prompt. The prompt forces Gemini to answer only from the provided context and cite its sources.

• In agent.py, below the retrieve_context function, add the system prompt:

SYSTEM_PROMPT = """You are a Docker support agent. You answer questions ONLY using the provided context from official Docker documentation.

Rules:
1. If the context contains relevant information, answer the question and cite your sources using [Source: filename] format.
2. If the context does NOT contain relevant information for the question, respond EXACTLY with: "I don't have enough information about that topic in my knowledge base. My expertise covers: Dockerfile syntax, Docker Compose, multi-stage builds, and common Docker troubleshooting."
3. Never make up information. Never answer from your own training data.
4. Be concise and technical. Format code examples with proper markdown.

Context:
{context}
"""

Why is this prompt so strict?

Rule 2 gives the model an exact refusal phrase. This makes it possible to programmatically detect when the agent refuses a question, which you will use in the next step to measure accuracy.

The {context} placeholder is where the retrieved document chunks will be injected at runtime.

• Below the system prompt, add the generate_response function:

def generate_response(question, context_chunks):
    """Generate a response using Gemini with the retrieved context."""
    if not context_chunks or context_chunks[0]["score"] < RELEVANCE_THRESHOLD:
        return "I don't have enough information about that topic in my knowledge base. My expertise covers: Dockerfile syntax, Docker Compose, multi-stage builds, and common Docker troubleshooting."

    context_text = ""
    for chunk in context_chunks:
        context_text += f"[Source: {chunk['source_file']}]\n{chunk['content']}\n\n"

    prompt = SYSTEM_PROMPT.format(context=context_text)

    response = gemini_client.models.generate_content(
        model="gemini-2.0-flash",
        contents=[
            types.Content(role="user", parts=[types.Part(text=question)]),
        ],
        config=types.GenerateContentConfig(
            system_instruction=prompt,
            temperature=0.2,
        ),
    )

    return response.text

What does this function do?

• The score gate at the top checks whether the best chunk is relevant enough. If the top score is below 0.7, it returns the refusal message without calling Gemini at all.
• It formats each chunk with a [Source: filename] tag so Gemini knows which doc to cite.
• Temperature 0.2 keeps the output deterministic and factual. Higher temperatures would introduce more creative (and potentially inaccurate) phrasing.

• Save agent.py.
• Verify the module still loads cleanly by running:

python -c "from agent import generate_response; print('Generation function ready')"

You should see Generation function ready with no errors.

Seeing a SyntaxError?

• The SYSTEM_PROMPT uses triple quotes. Make sure you have matching """ at both the start and end of the string.
• Check that the generate_response function is not accidentally indented inside retrieve_context. Both functions should be at the top level (no indentation).

Help me fix a syntax error in my agent.py file.

Test end-to-end with the hallucination questions

You have retrieval and generation wired together. Now you need the orchestration layer: a function that takes a question, runs it through the full pipeline, and prints the result with scores.

• In agent.py, below generate_response, add the ask function:

def ask(question):
    """Full RAG pipeline: retrieve then generate."""
    print(f"\nQ: {question}")
    print("-" * 40)

    chunks = retrieve_context(question)

    if chunks:
        print(f"Retrieved {len(chunks)} chunks (best score: {chunks[0]['score']:.3f})")
    else:
        print("No relevant chunks found.")

    answer = generate_response(question, chunks)
    print(f"\nA: {answer}")
    print("=" * 60)
    return answer

What does ask() do?

This function orchestrates the two pieces you just built. It calls retrieve_context to get chunks, then passes them to generate_response for the final answer. It also prints the best retrieval score so you can see exactly why the agent trusts or refuses a given query.

• Below the ask function, add the interactive chat loop and entry point:

def main():
    """Interactive chat loop."""
    print("Docker Support Agent")
    print("Type 'quit' to exit.\n")

    while True:
        question = input("\nYour question: ").strip()
        if question.lower() in ("quit", "exit", "q"):
            print("Goodbye!")
            break
        if not question:
            continue
        ask(question)


if __name__ == "__main__":
    main()

What does main() do?

The interactive loop lets you type questions in your terminal and get grounded answers in real time. Typing quit, exit, or q stops the loop.

• Save agent.py.

Before you run the agent, think about this: you are about to ask it the same question that Gemini hallucinated on in Step 2. Do you expect the RAG pipeline to give a different answer this time? What will be different about it?

• Run the agent by executing:

python agent.py

• When you see the Your question: prompt, type this question:

What changed in Compose V2 regarding the 'links' directive?

You should see the agent retrieve 3 chunks with a score above 0.7, then produce an answer that cites a specific [Source: compose-networking.txt] tag. The answer is grounded in the actual Docker documentation you indexed.

• Try one more question to confirm the citation pattern:

What is the difference between CMD and ENTRYPOINT?

• Type quit to exit the agent.

Now compare these answers to baseline_responses.txt from Step 2. The hallucinated version gave confident but unverifiable answers. Your RAG agent gives cited, grounded responses you can check against the source docs.

Why is this comparison important?

In interviews, you can show both outputs side by side: the hallucinated baseline and the grounded RAG answer. This demonstrates that you understand the problem (hallucination), built a solution (retrieval-augmented generation), and can verify the result (source citations).

Agent returning empty or refusing in-scope questions?

• Check the printed score. If it is below 0.7, the threshold is too strict for your index. Try lowering RELEVANCE_THRESHOLD to 0.5 temporarily and re-run.
• If you see an API error from Gemini, verify that GEMINI_API_KEY in your .env is correct and your free-tier quota has not been exceeded.
• If the search returns no results, confirm your index still has documents by re-running test_retrieval.py from the previous step.

Help me debug why my RAG agent is refusing questions that should be in scope.

✔️ Awesome, I've got everything!

Great. Double check you have saved agent.py and that your test questions produced cited answers.

ⓧ I'd like to double check the full code

Your agent.py should contain these sections in order. Scroll up and confirm each is present:

• Imports and client setup (import os through RELEVANCE_THRESHOLD = 0.7).
• SYSTEM_PROMPT triple-quoted string with 4 rules and a {context} placeholder.
• retrieve_context(question, top_k=3) function that embeds and searches.
• generate_response(question, context_chunks) function that gates on score and calls Gemini.
• ask(question) function that orchestrates retrieve + generate.
• main() interactive loop with if __name__ == "__main__" guard.

Your RAG pipeline is live and producing cited answers. Next up, you will add production guardrails so the agent correctly refuses out-of-scope questions and validate it all against a 20-question eval set.

Add Production Guardrails and Run the Eval

Your RAG pipeline delivers cited, grounded answers for Docker questions. That is a huge upgrade from the hallucinating baseline.

But what happens when someone asks about Kubernetes, Terraform, or AWS ECS? Right now, the agent retrieves whatever chunks score highest and forces an answer from irrelevant context. A production agent must know when to say "I don't know." In this step, you will validate that the relevance threshold actually works by running a systematic 20-question eval set and reporting concrete accuracy numbers.

In this step, get ready to:

• Test the agent with an out-of-scope question to see the problem firsthand.
• Write a 20-question eval set covering both in-scope and out-of-scope queries.
• Build and run an evaluation script that reports pass/fail per question and prints accuracy numbers.

Test the agent with an out-of-scope question

Your agent.py already has a RELEVANCE_THRESHOLD = 0.7 check that gates generation. If the top retrieval score falls below that value, the agent returns a refusal instead of forcing an answer. But does that threshold actually catch every out-of-scope question?

• Run the agent interactively to test it with a question outside your knowledge base:

python agent.py

• When the prompt appears, type this question:

How do I set up a Linux bridge network manually?

Before you press Enter, predict: will the agent refuse or attempt an answer? The Docker networking docs mention bridge networks, so the retrieval might score this above 0.7 even though the question is about general Linux networking, not Docker.

• Press Enter and observe the output.
• Note the score printed in the output (e.g., best score: 0.XXX).
• Type quit and press Enter to exit.

Why a single test is not enough

Whether the agent refused or answered, a single query proves nothing about the system as a whole. Maybe this one question was caught, but five others leak through. The only way to know your threshold works is to run it against a structured eval set with known expected outcomes.

This is the difference between "vibes-based" testing and production-grade evaluation. In an interview, you can say "I validated against 20 questions and got X% accuracy" instead of "I tried a couple and it seemed fine."

Create the evaluation question set

A proper eval needs questions with known expected outcomes. You will create 10 in-scope questions (topics your docs cover) and 10 out-of-scope questions (topics the agent should refuse).

• Create a new file called eval_questions.json in your project folder and paste this content:

{
  "in_scope": [
    {"question": "How do I expose a port in a Dockerfile?", "expected_source": "dockerfile-reference.txt"},
    {"question": "What is the difference between CMD and ENTRYPOINT?", "expected_source": "dockerfile-reference.txt"},
    {"question": "How do containers communicate in Docker Compose?", "expected_source": "compose-networking.txt"},
    {"question": "What is a multi-stage build and why would I use one?", "expected_source": "multi-stage-builds.txt"},
    {"question": "How do I set environment variables in a Dockerfile?", "expected_source": "dockerfile-reference.txt"},
    {"question": "What is the default network driver in Docker Compose?", "expected_source": "compose-networking.txt"},
    {"question": "How do I copy files into a Docker image?", "expected_source": "dockerfile-reference.txt"},
    {"question": "What changed with links in Compose V2?", "expected_source": "compose-networking.txt"},
    {"question": "How do I reduce Docker image size?", "expected_source": "multi-stage-builds.txt"},
    {"question": "What is the WORKDIR instruction?", "expected_source": "dockerfile-reference.txt"}
  ],
  "out_of_scope": [
    {"question": "How do I configure Kubernetes ingress?"},
    {"question": "How do I deploy containers to AWS ECS?"},
    {"question": "What is the best Terraform module for Docker?"},
    {"question": "How do I write an Ansible playbook for container deployment?"},
    {"question": "How do I set up a Linux bridge network manually?"},
    {"question": "How do I debug a Python ImportError?"},
    {"question": "What is the best CI/CD pipeline for GitHub Actions?"},
    {"question": "How do I configure Nginx as a reverse proxy?"},
    {"question": "What is the difference between Azure AKS and ACI?"},
    {"question": "How do I install Node.js on Ubuntu?"}
  ]
}

What does this eval set cover?

• The in_scope questions target topics you know your docs cover: Dockerfile syntax, Compose networking, and multi-stage builds. Each includes the expected source file so you can verify the citation is correct.
• The out_of_scope questions cover technologies your knowledge base knows nothing about: Kubernetes, AWS ECS, Terraform, Ansible, general Linux networking, Python debugging, CI/CD, Nginx, Azure AKS/ACI, and Node.js. The expected result for all of these is refusal.

• Save eval_questions.json.
• Confirm the file exists by running:

ls eval_questions.json

You should see eval_questions.json listed in the output.

Build and run the evaluation script

Now you need a script that runs all 20 questions through the pipeline, checks whether each result matches the expected outcome, and prints a summary. This is the script you will show in interviews to prove your system works.

• Create a new file called run_eval.py in your project folder.
• Add the imports and client setup by pasting this code:

import os
import json
from pathlib import Path
from dotenv import load_dotenv
from google import genai
from google.genai import types
from azure.search.documents import SearchClient
from azure.search.documents.models import VectorizedQuery
from azure.core.credentials import AzureKeyCredential

load_dotenv()

gemini_client = genai.Client(api_key=os.getenv("GEMINI_API_KEY"))

search_endpoint = os.getenv("AZURE_SEARCH_ENDPOINT")
search_key = os.getenv("AZURE_SEARCH_KEY")
index_name = os.getenv("AZURE_SEARCH_INDEX")
search_client = SearchClient(
    search_endpoint, index_name, AzureKeyCredential(search_key)
)

What does this code do?

This is the same client setup pattern you used in agent.py. The eval script needs its own connection to both Gemini (for embeddings) and Azure AI Search (for retrieval) because it runs independently of the interactive agent.

• Next, add the constants and system prompt below your client setup:

RELEVANCE_THRESHOLD = 0.7

REFUSAL_PREFIX = "I don't have enough information"

SYSTEM_PROMPT = """You are a Docker support agent. You answer questions ONLY using the provided context from official Docker documentation.

Rules:
1. If the context contains relevant information, answer the question and cite your sources using [Source: filename] format.
2. If the context does NOT contain relevant information for the question, respond EXACTLY with: "I don't have enough information about that topic in my knowledge base. My expertise covers: Dockerfile syntax, Docker Compose, multi-stage builds, and common Docker troubleshooting."
3. Never make up information. Never answer from your own training data.
4. Be concise and technical. Format code examples with proper markdown.

Context:
{context}
"""

What does this code do?

• RELEVANCE_THRESHOLD is your initial threshold value (0.7). This is the value you will tune based on eval results.
• REFUSAL_PREFIX is the string the eval checks for to determine whether the agent refused a question. If the response starts with this prefix, the eval counts it as a refusal.
• SYSTEM_PROMPT is identical to the one in agent.py so the eval tests the exact same behavior.

• Add the first half of the retrieve_and_generate function below the constants. This handles embedding the question and retrieving chunks:

def retrieve_and_generate(question):
    """Full RAG pipeline returning answer, score, and sources."""
    embedding_result = gemini_client.models.embed_content(
        model="gemini-embedding-2",
        contents=question,
        config=types.EmbedContentConfig(output_dimensionality=768),
    )
    vector = embedding_result.embeddings[0].values

    vector_query = VectorizedQuery(
        vector=vector, k_nearest_neighbors=3, fields="content_vector"
    )

    results = search_client.search(
        search_text=None,
        vector_queries=[vector_query],
        select=["content", "source_file", "chunk_index"],
    )

    chunks = []
    for result in results:
        chunks.append(
            {
                "content": result["content"],
                "source_file": result["source_file"],
                "score": result["@search.score"],
            }
        )

    top_score = chunks[0]["score"] if chunks else 0.0

What does this code do?

This is the retrieval half of the pipeline. It embeds the question using gemini-embedding-2, runs a vector search against your Azure AI Search index, and collects the top 3 chunks with their scores. The top_score variable captures the highest-scoring chunk's relevance, which is what the threshold gate uses.

• Continue the retrieve_and_generate function by adding the threshold gate and generation logic directly below the last line:

    if not chunks or top_score < RELEVANCE_THRESHOLD:
        return {
            "answer": REFUSAL_PREFIX,
            "top_score": top_score,
            "refused": True,
            "sources": [],
        }

    context_text = ""
    for chunk in chunks:
        context_text += f"[Source: {chunk['source_file']}]\n{chunk['content']}\n\n"

    prompt = SYSTEM_PROMPT.format(context=context_text)

    response = gemini_client.models.generate_content(
        model="gemini-2.0-flash",
        contents=[
            types.Content(role="user", parts=[types.Part(text=question)]),
        ],
        config=types.GenerateContentConfig(
            system_instruction=prompt,
            temperature=0.2,
        ),
    )

    return {
        "answer": response.text,
        "top_score": top_score,
        "refused": response.text.startswith(REFUSAL_PREFIX),
        "sources": [c["source_file"] for c in chunks],
    }

What does this code do?

• The threshold gate is the hard boundary. If top_score falls below RELEVANCE_THRESHOLD, the function returns a refusal immediately without ever calling Gemini for generation. No LLM self-assessment, no "are you sure?" prompt. Just a concrete score check.
• The return dictionary includes refused as a boolean, which the eval uses to determine pass/fail without parsing the full answer text.

• Add the first half of the run_eval function below retrieve_and_generate. This evaluates the in-scope questions:

def run_eval():
    """Run the full evaluation set and report results."""
    eval_data = json.loads(Path("eval_questions.json").read_text(encoding="utf-8"))

    in_scope_pass = 0
    out_scope_pass = 0
    results_log = []

    print("EVALUATION: In-scope questions (expect cited answers)")
    print("=" * 60)

    for item in eval_data["in_scope"]:
        question = item["question"]
        result = retrieve_and_generate(question)
        passed = not result["refused"]
        if passed:
            in_scope_pass += 1
        status = "PASS" if passed else "FAIL"
        print(f"[{status}] score={result['top_score']:.3f} | {question}")
        results_log.append(
            {"question": question, "type": "in_scope", "passed": passed,
             "score": result["top_score"], "refused": result["refused"]}
        )

    print(f"\nIn-scope: {in_scope_pass}/{len(eval_data['in_scope'])} passed")
    print("\n" + "=" * 60)
    print("EVALUATION: Out-of-scope questions (expect refusal)")
    print("=" * 60)

What does this code do?

For each in-scope question, the eval calls retrieve_and_generate and checks whether the agent produced an answer (not a refusal). A PASS means the agent answered. A FAIL means the threshold incorrectly blocked a question your docs DO cover. The score is printed for every question so you can see exactly why each decision was made.

• Continue the run_eval function by adding the out-of-scope evaluation and summary reporting directly below:

    for item in eval_data["out_of_scope"]:
        question = item["question"]
        result = retrieve_and_generate(question)
        passed = result["refused"]
        if passed:
            out_scope_pass += 1
        status = "PASS" if passed else "FAIL (false confident)"
        print(f"[{status}] score={result['top_score']:.3f} | {question}")
        results_log.append(
            {"question": question, "type": "out_of_scope", "passed": passed,
             "score": result["top_score"], "refused": result["refused"]}
        )

    print(f"\nOut-of-scope: {out_scope_pass}/{len(eval_data['out_of_scope'])} passed")

    total = in_scope_pass + out_scope_pass
    total_questions = len(eval_data["in_scope"]) + len(eval_data["out_of_scope"])
    print(f"\n{'=' * 60}")
    print(f"TOTAL: {total}/{total_questions} ({100 * total / total_questions:.0f}%)")
    print(f"Relevance threshold used: {RELEVANCE_THRESHOLD}")
    print(f"False-confident out-of-scope: {len(eval_data['out_of_scope']) - out_scope_pass}")

    Path("eval_results.json").write_text(
        json.dumps(results_log, indent=2), encoding="utf-8"
    )
    print(f"\nDetailed results saved to eval_results.json")


if __name__ == "__main__":
    run_eval()

What does this code do?

• For out-of-scope questions, the logic flips: a PASS means the agent refused (correct behavior). A FAIL marked "false confident" means the agent answered a question it should not have.
• The summary prints total accuracy, the threshold value used, and the count of false-confident answers. This is your demo-ready output.
• Results are also saved to eval_results.json so you have per-question data for debugging any failures.

• Save run_eval.py.

Seeing a syntax error when you save?

The second half of each function must be indented at exactly the same level as the first half. Check that the if not chunks line and the for item in eval_data["out_of_scope"] line are both indented with 4 spaces (inside their respective functions).

Help me fix my run_eval.py indentation

✔️ Awesome, I've got everything!

Great. Double check you have saved run_eval.py before running the eval.

ⓧ I'd like to double check the full code

Since run_eval.py is a longer file, scroll up and confirm you have all six chunks in order:

• Imports and client setup (ending with SearchClient initialization).
• Constants: RELEVANCE_THRESHOLD, REFUSAL_PREFIX, and SYSTEM_PROMPT.
• retrieve_and_generate function first half (embedding and retrieval, ending with top_score).
• retrieve_and_generate function second half (threshold gate and generation, ending with the return dictionary).
• run_eval function first half (in-scope loop).
• run_eval function second half (out-of-scope loop, summary, save) plus if __name__ == "__main__" at the bottom.

Now it is time to run the eval. Before you do, predict: do you think all 20 questions will pass on the first try with a threshold of 0.7? Or will some out-of-scope questions leak through because their scores are just above the boundary?

• Execute the evaluation by running:

python run_eval.py

The script takes 1-2 minutes to run all 20 questions (each requires an embedding call and potentially a generation call). Watch the output as it processes each question.

What should I see?

You will see each question printed with [PASS] or [FAIL], its retrieval score, and a final summary at the bottom. A successful run looks something like:

TOTAL: 18/20 (90%)
Relevance threshold used: 0.7
False-confident out-of-scope: 0

Your exact numbers will depend on your document content and chunk sizes. The key metric is zero false-confident out-of-scope answers. A few in-scope FAILs (where the threshold was too strict) are fixable by tuning.

Tune the threshold if needed

If you see out-of-scope questions marked FAIL (false confident), their scores were above 0.7. You need to raise the threshold. If you see in-scope questions being refused, their scores were below 0.7 despite relevant content existing. You need to lower the threshold.

To tune: open run_eval.py, change the RELEVANCE_THRESHOLD value, save, and re-run. Also update agent.py to match once you settle on a final value. Use the printed scores to find the boundary between your lowest in-scope score and your highest out-of-scope score.

• Confirm eval_results.json was saved by running:

ls eval_results.json

You should see eval_results.json listed. This file contains per-question data (question text, type, pass/fail, score, refused) that you can reference when explaining your system in interviews.

Seeing rate limit errors during the eval?

The Gemini free tier has rate limits. If you hit them, wait 60 seconds and re-run the script. The eval will restart from the beginning. If you consistently hit limits, add a time.sleep(1) call between questions in the eval loops.

Help me fix Gemini rate limit errors in my eval script

You now have a production-grade Docker support agent with measurable accuracy. You can state concrete numbers: "I validated against a 20-question eval set and achieved X% accuracy with zero false-confident out-of-scope answers." That is the difference between a learning exercise and an interview-ready project.

Secret mission

Deploy as a Web API on Azure App Service

Your Docker support agent works locally. But a local script only proves you can build RAG. Can you deploy it as a live web API on Azure so anyone can query it over HTTP? That is the difference between a learning exercise and a portfolio piece.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. All Azure resources in this project are on free tiers, so ongoing costs are effectively $0. The Storage Account draws a negligible amount (fractions of a cent) from your $100 student credit.

Cost warning

The Azure Storage Account uses a tiny amount of your Azure for Students credit for blob storage. The cost is fractions of a cent per month and will not meaningfully reduce your $100 balance.

If you want to be thorough, delete the resource group to reclaim everything at once.

Resources you used:

• Azure Resource Group docker-agent-rg (contains AI Search, Storage Account, and App Service if you completed the Secret Mission).
• Local project files (Python scripts, docs/ folder, virtual environment, generated JSON files).
• Gemini API key (created in Google AI Studio).

Keep everything running

No action needed. Choose this if you are still actively building or plan to extend the project soon.

• All Azure resources are on free tiers. Total ongoing cost is effectively $0.
• The Azure AI Search free-tier service may be automatically deleted by Microsoft after extended inactivity.
• If you deployed the App Service (Secret Mission), it cold-starts after idle periods. The first request after sleep may take 20-30 seconds.
• Your Gemini API key remains active. It is on the free tier and incurs no charges.

Pause - I'll come back to this later

Free-tier Azure services cannot be paused independently. They remain running at no cost until you delete them.

• You do not need to take any action. The behaviour is the same as the Keep option.
• If you want to free up local disk space, you can deactivate your virtual environment and close VS Code. Your Azure resources will remain available when you return.

Delete - I don't want to use this again

Remove all project resources and start fresh. This is good practice and demonstrates resource lifecycle management (an AZ-104 concept).

• Delete all Azure resources at once by running this command in your terminal:

az group delete --name docker-agent-rg --yes

This removes the resource group and everything inside it (AI Search, Storage Account, and App Service if present). The operation takes a few minutes to complete.

• Delete your Gemini API key to follow security best practices:

• Open the Google Cloud API Credentials page.
• Find the API key you used for this project and click the Actions icon.
• Select Delete API key.
• In the Delete credential modal, select Delete. Deletion takes a few minutes to propagate.

• Delete your local project files by running:

macOS

cd ~/Desktop
rm -rf [[PROJECT_FOLDER="your project folder name"]]

Windows

cd ~\Desktop
Remove-Item -Recurse -Force [[PROJECT_FOLDER="your project folder name"]]

• Unset the GEMINI_API_KEY environment variable if you exported it in your shell session. Close and reopen your terminal to clear it.

Nice Work!

Nice Work!

You just built a production-architecture RAG agent from scratch. Your Docker support agent retrieves answers from real documentation, generates cited responses, and knows when to say "I don't know."

You've learned how to:

• Build a RAG-powered agent that answers questions using real documentation with verifiable [Source: filename] citations.
• Design a multi-service Azure architecture (Azure AI Search + Blob Storage + Resource Group) and wire it to Gemini for both embeddings and generation.
• Implement production guardrails with a concrete, tuned relevance score threshold and validate the system against a 20-question eval set with measurable accuracy numbers.
• Secret Mission: Deploy the agent as a live web API on Azure App Service accessible over the internet.

Ready to quiz yourself?