LLM Evaluation Test Suite with Python

Build a pytest harness that catches AI hallucinations using semantic similarity.

Introduction

30 Second Summary

AI tools are writing emails, answering customer questions, and generating code in workplaces everywhere. The problem is that these tools sometimes sound perfectly confident while giving answers that are completely wrong.

In this project, you will build a Python test suite that automatically evaluates AI responses for quality and accuracy. You will send prompts to Google Gemini, then run automated checks that catch bad outputs the same way traditional test suites catch software bugs.

What You'll Build

You will run a single pytest command and watch your terminal fill with a test report showing exactly which AI responses passed your quality checks and which failed.

By the end of this project, you'll have:

  • An evaluation harness you can point at any prompt to automatically test the AI's response against its source material.
  • Deterministic test functions that catch the most common AI quality issues: forbidden phrases, missing keywords, and fabricated URLs.
  • Semantic evaluation tests that detect AI hallucinations by measuring whether a response stays faithful to its source material.
  • Secret Mission: Add prompt regression testing that compares outputs across prompt versions to catch quality drift.

Do I need Python experience?

This project is designed for beginners transitioning to Python. If you have experience with any programming language, the syntax will feel familiar. Step 1 walks you through installing everything from scratch.

Before We Start

Before you write any code, take a moment to lock in what you are about to build and why it matters for your career.

Set Up Your Python Environment and Gemini API Key

Testing AI is different from testing traditional software. Instead of checking that buttons work or databases return the right rows, you'll check that an AI's answers are accurate, safe, and grounded in facts. To do that, you need a Python environment and access to an AI model.

In this step, you'll set up VS Code as your editor (replacing Eclipse for this project), install Python, and grab a free API key from Google so your test suite can call the Gemini AI model.

In this step, get ready to:
  • Install Python 3.10+ and VS Code with the Python extension.
  • Create a project folder with a virtual environment and install your dependencies.
  • Get a free Gemini API key and set up your configuration files.
Install Python and VS Code

Python is the language your evaluation harness will be written in. VS Code is a lightweight editor that has a built-in terminal, so you can write code and run tests in the same window.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or Command Prompt (Windows) and press Enter to open it.
  • Check if Python is already installed by running this command:

macOS

python3 --version

Windows

python --version

✔️ I see version 3.10 or higher

Python is ready to go. Skip ahead to the VS Code install instructions below.

ⓧ I see an older version

You have Python installed, but your version is too old for the packages this project uses. You need 3.10 or higher.

  • Go to python.org/downloads and download the latest Python installer for your operating system.
  • Run the installer. On Windows, check the box that says Add Python to PATH before clicking Install Now.
  • Close and reopen your terminal, then run the version check command again to confirm you see 3.10 or higher.

macOS showing the old version after installing?

macOS ships with an older Python by default. Even after installing a newer version, python3 --version may still show the old one. Try running python3.14 --version (replacing 3.14 with the version you installed) to confirm the new install exists. You can also use the full path: /Library/Frameworks/Python.framework/Versions/3.14/bin/python3.

ⓧ Command not found

Python is not installed yet. Let's install it now.

  • Go to python.org/downloads and download the latest Python installer for your operating system.
  • Run the installer. On Windows, check the box that says Add Python to PATH before clicking Install Now.
  • On macOS, run through the installer wizard and accept the defaults.
  • Close and reopen your terminal, then run the version check command again to confirm you see 3.10 or higher.

Now let's install VS Code and its Python extension.

  • Go to code.visualstudio.com and download the installer for your operating system.
  • Run the installer and accept the defaults.
  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Visual Studio Code and press Enter to open it.
  • Open the Extensions panel by pressing Cmd+Shift+X (macOS) or Ctrl+Shift+X (Windows).
  • Type Python in the search box.
  • Find the extension published by Microsoft and click Install.

You should see the Install button change to a gear icon once the extension finishes installing.

Don't see VS Code when you search?

Make sure the installer finished completely before searching. On Windows, you may need to restart your computer for VS Code to appear in search results.

Help me install VS Code on my computer.

Create your project and install dependencies

Your project needs its own folder and a virtual environment. A virtual environment keeps your project's packages separate from any other Python projects on your machine.

  • Move to your Desktop by running this command in your system terminal:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir llm-eval-suite
cd llm-eval-suite
  • Confirm the folder was created by running:
ls

The output should be empty because the folder is brand new. That's expected.

Now open this folder in VS Code so you can work from inside the editor's terminal.

  • In VS Code, click File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop, select the llm-eval-suite folder, and click Open.

You should see llm-eval-suite in the Explorer sidebar on the left. The folder is empty for now.

  • Open the VS Code terminal by pressing Ctrl+` (that's the backtick key, next to the number 1 on most keyboards).
  • Create and activate a virtual environment by running these commands:

macOS

python3 -m venv venv
source venv/bin/activate

Windows

python -m venv venv
venv\Scripts\activate

Your terminal prompt should now show (venv) at the beginning. This means the virtual environment is active.

Seeing a permission error on Windows?

If you see a message about execution policies in PowerShell, VS Code may be blocking scripts. Open the VS Code terminal dropdown (the arrow next to the + icon in the terminal panel) and select Command Prompt instead. Then re-run the activate command.

Help me activate my Python virtual environment on Windows.

  • Install the three packages your project needs by running this command:
pip install google-genai==2.18.1 pytest==9.1.1 sentence-transformers==5.7.0

What are these three packages?

google-genai is the official Python SDK for calling Google's Gemini AI model. pytest is a testing framework that runs your evaluation tests and produces pass/fail reports.

sentence-transformers is a machine learning library that converts text into numerical representations called embeddings. You'll use it to measure how similar an AI response is to its source material. This package pulls in several dependencies, so the install may take a few minutes.

The install is finished when you see your terminal prompt return with (venv) at the front. You should also see a line near the end of the output that says Successfully installed followed by a list of packages.

Install failing or taking forever?

Make sure your virtual environment is active (you see (venv) in your prompt). If you run pip install outside the venv, packages may install in the wrong location or fail due to permissions.

Help me fix my pip install for this AI testing project.

Get your Gemini API key and create config files

Your test suite needs to call the Gemini AI model, which requires an API key. Google offers the Gemini API on a free tier with no credit card required.

  • Go to aistudio.google.com in your browser.
  • Sign in with your Google account.
  • Click Get API key in the left sidebar.
  • Click Create API key.
  • Copy the key that appears. You will only see it once, so paste it somewhere safe (like a notes app) for the next step.

Is this really free?

Yes. Google's Gemini API free tier charges nothing for input and output tokens. No billing account or credit card is required. This project stays well within the free tier limits.

Now store the key in your project so your code can find it.

  • In the VS Code Explorer sidebar, hover over the llm-eval-suite folder name and click the New File icon (the page icon with a +).
  • Name the file .env and press Enter.
  • Paste the following line into the file, replacing your-api-key-here with the actual API key you copied:
GEMINI_API_KEY=your-api-key-here
  • Save the file with Cmd+S (macOS) or Ctrl+S (Windows).

You should see .env appear in the Explorer sidebar.

Next, create a configuration file that pytest uses to share setup code across all your test files. This file loads the API key from .env and creates a reusable Gemini client.

  • Create a new file in the Explorer sidebar called conftest.py.
  • Paste the following code into conftest.py:
import os
import pytest
from google import genai


# Session-scoped fixture: runs once and shares the result across all tests
@pytest.fixture(scope="session")
def api_key():
    # Try the environment variable first
    key = os.environ.get("GEMINI_API_KEY")
    if not key:
        # Fall back to reading the .env file directly
        env_path = os.path.join(os.path.dirname(__file__), ".env")
        if os.path.exists(env_path):
            with open(env_path) as f:
                for line in f:
                    if line.startswith("GEMINI_API_KEY="):
                        key = line.strip().split("=", 1)[1]
    if not key:
        # Skip all tests if no key is available
        pytest.skip("GEMINI_API_KEY not set")
    return key


@pytest.fixture(scope="session")
def gemini_client(api_key):
    # Create a reusable Gemini client for the entire test session
    return genai.Client(api_key=api_key)

What does this code do?

conftest.py is a special file that pytest discovers automatically. Any fixture defined here is available to every test file in the project without needing an import.

The api_key fixture loads your key from the .env file. The gemini_client fixture creates a Google Gemini client that all your tests can reuse. Both are scope="session", meaning they only run once no matter how many tests use them.

  • Save conftest.py.

Your Explorer sidebar should now show two files: .env and conftest.py.

Don't see the .env file in the sidebar?

Files starting with a dot are sometimes hidden by default. Make sure you typed the full name .env (including the dot) when creating the file. If you don't see it, click File then Open Folder and re-select llm-eval-suite to refresh the view.

Help me find my .env file in VS Code.

✔️ Awesome, I've got everything!

Great. Make sure both files are saved before moving on.

ⓧ I'd like to double check the full code

Here is the complete conftest.py file. Make sure yours matches:

import os
import pytest
from google import genai


@pytest.fixture(scope="session")
def api_key():
    key = os.environ.get("GEMINI_API_KEY")
    if not key:
        env_path = os.path.join(os.path.dirname(__file__), ".env")
        if os.path.exists(env_path):
            with open(env_path) as f:
                for line in f:
                    if line.startswith("GEMINI_API_KEY="):
                        key = line.strip().split("=", 1)[1]
    if not key:
        pytest.skip("GEMINI_API_KEY not set")
    return key


@pytest.fixture(scope="session")
def gemini_client(api_key):
    return genai.Client(api_key=api_key)

Also confirm your .env file has GEMINI_API_KEY= followed by your actual API key (not the placeholder text).

Let's confirm your full setup is working. Before you run this, what Python version do you expect to see inside your virtual environment?

  • Verify your Python version inside the virtual environment by running:
python --version

You should see Python 3.10 or higher printed. Inside the virtual environment, the python command works on both macOS and Windows.

Your development environment is ready, with Python, VS Code, and a free Gemini API key all connected. Next up, you'll call the Gemini API and see your first AI response.

Call the Gemini API and See Your First AI Response

Your Python environment is set up, your packages are installed, and your Gemini API key is ready to go. Now it's time to put all of that to work.

Before you can test AI outputs, you need to see one. In this step, you'll build a script that sends a prompt with context to Gemini and captures the response.

In this step, get ready to:
  • Create llm_client.py with functions to call the Gemini API.
  • Send your first prompt and see an AI-generated answer in the terminal.
Create the API client and evaluation function

Your evaluation harness needs two things: a way to connect to the Gemini API using your key, and a function that sends a question along with source context. The source context is the key ingredient. It tells the model what information to base its answer on.

  • In VS Code, create a new file called llm_client.py in your project folder.
  • Add the imports and client function by pasting this code:
import os
from google import genai
from google.genai import types


def get_client():
    # Read the API key from environment or .env file
    api_key = os.environ.get("GEMINI_API_KEY")
    if not api_key:
        with open(".env") as f:
            for line in f:
                if line.startswith("GEMINI_API_KEY="):
                    api_key = line.strip().split("=", 1)[1]
    return genai.Client(api_key=api_key)

What does this code do?

  • The three imports bring in the google-genai SDK. genai is the main module, and types holds configuration objects you'll use later.
  • get_client() first checks for a GEMINI_API_KEY environment variable. If it doesn't find one, it reads the key from your .env file instead.
  • It returns a genai.Client object that handles all communication with the Gemini API.
  • Save llm_client.py.
  • Verify the file is valid by running this quick import check in your terminal:
python -c "from llm_client import get_client; print('Client ready')"

You should see Client ready printed in the terminal.

Seeing an error instead?

  • Check that your virtual environment is still active. You should see (venv) at the start of your terminal prompt.
  • Make sure your .env file is in the same folder as llm_client.py. The get_client() function looks for it in the current directory.
  • If you see a ModuleNotFoundError, double check that you ran pip install google-genai==2.18.1 inside your activated virtual environment.

Help me debug this import error in llm_client.py

  • Add the evaluation function below get_client() in llm_client.py by pasting this code:
def ask_with_context(question, context, system_instruction=None):
    # Create a client for this call
    client = get_client()
    # Use a default system instruction if none provided
    if system_instruction is None:
        system_instruction = (
            "You are a helpful assistant. Answer the question based ONLY on "
            "the provided context. Do not add information that is not in the "
            "context. If the context does not contain the answer, say "
            "'I don't have enough information to answer that.'"
        )

    # Combine context and question into one prompt
    prompt = f"Context:\n{context}\n\nQuestion: {question}"

    # Call Gemini with configuration for controlled output
    response = client.models.generate_content(
        model="gemini-2.5-flash",
        contents=prompt,
        config=types.GenerateContentConfig(
            system_instruction=system_instruction,
            temperature=0.3,
            max_output_tokens=500,
        ),
    )
    return response.text

What does this function do?

  • ask_with_context() is the heart of your evaluation harness. It takes a question, a context paragraph (the source material), and an optional system instruction.
  • The system_instruction tells the model to answer ONLY from the provided context. This is critical for testing. If the model ignores the instruction and adds outside knowledge, your later tests will catch it as a hallucination.
  • temperature=0.3 keeps the output relatively consistent between runs. Lower temperature means less randomness.
  • max_output_tokens=500 caps the response length so your tests deal with manageable output.
  • Save llm_client.py. The next substep puts this function to work with a real API call.
Run your first AI call

Your function is ready, but you need a way to actually run it. A Python __main__ block lets you execute the file directly from the terminal with a sample question and context.

  • Add the following code at the very bottom of llm_client.py, below the ask_with_context() function:
if __name__ == "__main__":
    # Sample context paragraph about Python
    context = (
        "Python is a high-level, general-purpose programming language. "
        "Its design philosophy emphasizes code readability with the use of "
        "significant indentation. Python is dynamically typed and "
        "garbage-collected. It supports multiple programming paradigms, "
        "including structured, object-oriented, and functional programming."
    )
    # Ask a question and print the AI's answer
    question = "What is Python used for?"
    answer = ask_with_context(question, context)
    print("Question:", question)
    print("Answer:", answer)

What does this block do?

  • if __name__ == "__main__" means this code only runs when you execute the file directly. It won't run when other files import the functions.
  • The context variable is a short paragraph about Python. This is the source material that the model must stick to when answering.
  • The script prints both the question and the AI-generated answer so you can see exactly what went in and what came out.
  • Save llm_client.py.

Before you run this, what do you think the output will look like? The model received a paragraph about Python and the question "What is Python used for?" Will it stay within the context, or add outside knowledge?

  • Run your script by entering this command in the terminal:
python llm_client.py

You should see the question printed first, followed by an AI-generated answer about Python programming. The answer draws from the context paragraph you provided. Your first AI response is live.

Why does the answer matter for testing?

This script is a baby version of your full evaluation harness. It sends a prompt with context and captures the response. Every test you write in the next steps will follow this same pattern: give the model a question and context, capture the answer, then check it.

Not seeing a response?

  • If you see an authentication or API key error, open your .env file and confirm GEMINI_API_KEY= is followed by your actual key with no spaces or quotes around the value.
  • If you see a network or connection error, check that your internet connection is active. The script needs to reach the Gemini API.
  • If you see a NameError mentioning get_client or ask_with_context, make sure the __main__ block is below both function definitions in the file.

Help me debug why my llm_client.py isn't returning a response

✔️ Awesome, I've got everything!

Double check you've saved llm_client.py. You're all set to move on.

ⓧ I'd like to double check the full code

Your llm_client.py should contain three sections, in this order. Scroll up and confirm each one is present:

  • Imports (os, genai, types) followed by the get_client() function that reads your API key.
  • The ask_with_context() function with the system instruction, prompt assembly, and the generate_content call using gemini-2.5-flash.
  • The if __name__ == "__main__" block with the sample context about Python, the question, and two print() statements.

Your evaluation harness has its first working piece: a script that sends a prompt with context and captures the AI response. Next up, you'll write automated tests that check whether those responses are actually good.

Write Deterministic Evaluation Tests

Your API connection is live and you have a real AI response printed in your terminal. The next question is: how do you know if that response is actually good?

Deterministic tests are the simplest way to check AI outputs. They use exact rules like string matching and regex patterns to catch surface-level problems. If you have written assertions in Selenium, these will feel very familiar.

In this step, get ready to:
  • Write pytest functions that check AI responses for forbidden content and required keywords.
  • Add guardrail tests for response length and fabricated URLs.
  • Discover why these checks alone are not enough for AI testing.
Add shared test data to your configuration

Before you write any tests, pytest needs test data to work with. You will add three fixtures to conftest.py that provide a sample context paragraph, a sample question, and a live AI response fetched from Gemini.

  • Switch back to conftest.py in VS Code and add the following code at the bottom of the file, below the existing gemini_client fixture:
@pytest.fixture(scope="session")
def sample_context():
    # The source material the AI must base its answer on
    return (
        "Python is a high-level, general-purpose programming language. "
        "Its design philosophy emphasizes code readability with the use of "
        "significant indentation. Python is dynamically typed and "
        "garbage-collected. It supports multiple programming paradigms, "
        "including structured, object-oriented, and functional programming."
    )


@pytest.fixture(scope="session")
def sample_question():
    return "What is Python used for?"


@pytest.fixture(scope="session")
def sample_response(sample_question, sample_context):
    # Call the real API once and reuse the response across all tests
    from llm_client import ask_with_context
    return ask_with_context(sample_question, sample_context)

What does this code do?

These three fixtures create a chain of test data that pytest shares across every test function.

  • sample_context returns the same paragraph you used in llm_client.py. This is the source of truth the AI should stick to.
  • sample_question returns a fixed question so every test evaluates the same prompt.
  • sample_response calls the Gemini API once, then reuses that response for every test in the session. This keeps your tests fast and your API usage low.
  • Save conftest.py.

✔️ Awesome, I've got everything!

Great. Make sure conftest.py is saved and you can see the three new fixtures below the existing ones.

ⓧ I'd like to double check the full code

Your conftest.py should have five fixtures in total. Scroll through the file and confirm these appear in order:

  • api_key (from Step 1).
  • gemini_client (from Step 1).
  • sample_context (just added).
  • sample_question (just added).
  • sample_response (just added).
Write tests for empty responses and forbidden content

The first two tests check the most basic quality bars. Did the AI actually return something? And did it answer like a helpful assistant instead of deflecting with canned phrases?

  • Create a new file called test_deterministic.py in your project folder and paste the following code:
import re


def test_response_not_empty(sample_response):
    # Check that the AI returned a non-empty response
    assert sample_response is not None
    assert len(sample_response.strip()) > 0


def test_no_forbidden_content(sample_response):
    # Check for phrases that reveal the AI is deflecting
    forbidden_phrases = [
        "I'm just an AI",
        "As an AI language model",
        "I cannot provide",
        "I apologize, but",
    ]
    response_lower = sample_response.lower()
    for phrase in forbidden_phrases:
        assert phrase.lower() not in response_lower, (
            f"Response contains forbidden phrase: '{phrase}'"
        )

What does this code do?

  • test_response_not_empty checks that Gemini returned actual content. A blank or None response is an instant failure.
  • test_no_forbidden_content scans for phrases like "I'm just an AI" that signal the model is deflecting instead of answering. These phrases sneak into production AI responses more often than you would expect.
  • Each function takes sample_response as a parameter. pytest automatically injects the fixture you defined in conftest.py.
  • Save test_deterministic.py.
  • Run the tests by entering this command in your terminal:
pytest test_deterministic.py -v

You should see two lines ending in PASSED, one for each test function.

Seeing FAILED or an error?

  • If you see fixture 'sample_response' not found, make sure you saved conftest.py after adding the three new fixtures. The file must be in the same folder as your test file.
  • If test_no_forbidden_content fails, Gemini included one of the forbidden phrases. This is actually a real catch. Re-run the test and the fresh API response may differ.
  • If you see a connection error, check that your .env file still contains a valid API key and that your virtual environment is activated.

Need more help? Why are my pytest fixtures not being found?

Add a keyword test and two guardrail tests

The next three tests check whether the response talks about the right topic and stays within reasonable bounds.

  • In test_deterministic.py, add the following function below test_no_forbidden_content:
def test_contains_required_keywords(sample_response):
    # Verify the response mentions key terms from the context
    required_keywords = ["python", "programming"]
    response_lower = sample_response.lower()
    for keyword in required_keywords:
        assert keyword in response_lower, (
            f"Response missing required keyword: '{keyword}'"
        )

What does this test do?

This test checks that the AI response actually mentions the key terms from the source context. If the context is about Python programming and the response never says "python" or "programming", something went wrong.

  • Now add these two guardrail tests below test_contains_required_keywords:
def test_response_length_within_bounds(sample_response):
    # Ensure the response is neither too short nor too long
    length = len(sample_response)
    assert 50 <= length <= 2000, (
        f"Response length {length} is outside bounds [50, 2000]"
    )


def test_no_hallucinated_urls(sample_response):
    # Check that the AI did not fabricate any URLs
    url_pattern = r"https?://[^\s]+"
    urls_found = re.findall(url_pattern, sample_response)
    assert len(urls_found) == 0, (
        f"Response contains fabricated URLs: {urls_found}"
    )

What do these tests catch?

  • test_response_length_within_bounds catches two failure modes at once. A response shorter than 50 characters is probably a non-answer. A response longer than 2000 characters means the AI is rambling.
  • test_no_hallucinated_urls uses a regex pattern to find any URLs in the response. The source context has no URLs, so any URL the AI includes is fabricated.
  • Save test_deterministic.py.

✔️ Awesome, I've got everything!

Great. Make sure test_deterministic.py is saved. You should have five test functions in total plus the import re at the top.

ⓧ I'd like to double check the full code

Scroll through test_deterministic.py and confirm these sections appear in order:

  • import re at the top of the file.
  • test_response_not_empty (two assertions checking for None and empty strings).
  • test_no_forbidden_content (a list of four forbidden phrases with a loop checking each one).
  • test_contains_required_keywords (checks for "python" and "programming").
  • test_response_length_within_bounds (asserts length is between 50 and 2000).
  • test_no_hallucinated_urls (regex check for fabricated URLs).
Run the full test suite and discover the gap

You now have five test functions that check an AI response from five different angles. Before you run them, take a moment to predict what will happen.

Predict before you run

You asked Gemini a question about Python and gave it a factual context paragraph. Do you think all five tests will pass? Which test is most likely to fail, and why?

  • Run all five tests by entering this command:
pytest test_deterministic.py -v

You should see five lines, each ending with PASSED. Every test is green.

One or more tests failed?

  • If test_contains_required_keywords failed, Gemini may have rephrased its answer without using the exact words "python" or "programming". Re-run the tests. The fixture calls the API fresh each session.
  • If test_response_length_within_bounds failed, check the printed length. If it is very short, your API key may have hit a rate limit. Wait a minute and try again.
  • If test_no_hallucinated_urls failed, Gemini included a URL it made up. That is actually a real hallucination your test just caught.

Still stuck? Help me debug my failing deterministic tests.

All five tests passed. That feels good. But think about this: would these tests catch a response that sounds authoritative but makes up facts?

Why passing tests are not enough

Imagine Gemini responded with: "Python was invented by NASA in 2020 to control Mars rovers." That response is not empty, contains no forbidden phrases, mentions "python" and "programming", is a reasonable length, and has no URLs.

All five of your tests would pass on that response. Every single one. Yet the answer is completely fabricated. Deterministic checks catch formatting and structure problems, but they cannot tell you whether the content is actually true. That gap is exactly what the next step solves.

Your deterministic tests are solid for catching surface-level issues. Next up, you will build semantic evaluators that measure how faithful an AI response is to its source material.

Catch a Hallucination with Semantic Evaluation

Your deterministic tests from the previous step all passed. Every response cleared the forbidden-phrase filters, keyword checks, length bounds, and URL checks.

Here is the gap those tests leave open. A response can contain all the right keywords, avoid every forbidden phrase, and land within the character limit, yet still fabricate facts that never appeared in the source material. Keyword matching cannot detect hallucinations. In this step, you will build a semantic evaluator that measures how faithful a response is to its source context.

In this step, get ready to:
  • Build semantic evaluation functions that measure how faithful an AI response is to its source context using embedding similarity.
  • Write pytest tests that catch hallucinated responses by scoring them against the original context.
Create the semantic evaluator module

Semantic evaluation works by converting text into numerical vectors called embeddings. If an AI response is semantically close to the source context, it is likely faithful. If it drifts far away, it may contain fabricated content.

  • Create a new file called evaluators.py in your project folder and add this code:
from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim


# Load a lightweight embedding model (downloads ~100MB on first run)
model = SentenceTransformer("all-MiniLM-L6-v2")


def faithfulness_score(answer, context):
    # Encode both texts into vectors and measure similarity
    embeddings = model.encode([answer, context])
    similarity = cos_sim(embeddings[0], embeddings[1]).item()
    return similarity


def contains_hallucination(answer, context, threshold=0.3):
    # A score below the threshold means the answer drifted from the context
    score = faithfulness_score(answer, context)
    return score < threshold


def relevance_score(answer, question):
    # Measures how closely the answer relates to the question asked
    embeddings = model.encode([answer, question])
    similarity = cos_sim(embeddings[0], embeddings[1]).item()
    return similarity

What does this code do?

  • SentenceTransformer("all-MiniLM-L6-v2") loads a lightweight embedding model that converts text into 384-dimensional vectors. The first time you run this, it downloads the model weights from Hugging Face (about 100MB).
  • faithfulness_score encodes both the answer and the context into vectors, then computes cosine similarity. The result is a float between -1 and 1, where higher means more similar.
  • contains_hallucination returns True when the faithfulness score drops below a threshold (default 0.3), indicating the response drifted from the source material.
  • relevance_score measures how closely the answer relates to the question itself. You will use this in a later step.
  • Save evaluators.py.
  • Test that the module works by running this command in your terminal:
python -c "from evaluators import faithfulness_score; print(faithfulness_score('Python is a programming language', 'Python is a high-level programming language'))"

You should see a number printed, something like 0.85 or higher. This confirms the embedding model loaded and the scoring function works.

First run takes longer

The first time you run this command, the all-MiniLM-L6-v2 model downloads from Hugging Face. This can take 30 seconds or more depending on your internet speed. After the first download, the model is cached locally and loads in under a second.

Not seeing a score?

  • Make sure your virtual environment is activated. You should see (venv) at the start of your terminal prompt.
  • Check that sentence-transformers installed correctly in Step 1. Try running pip install sentence-transformers==5.7.0 again.
  • If you see a connection error, check your internet connection. The model needs to download from Hugging Face on the first run.

Ask for help: Why can't Python import my evaluators module?

Write tests that catch hallucinations

Your evaluator functions need a shared context fixture that both test files can use. You will add this to conftest.py so pytest injects it automatically into any test function that asks for it.

  • In conftest.py, find the gemini_client fixture at the bottom of the file.
  • Add this new fixture below it:
@pytest.fixture(scope="session")
def sample_context():
    return (
        "Python is a high-level, general-purpose programming language. "
        "Its design philosophy emphasizes code readability with the use of "
        "significant indentation. Python is dynamically typed and "
        "garbage-collected. It supports multiple programming paradigms, "
        "including structured, object-oriented, and functional programming."
    )

What is a pytest fixture?

A fixture is a reusable piece of test setup. When a test function has a parameter named sample_context, pytest automatically looks for a fixture with that name and passes the return value as the argument. Defining fixtures in conftest.py makes them available to every test file in the project.

  • Save conftest.py.
  • Create a new file called test_semantic.py in your project folder and add this code:
from evaluators import faithfulness_score, contains_hallucination, relevance_score


def test_faithful_response_passes(sample_context):
    # A response that sticks to facts from the context
    faithful_answer = (
        "Python is a high-level, general-purpose programming language "
        "that supports multiple programming paradigms including structured, "
        "object-oriented, and functional programming. It emphasizes "
        "code readability."
    )
    score = faithfulness_score(faithful_answer, sample_context)
    assert score >= 0.4, (
        f"Faithfulness score {score:.2f} is below threshold 0.4"
    )

What does this test do?

This test provides a response that closely mirrors the source context. It asserts the faithfulness score is at least 0.4. A grounded response should score well above this threshold because the words and meaning overlap heavily with the original context.

  • Add the second test function below the first one in test_semantic.py:
def test_hallucinated_response_fails(sample_context):
    # A response full of fabricated claims not in the context
    hallucinated_answer = (
        "Python was invented by NASA in 2020 to control Mars rovers. "
        "It runs exclusively on quantum computers and is the only "
        "language approved for interstellar communication protocols."
    )
    assert contains_hallucination(hallucinated_answer, sample_context), (
        "Hallucinated response was not detected as a hallucination"
    )

What does this test do?

This test feeds in a completely fabricated response. None of those claims appear in the source context. The contains_hallucination function should return True because the faithfulness score falls below the 0.3 threshold.

  • Save test_semantic.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved both conftest.py and test_semantic.py.

ⓧ I'd like to double check the full code

Here is the full evaluators.py:

from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim


model = SentenceTransformer("all-MiniLM-L6-v2")


def faithfulness_score(answer, context):
    embeddings = model.encode([answer, context])
    similarity = cos_sim(embeddings[0], embeddings[1]).item()
    return similarity


def contains_hallucination(answer, context, threshold=0.3):
    score = faithfulness_score(answer, context)
    return score < threshold


def relevance_score(answer, question):
    embeddings = model.encode([answer, question])
    similarity = cos_sim(embeddings[0], embeddings[1]).item()
    return similarity

Here is the full test_semantic.py:

from evaluators import faithfulness_score, contains_hallucination, relevance_score


def test_faithful_response_passes(sample_context):
    faithful_answer = (
        "Python is a high-level, general-purpose programming language "
        "that supports multiple programming paradigms including structured, "
        "object-oriented, and functional programming. It emphasizes "
        "code readability."
    )
    score = faithfulness_score(faithful_answer, sample_context)
    assert score >= 0.4, (
        f"Faithfulness score {score:.2f} is below threshold 0.4"
    )


def test_hallucinated_response_fails(sample_context):
    hallucinated_answer = (
        "Python was invented by NASA in 2020 to control Mars rovers. "
        "It runs exclusively on quantum computers and is the only "
        "language approved for interstellar communication protocols."
    )
    assert contains_hallucination(hallucinated_answer, sample_context), (
        "Hallucinated response was not detected as a hallucination"
    )

For conftest.py, scroll up and confirm the sample_context fixture is present below the gemini_client fixture.

Run the semantic tests

Before you run the tests, think about what you expect to happen. You have one test with a faithful response and one with a completely fabricated response. Do you think both tests will pass?

  • Run the semantic test suite:
pytest test_semantic.py -v

Both tests pass. The faithful response scores well above the 0.4 threshold. The hallucinated response scores below 0.3, so contains_hallucination correctly returns True.

Why is this a breakthrough?

Your deterministic tests from Step 3 would have passed the hallucinated response without blinking. It contains no forbidden phrases, it has keywords like "Python", and it falls within the length bounds.

Semantic evaluation catches it because it measures meaning, not just keywords. The fabricated NASA and quantum computing claims have no semantic overlap with the real context about Python's design philosophy. This is the core skill of AI testing that separates it from traditional QA.

Seeing test failures?

  • If you see an error about sample_context not being found, make sure you added the sample_context fixture to conftest.py and saved the file.
  • If the faithfulness test fails with a score below 0.4, try running the test again. Embedding models are deterministic, so a low score likely means the evaluators.py code has a typo.
  • Make sure evaluators.py is saved in the same folder as test_semantic.py. Python needs both files in the same directory to resolve the import.

Ask for help: Why are my semantic tests failing?

Your test suite now catches something that keyword matching never could. Next up, you will combine all your tests into a single evaluation harness that runs with one command.

Build the Full Evaluation Suite

Your semantic evaluator just caught a hallucination that every keyword check missed. That is a powerful capability, but right now your tests live in separate files with no shared setup.

A real evaluation harness runs all checks together against live API responses and produces a single pass/fail report. In this step, you will wire your test files into a unified pytest suite that an employer could run with one command to see your AI testing skills in action.

In this step, get ready to:
  • Add shared test fixtures to your configuration file so both test files share a single live API response.
  • Add a relevance test that checks the AI actually answered the right question.
  • Run the full evaluation suite and see a unified pass/fail report.
Add shared fixtures to conftest.py

Right now, your test functions declare parameters like sample_response and sample_context, but pytest does not know where to find those values yet. Pytest uses a file called conftest.py to automatically inject shared fixtures into any test function that requests them by name.

  • In VS Code, open conftest.py.
  • Find the gemini_client fixture at the bottom of the file. Copy the following code below it:

@pytest.fixture(scope="session")
def sample_context():
    return (
        "Python is a high-level, general-purpose programming language. "
        "Its design philosophy emphasizes code readability with the use of "
        "significant indentation. Python is dynamically typed and "
        "garbage-collected. It supports multiple programming paradigms, "
        "including structured, object-oriented, and functional programming."
    )


@pytest.fixture(scope="session")
def sample_question():
    return "What is Python used for?"


@pytest.fixture(scope="session")
def sample_response(sample_question, sample_context):
    from llm_client import ask_with_context
    return ask_with_context(sample_question, sample_context)

What does this code do?

These three fixtures provide the shared test data for your entire suite.

  • sample_context returns the Python paragraph that serves as the AI's source material.
  • sample_question returns the question your tests ask the AI.
  • sample_response calls the live Gemini API using ask_with_context and caches the result for the entire test session. Every test that needs the AI response receives the same one, so you only make one API call per run.
  • Save conftest.py.
  • Confirm the fixtures are wired up correctly by running this command in your terminal:
pytest test_deterministic.py -v

You should see all 5 deterministic tests pass with green PASSED labels. This time, they are running against a live Gemini API response instead of hardcoded text.

Seeing fixture errors?

  • If you see fixture 'sample_response' not found, make sure you added the new fixtures below the existing gemini_client fixture in conftest.py and saved the file.
  • If you see an API error, check that your .env file still contains your Gemini API key and that your virtual environment is active.

Still stuck? Ask for help.

✔️ Awesome, I've got everything!

Great. Make sure you have saved conftest.py before moving on.

ⓧ I'd like to double check the full code

Your conftest.py is longer than a single screen now. Scroll through it and confirm these sections appear in order:

  • The imports at the top: os, pytest, and genai.
  • The api_key fixture that reads from the environment or .env file.
  • The gemini_client fixture that creates the API client.
  • The three new fixtures: sample_context, sample_question, and sample_response.
Add a relevance test to test_semantic.py

Your faithfulness tests check whether the AI sticks to the source material. But there is another failure mode: the AI could produce a response that is faithful to the context yet answers a completely different question. A relevance test catches this by measuring how closely the response relates to the question that was asked.

  • In VS Code, open test_semantic.py.
  • Find the test_hallucinated_response_fails function at the bottom of the file. Copy the following code below it:

def test_answer_relevance(sample_response, sample_question):
    score = relevance_score(sample_response, sample_question)
    assert score >= 0.3, (
        f"Relevance score {score:.2f} is below threshold 0.3"
    )

What does this test do?

This test uses the relevance_score function from evaluators.py to measure how semantically similar the AI's response is to the original question. A score below 0.3 means the model drifted off-topic.

  • Save test_semantic.py.
  • Confirm the new test passes by running this command:
pytest test_semantic.py -v

You should see all three semantic tests pass: test_faithful_response_passes, test_hallucinated_response_fails, and your new test_answer_relevance.

Relevance test failing?

  • Check that the import line at the top of test_semantic.py includes relevance_score. It should read: from evaluators import faithfulness_score, contains_hallucination, relevance_score.
  • Make sure the new function is named exactly test_answer_relevance. Pytest only discovers functions that start with test_.

Need help debugging? Ask here.

✔️ Awesome, I've got everything!

Great. Make sure you have saved test_semantic.py before moving on.

ⓧ I'd like to double check the full code

Here is the complete test_semantic.py file. Compare yours against it:

from evaluators import faithfulness_score, contains_hallucination, relevance_score


def test_faithful_response_passes(sample_context):
    faithful_answer = (
        "Python is a high-level, general-purpose programming language "
        "that supports multiple programming paradigms including structured, "
        "object-oriented, and functional programming. It emphasizes "
        "code readability."
    )
    score = faithfulness_score(faithful_answer, sample_context)
    assert score >= 0.4, (
        f"Faithfulness score {score:.2f} is below threshold 0.4"
    )


def test_hallucinated_response_fails(sample_context):
    hallucinated_answer = (
        "Python was invented by NASA in 2020 to control Mars rovers. "
        "It runs exclusively on quantum computers and is the only "
        "language approved for interstellar communication protocols."
    )
    assert contains_hallucination(hallucinated_answer, sample_context), (
        "Hallucinated response was not detected as a hallucination"
    )


def test_answer_relevance(sample_response, sample_question):
    score = relevance_score(sample_response, sample_question)
    assert score >= 0.3, (
        f"Relevance score {score:.2f} is below threshold 0.3"
    )
Run the full evaluation suite

Everything is wired up. Your deterministic tests and semantic tests both pull from the same shared fixtures, and pytest can discover all of them in a single run.

Before you run the full suite, think about what you expect to see. How many total tests should appear in the report? Do you think they will all pass?

  • Run the full evaluation suite by running this command:
pytest --tb=short -v

What does this command do?

  • pytest with no file argument discovers and runs every test file in the current directory.
  • --tb=short shortens the traceback output so failures are easier to read.
  • -v (verbose) shows each test name with its PASSED or FAILED status.

You should see 8 tests run together in a single report. Five deterministic tests from test_deterministic.py and three semantic tests from test_semantic.py, each with a green PASSED label. The summary line at the bottom shows the total count.

This is your evaluation harness

That one command just ran every quality check you have built across this project. It called the Gemini API, captured the response, checked for forbidden content, verified keywords and length, scanned for fabricated URLs, measured faithfulness to source context, and confirmed the response was relevant to the question.

Anyone can clone your project, set a GEMINI_API_KEY, and run pytest --tb=short -v to see exactly what your AI testing skills look like in practice.

Some tests failing?

  • If a deterministic test like test_contains_required_keywords fails, this is expected occasionally. The AI's live response may not always include the exact keywords. Re-run the suite once. If it keeps failing, the model may have generated an unusually short response.
  • If you see SKIPPED on every test, your GEMINI_API_KEY is not being found. Confirm your .env file is in the project root alongside your test files.
  • If you see import errors, make sure your virtual environment is still active. You should see (venv) at the start of your terminal prompt.

Need help troubleshooting your test results? Ask here.

Secret mission

Add Prompt Regression Testing

Your evaluation harness catches hallucinations and quality issues in a single response. But in production, prompts change constantly. A small tweak to a system instruction can silently degrade output quality. Can you build a regression test that compares outputs across prompt versions and flags quality drift before users notice?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on your local machine using the free tier of the Gemini API, so there are no ongoing costs. Decide whether to keep your resources for future experiments, pause them to come back later, or delete them entirely.

Resources you used:

  • Project folder on your Desktop containing all Python files, the virtual environment, and your .env file.
  • Gemini API key created in Google AI Studio.
  • Cached all-MiniLM-L6-v2 model downloaded by sentence-transformers to your Hugging Face cache.

Keep everything running

No action needed. Choose this if you plan to keep experimenting with your evaluation harness, adding new tests, or tweaking prompts.

  • Your project folder, virtual environment, and all test files stay on your Desktop.
  • Your Gemini API key remains active. The free tier has no billing, so there is no cost to leaving it in place.
  • The cached all-MiniLM-L6-v2 model stays in ~/.cache/huggingface/hub for faster loading next time.

Pause - I'll come back to this later

Shut down running processes and deactivate your virtual environment to free up terminal resources. Your files stay intact.

  • If your virtual environment is still active, deactivate it by running:
deactivate
  • Close VS Code if you are done for now.
  • When you come back, open the project folder in VS Code, activate the virtual environment again, and pick up where you left off.

Delete - I don't want to use this again

Remove all project resources and cached data to start completely fresh.

  • If your virtual environment is still active, deactivate it first by running:
deactivate
  • Delete the project folder from your Desktop by running:

macOS

cd ~/Desktop
rm -rf llm-eval-suite

Windows

cd ~\Desktop
rmdir /s /q llm-eval-suite
  • Delete the cached all-MiniLM-L6-v2 model that sentence-transformers downloaded. This frees up disk space.

macOS

rm -rf ~/.cache/huggingface/hub/models--sentence-transformers--all-MiniLM-L6-v2

Windows

rmdir /s /q %USERPROFILE%\.cache\huggingface\hub\models--sentence-transformers--all-MiniLM-L6-v2
  • Delete your Gemini API key so it cannot be used by anyone else.
  • Open the Google Cloud API Credentials page.
  • Find the API key you created for this project and click the Actions icon next to it.
  • Select Delete API key.
  • In the Delete credential modal, click Delete to confirm. Deletion takes a few minutes to propagate.

Nice Work!

Nice Work!

You just built a production-style LLM evaluation test suite from scratch. You can now run a single pytest command and get a pass/fail report on any AI response, the exact skill set that separates AI testers from traditional QA.

You've learned how to:

  • Build a Python evaluation harness that calls Google Gemini and runs automated quality checks on every AI response.
  • Write deterministic tests that catch forbidden content, missing keywords, and fabricated URLs, then discover firsthand why keyword matching alone cannot catch hallucinations.
  • Build semantic evaluators using sentence-transformers that measure faithfulness to source context, catching AI responses that sound correct but fabricate facts.
  • Secret Mission: Add prompt regression testing to catch quality drift across prompt versions.

Ready to quiz yourself?