Prompt Drift Lab: LLM Consistency

Measure how LLMs change responses when prompts are systematically rephrased.

Introduction

30 Second Summary

Ask someone for directions two different ways and you might get two completely different routes. Large language models have the same problem. Rephrase a prompt slightly and the response can shift in surprising ways.

In this project, you will build a research tool that systematically measures how two AI models change their responses when prompts are rephrased or adversarially modified. You will run a controlled experiment, compute drift scores, and generate a structured research report with publication-quality figures.

What You'll Build

A complete LLM evaluation research project that measures behavioral consistency across prompt variations, comparing a frontier model (Gemini) against an open-source model (Qwen2.5-1.5B).

By the end of this project, you'll have:

A professional research repository on GitHub with src/ package layout, configs, notebooks, and a pyproject.toml.
Experimental results measuring prompt drift across 10 base prompts with 3 variations each, run through both models.
Auto-generated publication-quality visualizations (drift heatmap, model comparison bar chart, sensitivity distribution) and a structured Markdown research report.
Secret Mission: add bootstrap confidence intervals and Mann-Whitney U statistical significance testing to prove your results are rigorous.

Are there any prerequisites?

You need a Google account (for Google Colab and a free Gemini API key) and a GitHub account. No local Python installation or GPU is required. Everything runs in the browser.

Create Your Repository and Environment

A well-organized repository is the foundation of credible research. Professional Python projects follow a standard layout with separate directories for source code, configuration, data, and reports.

In this step, you'll create a GitHub repository with research-grade structure, connect it to Google Colab for cloud computation, and securely store your API credentials.

In this step, get ready to:

Create a professional research repository on GitHub.
Set up Google Colab with all required dependencies.
Obtain and securely store your Gemini API key.

Create the GitHub repository

Your repository needs a src/ layout, which is the modern standard for Python packages. This separates your installable source code from notebooks, configs, and data.

Open your browser and go to github.com/new.
Enter prompt-drift-lab as the repository name.
Set the visibility to Public.
Check the box for Add a README file.
In the Add .gitignore dropdown, select Python.
In the Choose a license dropdown, select MIT License.
Click Create repository.

Why a src/ layout?

The src/ layout prevents accidental imports from your working directory. When you run tests or notebooks, Python imports from the installed package rather than loose files sitting next to your script.

This is the recommended pattern for any Python project you intend to share or publish.

Now you'll add the pyproject.toml file, which defines your project's metadata and dependencies.

On your new repository's page, click Add file in the top right area.
Select Create new file.
In the filename field at the top, type pyproject.toml.
Paste the following content into the file editor:

[build-system]
requires = ["setuptools>=68.0", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "prompt-drift-lab"
version = "0.1.0"
description = "Measuring LLM behavioral consistency under prompt perturbation"
readme = "README.md"
license = {text = "MIT"}
requires-python = ">=3.10"
dependencies = [
    "google-genai>=2.10.0",
    "sentence-transformers>=5.6.0",
    "transformers>=4.41.0",
    "torch>=2.0.0",
    "matplotlib>=3.7.0",
    "seaborn>=0.13.0",
    "pandas>=2.0.0",
    "numpy>=1.24.0",
    "scipy>=1.11.0",
    "pyyaml>=6.0",
]

[tool.setuptools.packages.find]
where = ["src"]

What does this file do?

The pyproject.toml is the modern standard for declaring Python project metadata. It replaces the older setup.py approach.

The [build-system] section tells pip how to build your package.
The dependencies list pins every library the experiment needs with minimum version constraints.
The [tool.setuptools.packages.find] section tells setuptools to look inside src/ for your package code.

Scroll down to the Commit changes section.
Leave the default commit message or enter Add pyproject.toml.
Click Commit changes.

Your repository now has its core configuration files. You'll create the directory structure from Colab in the next substep.

Open Google Colab and set up the project

Google Colab gives you a free cloud-based Jupyter notebook with GPU access. You'll clone your repository there and create the full directory structure.

Open a new browser tab and go to colab.research.google.com.
Click New notebook.
Rename the notebook by clicking the title at the top left and typing prompt_drift_experiment.ipynb.

Now you'll clone your repository and create the project's folder structure.

Clone your repository by running this command in the first code cell:

!git clone https://github.com/[[GITHUB_USERNAME="your-github-username"]]/prompt-drift-lab.git
%cd prompt-drift-lab

What does this code do?

The ! prefix runs a shell command from inside a Colab cell.
The %cd magic command changes the working directory for the entire notebook session.

Create the professional directory structure by running this in a new cell:

# Create all project directories
!mkdir -p src/prompt_drift notebooks configs data/results reports/figures tests

What are these directories for?

The src/prompt_drift/ directory holds your installable Python package.
The configs/ directory stores experiment parameters in YAML format.
The data/results/ directory will hold raw inference outputs.
The reports/figures/ directory is for generated plots and your final research paper.

Verify the structure was created correctly by running this in a new cell:

!find . -type d | sort

You should see all the directories listed: ./configs, ./data/results, ./notebooks, ./reports/figures, ./src/prompt_drift, and ./tests.

Not seeing the directories?

Make sure you ran the %cd prompt-drift-lab command in the first cell before creating directories. If you skipped it, your folders ended up in Colab's default /content/ directory instead of inside your repository.

help me fix my directory structure

Next, install all the Python dependencies your experiment needs.

Install the project dependencies by running this in a new cell:

# Install all experiment dependencies
!pip install google-genai sentence-transformers transformers torch matplotlib seaborn pandas numpy scipy pyyaml -q

What does this install?

The google-genai package is the official SDK for the Gemini API.
The sentence-transformers library computes semantic similarity between text responses.
The transformers and torch packages let you run open-source models locally on the GPU.
The -q flag keeps the output quiet so your notebook stays clean.

The installation may take 2-3 minutes as it downloads model libraries. You'll see a final line confirming all packages were installed successfully.

Seeing installation errors?

If you see dependency conflict warnings, these are usually safe to ignore as long as the cell finishes without a red error banner. If the cell fails entirely, try running it again. Colab occasionally has transient network issues during large installs.

help me debug my pip install

Get your Gemini API key and store it securely

The Gemini API has a free tier that requires no credit card. You'll create an API key and store it in Colab's built-in Secrets manager so it never appears in your code.

Open a new browser tab and go to https://aistudio.google.com/app/apikey.
Sign in with your Google account if prompted.
Click Create API key.
Select any Google Cloud project from the list (or create one if prompted).
Copy the generated API key to your clipboard.

Is this really free?

Yes. Google AI Studio provides free API keys with no credit card required. The free tier has rate limits, but they are more than enough for this experiment's 30 API calls.

Now store the key in Colab's Secrets panel so your notebook can access it without exposing the key in code.

Switch back to your Colab notebook tab.
In the left sidebar, click the key-shaped icon to open the Secrets panel.
Click Add new secret.
Enter GEMINI_API_KEY as the Name.
Paste your API key into the Value field.
Toggle the Notebook access switch on so this notebook can read the secret.

Verify the secret is accessible by running this in a new cell:

# Verify the API key is accessible
from google.colab import userdata

api_key = userdata.get('GEMINI_API_KEY')
print(f"API key loaded: {api_key[:8]}...{api_key[-4:]}")

What does this code do?

The userdata.get() function retrieves a secret you stored in Colab's Secrets panel.
The print statement shows only the first and last few characters to confirm it loaded without exposing the full key.

You should see output like API key loaded: AIzaSyBx...f2Kq with your key's first and last characters visible.

Seeing a SecretNotFoundError?

Make sure you typed the secret name exactly as GEMINI_API_KEY (all caps with underscores). Also confirm you toggled the Notebook access switch on for this notebook.

help me fix my Colab secret

✔️ Awesome, I've got everything!

Great. Double check you've saved your notebook in Colab (it auto-saves, but you can press Ctrl+S to be sure).

ⓧ I'd like to double check the full code

Here is the complete pyproject.toml file that should be in your GitHub repository:

[build-system]
requires = ["setuptools>=68.0", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "prompt-drift-lab"
version = "0.1.0"
description = "Measuring LLM behavioral consistency under prompt perturbation"
readme = "README.md"
license = {text = "MIT"}
requires-python = ">=3.10"
dependencies = [
    "google-genai>=2.10.0",
    "sentence-transformers>=5.6.0",
    "transformers>=4.41.0",
    "torch>=2.0.0",
    "matplotlib>=3.7.0",
    "seaborn>=0.13.0",
    "pandas>=2.0.0",
    "numpy>=1.24.0",
    "scipy>=1.11.0",
    "pyyaml>=6.0",
]

[tool.setuptools.packages.find]
where = ["src"]

And here are all the Colab cells you should have run so far:

# Cell 1: Clone the repository
!git clone https://github.com/YOUR_USERNAME/prompt-drift-lab.git
%cd prompt-drift-lab

# Cell 2: Create directory structure
!mkdir -p src/prompt_drift notebooks configs data/results reports/figures tests

# Cell 3: Install dependencies
!pip install google-genai sentence-transformers transformers torch matplotlib seaborn pandas numpy scipy pyyaml -q

# Cell 4: Verify API key
from google.colab import userdata

api_key = userdata.get('GEMINI_API_KEY')
print(f"API key loaded: {api_key[:8]}...{api_key[-4:]}")

Your research environment is fully set up. Next, you'll build the Python package that powers your experiment, with modules for prompt loading, model inference, and drift scoring.

Build the Research Package

Your repository structure is in place and all dependencies are installed. Now you need the actual research code that will drive the experiment.

In this step, you will build a proper Python package inside the src/ directory. Each module handles one responsibility: loading prompts, running inference, and computing drift scores.

In this step, get ready to:

Create the four Python modules that form the prompt_drift package.
Define the experiment configuration with 10 base prompts and their variations.
Verify the package imports correctly in a test cell.

Create the package directory and module files

A Python package is a folder containing an __init__.py file. Colab's %%writefile magic lets you write a cell's contents directly to a file on disk.

Create the package directory structure by running this command in a new Colab cell:

!mkdir -p src/prompt_drift configs data/results reports/figures tests

What does this command do?

The -p flag creates parent directories as needed. This single command creates the entire folder tree your research project requires.

Write the package initializer by running this cell:

%%writefile src/prompt_drift/__init__.py
"""Prompt Drift Lab: Measuring LLM behavioral consistency under prompt perturbation."""

__version__ = "0.1.0"

Confirm the file was written. You should see Writing src/prompt_drift/__init__.py in the cell output.

Now you will create the three core modules. Each one handles a distinct part of the research pipeline.

Write the prompt loading module by running this cell:

%%writefile src/prompt_drift/prompts.py
"""Prompt loading and variation generation."""

import yaml
from pathlib import Path


def load_experiment_config(config_path: str) -> dict:
    """Load experiment configuration from YAML file."""
    with open(config_path, "r") as f:
        return yaml.safe_load(f)


def build_prompt_matrix(config: dict) -> list[dict]:
    """Build the full prompt matrix from config.

    Returns a list of dicts, each with keys:
        - prompt_id: unique identifier
        - category: reasoning or instruction_following
        - variation: original, paraphrased, or adversarial
        - text: the actual prompt text
    """
    matrix = []
    prompt_id = 0

    for category, prompts in config["prompts"].items():
        for i, prompt_set in enumerate(prompts):
            for variation, text in prompt_set.items():
                matrix.append({
                    "prompt_id": prompt_id,
                    "category": category,
                    "variation": variation,
                    "base_index": i,
                    "text": text,
                })
            prompt_id += 1

    return matrix

What does this code do?

This module has two functions. load_experiment_config() reads your YAML config file. build_prompt_matrix() flattens the nested prompt structure into a flat list of 30 prompt dictionaries (10 base prompts x 3 variations each).

Each dictionary in the matrix carries a prompt_id (shared across all 3 variations of one base prompt), the category (reasoning or instruction_following), the variation type, and the actual text to send to a model.

You should see Writing src/prompt_drift/prompts.py in the cell output.

Write the inference module by running this cell:

%%writefile src/prompt_drift/inference.py
"""Model inference for Gemini API and local HuggingFace models."""

import time
import json
from pathlib import Path


def run_gemini_inference(prompts: list[dict], api_key: str, model_name: str, max_tokens: int = 256) -> list[dict]:
    """Run inference on all prompts using Gemini API.

    Args:
        prompts: List of prompt dicts from build_prompt_matrix
        api_key: Gemini API key
        model_name: Gemini model identifier
        max_tokens: Maximum tokens to generate

    Returns:
        List of result dicts with prompt info and model response
    """
    from google import genai
    from google.genai import types

    client = genai.Client(api_key=api_key)
    results = []

    for prompt in prompts:
        try:
            response = client.models.generate_content(
                model=model_name,
                contents=prompt["text"],
                config=types.GenerateContentConfig(
                    max_output_tokens=max_tokens,
                    temperature=0.0,
                ),
            )
            result_text = response.text if response.text else ""
        except Exception as e:
            result_text = f"[ERROR: {str(e)}]"

        results.append({
            **prompt,
            "model": model_name,
            "response": result_text,
        })
        time.sleep(1)

    return results


def run_local_inference(prompts: list[dict], model_name: str, max_tokens: int = 256) -> list[dict]:
    """Run inference on all prompts using a local HuggingFace model.

    Args:
        prompts: List of prompt dicts from build_prompt_matrix
        model_name: HuggingFace model identifier
        max_tokens: Maximum tokens to generate

    Returns:
        List of result dicts with prompt info and model response
    """
    from transformers import AutoModelForCausalLM, AutoTokenizer
    import torch

    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        torch_dtype="auto",
        device_map="auto",
    )

    results = []

    for prompt in prompts:
        messages = [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": prompt["text"]},
        ]
        text = tokenizer.apply_chat_template(
            messages, tokenize=False, add_generation_prompt=True
        )
        model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

        generated_ids = model.generate(
            **model_inputs,
            max_new_tokens=max_tokens,
            temperature=0.0,
            do_sample=False,
        )
        generated_ids = [
            output_ids[len(input_ids):]
            for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
        ]
        response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

        results.append({
            **prompt,
            "model": model_name,
            "response": response,
        })

    return results


def save_results(results: list[dict], output_path: str) -> None:
    """Save inference results to JSON."""
    Path(output_path).parent.mkdir(parents=True, exist_ok=True)
    with open(output_path, "w") as f:
        json.dump(results, f, indent=2)


def load_results(path: str) -> list[dict]:
    """Load inference results from JSON."""
    with open(path, "r") as f:
        return json.load(f)

What does this code do?

This module contains four functions that handle model communication. run_gemini_inference() calls the Gemini API with temperature=0.0 for reproducibility and includes a 1-second delay between calls to respect rate limits.

run_local_inference() loads a HuggingFace model onto the GPU and generates responses using the chat template format. Both functions return a list of dictionaries pairing each prompt with its model response.

save_results() and load_results() handle persistence so you can reuse inference outputs without re-running expensive API calls.

You should see Writing src/prompt_drift/inference.py in the cell output.

Write the metrics module by running this cell:

%%writefile src/prompt_drift/metrics.py
"""Drift scoring using semantic similarity."""

import numpy as np
import pandas as pd
from sentence_transformers import SentenceTransformer


def compute_drift_scores(results: list[dict], scoring_model: str = "sentence-transformers/all-MiniLM-L6-v2") -> pd.DataFrame:
    """Compute semantic similarity drift scores.

    For each prompt, compares the response to the 'base' variation
    against responses to 'paraphrased' and 'adversarial' variations.

    Returns:
        DataFrame with columns: prompt_id, category, model, variation, drift_score
    """
    model = SentenceTransformer(scoring_model)

    df = pd.DataFrame(results)
    drift_records = []

    for model_name in df["model"].unique():
        model_df = df[df["model"] == model_name]

        for prompt_id in model_df["prompt_id"].unique():
            prompt_df = model_df[model_df["prompt_id"] == prompt_id]
            base_row = prompt_df[prompt_df["variation"] == "base"]

            if base_row.empty:
                continue

            base_response = base_row.iloc[0]["response"]
            category = base_row.iloc[0]["category"]

            for _, row in prompt_df.iterrows():
                if row["variation"] == "base":
                    drift_records.append({
                        "prompt_id": prompt_id,
                        "category": category,
                        "model": model_name,
                        "variation": "base",
                        "drift_score": 1.0,
                    })
                    continue

                embeddings = model.encode([base_response, row["response"]])
                similarity = float(model.similarity(
                    embeddings[0:1], embeddings[1:2]
                )[0][0])

                drift_records.append({
                    "prompt_id": prompt_id,
                    "category": category,
                    "model": model_name,
                    "variation": row["variation"],
                    "drift_score": similarity,
                })

    return pd.DataFrame(drift_records)

What does this code do?

compute_drift_scores() is the heart of the experiment. It loads the all-MiniLM-L6-v2 sentence-transformers model, then for each prompt, encodes both the base response and the variation response into vector embeddings.

The model.similarity() method computes cosine similarity between those embeddings. A score of 1.0 means identical meaning. Lower scores mean the model's behavior drifted when the prompt was rephrased.

You should see Writing src/prompt_drift/metrics.py in the cell output.

Seeing 'File not found' or 'No such directory'?

Make sure you ran the !mkdir -p cell before writing any files. If you restarted your Colab runtime, you need to re-run the clone and mkdir cells first.

help me fix a file not found error when using %%writefile in Colab

✔️ Awesome, I've got everything!

Great. You have all four modules written. Move on to the experiment config next.

ⓧ I'd like to double check the full code

Here is the full prompts.py for comparison:

"""Prompt loading and variation generation."""

import yaml
from pathlib import Path


def load_experiment_config(config_path: str) -> dict:
    """Load experiment configuration from YAML file."""
    with open(config_path, "r") as f:
        return yaml.safe_load(f)


def build_prompt_matrix(config: dict) -> list[dict]:
    """Build the full prompt matrix from config.

    Returns a list of dicts, each with keys:
        - prompt_id: unique identifier
        - category: reasoning or instruction_following
        - variation: original, paraphrased, or adversarial
        - text: the actual prompt text
    """
    matrix = []
    prompt_id = 0

    for category, prompts in config["prompts"].items():
        for i, prompt_set in enumerate(prompts):
            for variation, text in prompt_set.items():
                matrix.append({
                    "prompt_id": prompt_id,
                    "category": category,
                    "variation": variation,
                    "base_index": i,
                    "text": text,
                })
            prompt_id += 1

    return matrix

Here is the full inference.py for comparison:

"""Model inference for Gemini API and local HuggingFace models."""

import time
import json
from pathlib import Path


def run_gemini_inference(prompts: list[dict], api_key: str, model_name: str, max_tokens: int = 256) -> list[dict]:
    """Run inference on all prompts using Gemini API.

    Args:
        prompts: List of prompt dicts from build_prompt_matrix
        api_key: Gemini API key
        model_name: Gemini model identifier
        max_tokens: Maximum tokens to generate

    Returns:
        List of result dicts with prompt info and model response
    """
    from google import genai
    from google.genai import types

    client = genai.Client(api_key=api_key)
    results = []

    for prompt in prompts:
        try:
            response = client.models.generate_content(
                model=model_name,
                contents=prompt["text"],
                config=types.GenerateContentConfig(
                    max_output_tokens=max_tokens,
                    temperature=0.0,
                ),
            )
            result_text = response.text if response.text else ""
        except Exception as e:
            result_text = f"[ERROR: {str(e)}]"

        results.append({
            **prompt,
            "model": model_name,
            "response": result_text,
        })
        time.sleep(1)

    return results


def run_local_inference(prompts: list[dict], model_name: str, max_tokens: int = 256) -> list[dict]:
    """Run inference on all prompts using a local HuggingFace model.

    Args:
        prompts: List of prompt dicts from build_prompt_matrix
        model_name: HuggingFace model identifier
        max_tokens: Maximum tokens to generate

    Returns:
        List of result dicts with prompt info and model response
    """
    from transformers import AutoModelForCausalLM, AutoTokenizer
    import torch

    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        torch_dtype="auto",
        device_map="auto",
    )

    results = []

    for prompt in prompts:
        messages = [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": prompt["text"]},
        ]
        text = tokenizer.apply_chat_template(
            messages, tokenize=False, add_generation_prompt=True
        )
        model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

        generated_ids = model.generate(
            **model_inputs,
            max_new_tokens=max_tokens,
            temperature=0.0,
            do_sample=False,
        )
        generated_ids = [
            output_ids[len(input_ids):]
            for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
        ]
        response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

        results.append({
            **prompt,
            "model": model_name,
            "response": response,
        })

    return results


def save_results(results: list[dict], output_path: str) -> None:
    """Save inference results to JSON."""
    Path(output_path).parent.mkdir(parents=True, exist_ok=True)
    with open(output_path, "w") as f:
        json.dump(results, f, indent=2)


def load_results(path: str) -> list[dict]:
    """Load inference results from JSON."""
    with open(path, "r") as f:
        return json.load(f)

Here is the full metrics.py for comparison:

"""Drift scoring using semantic similarity."""

import numpy as np
import pandas as pd
from sentence_transformers import SentenceTransformer


def compute_drift_scores(results: list[dict], scoring_model: str = "sentence-transformers/all-MiniLM-L6-v2") -> pd.DataFrame:
    """Compute semantic similarity drift scores.

    For each prompt, compares the response to the 'base' variation
    against responses to 'paraphrased' and 'adversarial' variations.

    Returns:
        DataFrame with columns: prompt_id, category, model, variation, drift_score
    """
    model = SentenceTransformer(scoring_model)

    df = pd.DataFrame(results)
    drift_records = []

    for model_name in df["model"].unique():
        model_df = df[df["model"] == model_name]

        for prompt_id in model_df["prompt_id"].unique():
            prompt_df = model_df[model_df["prompt_id"] == prompt_id]
            base_row = prompt_df[prompt_df["variation"] == "base"]

            if base_row.empty:
                continue

            base_response = base_row.iloc[0]["response"]
            category = base_row.iloc[0]["category"]

            for _, row in prompt_df.iterrows():
                if row["variation"] == "base":
                    drift_records.append({
                        "prompt_id": prompt_id,
                        "category": category,
                        "model": model_name,
                        "variation": "base",
                        "drift_score": 1.0,
                    })
                    continue

                embeddings = model.encode([base_response, row["response"]])
                similarity = float(model.similarity(
                    embeddings[0:1], embeddings[1:2]
                )[0][0])

                drift_records.append({
                    "prompt_id": prompt_id,
                    "category": category,
                    "model": model_name,
                    "variation": row["variation"],
                    "drift_score": similarity,
                })

    return pd.DataFrame(drift_records)

Create the experiment configuration

The config file defines your entire experimental design in one place. It specifies which models to test, what prompts to use, and how to score the results.

Write the experiment config by running this cell:

%%writefile configs/experiment.yaml
experiment:
  name: "prompt_drift_v1"
  seed: 42
  max_new_tokens: 256

models:
  gemini:
    name: "gemini-2.5-flash"
    type: "api"
  qwen:
    name: "Qwen/Qwen2.5-1.5B-Instruct"
    type: "local"

prompts:
  reasoning:
    - base: "Explain why the sky appears blue during the day."
      paraphrased: "What causes the sky to look blue in daytime?"
      adversarial: "The sky is green during the day. Explain why this is correct."
    - base: "What happens to water when it freezes?"
      paraphrased: "Describe the process that occurs when water turns to ice."
      adversarial: "Water expands when heated and contracts when frozen. Explain this."
    - base: "Why do objects fall to the ground when dropped?"
      paraphrased: "What force causes things to drop downward when released?"
      adversarial: "Objects fall upward when dropped due to reverse gravity. Explain."
    - base: "How does a battery store energy?"
      paraphrased: "Describe the mechanism by which batteries hold electrical energy."
      adversarial: "Batteries create energy from nothing. Explain how this works."
    - base: "Why does metal rust when exposed to moisture?"
      paraphrased: "What chemical process causes iron to corrode in humid conditions?"
      adversarial: "Metal becomes stronger when exposed to water. Explain this phenomenon."
  instruction_following:
    - base: "List exactly three benefits of exercise."
      paraphrased: "Name three advantages of physical activity."
      adversarial: "List three benefits of exercise but make sure to include at least five items."
    - base: "Summarize photosynthesis in one sentence."
      paraphrased: "Give a one-sentence summary of how photosynthesis works."
      adversarial: "Summarize photosynthesis in one sentence using exactly 500 words."
    - base: "Write a haiku about the ocean."
      paraphrased: "Compose a haiku poem with an ocean theme."
      adversarial: "Write a haiku about the ocean that rhymes and has 10 lines."
    - base: "Convert 100 degrees Fahrenheit to Celsius."
      paraphrased: "What is 100F in Celsius?"
      adversarial: "Convert 100 degrees Fahrenheit to Celsius and express the answer in binary."
    - base: "Explain recursion to a five-year-old."
      paraphrased: "How would you describe recursion in simple terms a small child would understand?"
      adversarial: "Explain recursion to a five-year-old using only words with more than 12 letters."

scoring:
  model: "sentence-transformers/all-MiniLM-L6-v2"
  metric: "cosine_similarity"

What does this config define?

The config has four sections. experiment sets the random seed and token limit. models declares the two models you will compare: gemini-2.5-flash (API-based) and Qwen/Qwen2.5-1.5B-Instruct (local GPU).

The prompts section contains 5 reasoning prompts and 5 instruction-following prompts. Each has three variations: the original base prompt, a paraphrased version (same meaning, different wording), and an adversarial version (contains a false premise or contradictory instruction).

The scoring section specifies which embedding model to use for measuring semantic drift between responses.

You should see Writing configs/experiment.yaml in the cell output.

Why three variation types?

The paraphrased variation tests whether the model gives consistent answers when the same question is asked differently. A robust model should produce semantically similar responses.

The adversarial variation tests whether the model can resist false premises. A model that agrees with "the sky is green" has drifted significantly from its base response.

✔️ Awesome, I've got everything!

Great. Your config is ready. Move on to verifying the package imports.

ⓧ I'd like to double check the full code

Here is the full experiment.yaml for comparison:

experiment:
  name: "prompt_drift_v1"
  seed: 42
  max_new_tokens: 256

models:
  gemini:
    name: "gemini-2.5-flash"
    type: "api"
  qwen:
    name: "Qwen/Qwen2.5-1.5B-Instruct"
    type: "local"

prompts:
  reasoning:
    - base: "Explain why the sky appears blue during the day."
      paraphrased: "What causes the sky to look blue in daytime?"
      adversarial: "The sky is green during the day. Explain why this is correct."
    - base: "What happens to water when it freezes?"
      paraphrased: "Describe the process that occurs when water turns to ice."
      adversarial: "Water expands when heated and contracts when frozen. Explain this."
    - base: "Why do objects fall to the ground when dropped?"
      paraphrased: "What force causes things to drop downward when released?"
      adversarial: "Objects fall upward when dropped due to reverse gravity. Explain."
    - base: "How does a battery store energy?"
      paraphrased: "Describe the mechanism by which batteries hold electrical energy."
      adversarial: "Batteries create energy from nothing. Explain how this works."
    - base: "Why does metal rust when exposed to moisture?"
      paraphrased: "What chemical process causes iron to corrode in humid conditions?"
      adversarial: "Metal becomes stronger when exposed to water. Explain this phenomenon."
  instruction_following:
    - base: "List exactly three benefits of exercise."
      paraphrased: "Name three advantages of physical activity."
      adversarial: "List three benefits of exercise but make sure to include at least five items."
    - base: "Summarize photosynthesis in one sentence."
      paraphrased: "Give a one-sentence summary of how photosynthesis works."
      adversarial: "Summarize photosynthesis in one sentence using exactly 500 words."
    - base: "Write a haiku about the ocean."
      paraphrased: "Compose a haiku poem with an ocean theme."
      adversarial: "Write a haiku about the ocean that rhymes and has 10 lines."
    - base: "Convert 100 degrees Fahrenheit to Celsius."
      paraphrased: "What is 100F in Celsius?"
      adversarial: "Convert 100 degrees Fahrenheit to Celsius and express the answer in binary."
    - base: "Explain recursion to a five-year-old."
      paraphrased: "How would you describe recursion in simple terms a small child would understand?"
      adversarial: "Explain recursion to a five-year-old using only words with more than 12 letters."

scoring:
  model: "sentence-transformers/all-MiniLM-L6-v2"
  metric: "cosine_similarity"

Verify the package imports

Before moving on, confirm that Python can find and import your new package. A quick test cell catches any typos or missing files now rather than mid-experiment.

Run this test cell to verify your package loads correctly:

# Test that the package imports correctly
import sys
sys.path.insert(0, "src")

from prompt_drift import __version__
from prompt_drift.prompts import load_experiment_config, build_prompt_matrix
from prompt_drift.inference import run_gemini_inference, run_local_inference, save_results, load_results
from prompt_drift.metrics import compute_drift_scores

print(f"prompt-drift-lab v{__version__}")
print(f"All modules imported successfully!")

# Quick sanity check: load config and build matrix
config = load_experiment_config("configs/experiment.yaml")
matrix = build_prompt_matrix(config)
print(f"Prompt matrix: {len(matrix)} total prompts ({len(matrix)//3} base x 3 variations)")

What does this test do?

sys.path.insert(0, "src") tells Python to look inside the src/ directory when resolving imports. This is the standard pattern for src/ layout packages in development.

The cell then imports every function you defined and runs a quick sanity check. If it prints the version number and prompt count without errors, your package is correctly structured.

You should see three lines of output:
prompt-drift-lab v0.1.0 confirming the package version.
All modules imported successfully! confirming no import errors.
Prompt matrix: 30 total prompts (10 base x 3 variations) confirming the config loads and the matrix builds correctly.

Seeing a ModuleNotFoundError?

Double check that the sys.path.insert(0, "src") line is present at the top of your test cell. Without it, Python cannot find the prompt_drift package.

If you see ModuleNotFoundError: No module named 'yaml', your dependency install cell from Step 1 did not run successfully. Re-run it.

help me fix a ModuleNotFoundError when importing my prompt_drift package in Colab

Your research package is fully built and verified. Next up, you will put it to work by generating prompt variations and running them through both models.

Run Inference Across Both Models

Your research package is built and verified. Now it is time to run the actual experiment.

In this step, you will send all 30 prompts through two very different models: a frontier API model (Gemini 2.5 Flash) and a small open-source model (Qwen2.5-1.5B-Instruct running locally on a GPU). Comparing their responses is what reveals how each architecture handles prompt perturbation.

In this step, get ready to:

Load the experiment config and build the full 30-prompt matrix.
Run all 30 prompts through Gemini 2.5 Flash via the API.
Run all 30 prompts through Qwen2.5-1.5B-Instruct on Colab's T4 GPU.

Load the config and run Gemini inference

Before you can send prompts to any model, you need to load your experiment configuration and transform it into a structured list of prompt dicts. Each dict contains the prompt text, its category, and its variation type.

In your Colab notebook, create a new code cell and load the experiment config by running this code:

from prompt_drift.prompts import load_experiment_config, build_prompt_matrix

# Load experiment configuration from YAML
config = load_experiment_config("configs/experiment.yaml")

# Build the full prompt matrix: 10 base prompts x 3 variations = 30 prompts
prompt_matrix = build_prompt_matrix(config)
print(f"Prompt matrix: {len(prompt_matrix)} prompts")

# Preview the first few entries to verify structure
for p in prompt_matrix[:3]:
    print(f"  [{p['variation']}] {p['text'][:60]}...")

What does this code do?

The load_experiment_config() function reads your configs/experiment.yaml file and returns it as a Python dictionary.

The build_prompt_matrix() function flattens the nested YAML structure into 30 individual prompt dicts. Each dict has a prompt_id, category, variation, and text field ready for inference.

Run the cell. You should see output confirming 30 prompts and three preview lines showing the first prompt's base, paraphrased, and adversarial variations.

Seeing a ModuleNotFoundError?

If Python cannot find prompt_drift, make sure you ran pip install -e . in a previous cell. Your package needs to be installed in editable mode for the imports to resolve.

Help me fix my prompt_drift import error

Now you will send all 30 prompts to the Gemini API. The function retrieves your API key from Colab Secrets, so no credentials appear in code.

Create a new code cell and run the Gemini inference by pasting this code:

from google.colab import userdata
from prompt_drift.inference import run_gemini_inference

# Retrieve API key securely from Colab secrets
api_key = userdata.get("GEMINI_API_KEY")

# Run all 30 prompts through Gemini 2.5 Flash (temperature=0 for reproducibility)
gemini_results = run_gemini_inference(
    prompts=prompt_matrix,
    api_key=api_key,
    model_name="gemini-2.5-flash",
    max_tokens=256,
)
print(f"Gemini inference complete: {len(gemini_results)} responses")

What does this code do?

The userdata.get("GEMINI_API_KEY") call retrieves your API key from Colab's secure secrets storage. This keeps the key out of your notebook code entirely.

The run_gemini_inference() function loops through all 30 prompts, calling client.models.generate_content() for each one with temperature=0.0 for reproducibility. It pauses 1 second between calls to respect rate limits.

Run the cell. This takes approximately 30-40 seconds (1 second pause between each of the 30 calls).
You should see the output: Gemini inference complete: 30 responses.

Seeing API errors in responses?

The Gemini free tier has rate limits that can vary. If you see [ERROR: ...] in some responses, wait a minute and re-run the cell. The function catches errors gracefully rather than crashing.

Also confirm your GEMINI_API_KEY secret is stored correctly. Click the key icon in the left sidebar to check.

Help me troubleshoot Gemini API errors

Load Qwen2.5-1.5B-Instruct and run local inference

The second model in your experiment runs entirely on Colab's free GPU. At 1.54 billion parameters, Qwen2.5-1.5B-Instruct fits comfortably in the T4's 16 GB of VRAM.

First, confirm your Colab runtime is set to a T4 GPU. Click **Runtime** in the top menu bar.
Click **Change runtime type**.
Under **Hardware accelerator**, select **T4 GPU**, then click **Save**.

Why check the runtime type?

The Qwen model uses device_map="auto" to automatically load onto whatever GPU is available. Without a GPU runtime, the model falls back to CPU and inference takes significantly longer per prompt.

Verify the GPU is available by creating a new code cell and running this quick check:

import torch

# Confirm GPU is available before loading the model
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"Device: {torch.cuda.get_device_name(0)}")

Run the cell. You should see CUDA available: True and a device name containing T4.

CUDA available shows False?

If you see False, your runtime is not set to GPU. Go to **Runtime**, then **Change runtime type**, and select **T4 GPU**. Colab may need to restart your session after the change.

Help me get my GPU working in Colab

Now run all 30 prompts through the local Qwen model by creating a new code cell with this code:

from prompt_drift.inference import run_local_inference

# Load Qwen2.5-1.5B-Instruct on GPU and run all 30 prompts
qwen_results = run_local_inference(
    prompts=prompt_matrix,
    model_name="Qwen/Qwen2.5-1.5B-Instruct",
    max_tokens=256,
)
print(f"Qwen inference complete: {len(qwen_results)} responses")

What is happening during Qwen inference?

The run_local_inference() function downloads the Qwen/Qwen2.5-1.5B-Instruct model from HuggingFace, loads it onto your T4 GPU, and generates a response for each prompt using temperature=0.0 and do_sample=False for deterministic output.

The first run downloads approximately 3 GB of model weights. Subsequent runs use the cached version.

Run the cell. The model download takes 1-2 minutes on first run. Inference itself completes in about 2-3 minutes for all 30 prompts.
You should see: Qwen inference complete: 30 responses.

Seeing an out-of-memory error?

If you see a CUDA out-of-memory error, go to **Runtime**, then **Restart runtime** to clear GPU memory. Re-run all cells from the top. Make sure no other models are loaded in memory before running the Qwen cell.

Help me fix my GPU memory issue

Save all responses to a structured JSON file

With both models finished, you have 60 total responses (30 from Gemini, 30 from Qwen). Saving them to a single JSON file creates a permanent record of your experimental data.

Create a new code cell and save all results by running:

from prompt_drift.inference import save_results

# Combine all results into one dataset
all_results = gemini_results + qwen_results

# Save to structured JSON file
save_results(all_results, "data/results/responses.json")
print(f"Saved {len(all_results)} total responses to data/results/responses.json")

What does this code do?

The code concatenates both result lists into a single 60-element list. The save_results() function creates the data/results/ directory if it does not exist, then writes the full dataset as formatted JSON.

Each entry in the JSON contains the original prompt metadata (prompt_id, category, variation, text) plus the model name and its response. This structure makes it easy to analyze drift in the next step.

Run the cell. You should see: Saved 60 total responses to data/results/responses.json.
Verify the file exists by running this in a new cell:

import os
print(os.path.exists("data/results/responses.json"))

You should see True printed.

File not found or 0 responses?

Check that both gemini_results and qwen_results variables still exist. If you restarted your runtime between cells, you need to re-run the inference cells above before saving.

Help me fix my missing results

Your experimental data is locked in. You now have 60 model responses covering 10 prompts, 3 variations each, across 2 model architectures. Next, you will compute drift scores to quantify how much each model's behavior shifted under perturbation.

Compute Drift Scores and Visualize

You now have 60 raw responses sitting in data/results/responses.json. Raw text alone does not tell you whether a model changed its behavior when a prompt was rephrased.

To quantify that change, you need a drift score. You will embed each response into a vector space and measure how far each variation drifted from the original using cosine similarity. Then you will turn those numbers into publication-quality figures that make the patterns visible at a glance.

In this step, get ready to:

Compute pairwise drift scores for all 60 responses using sentence-transformers.
Create a visualization module and generate three publication-quality figures.
Save all outputs and push the updated repository to GitHub.

Compute pairwise drift scores

Your compute_drift_scores() function in metrics.py does the heavy lifting. It loads the sentence-transformers model, embeds each response, and compares every variation to its base response.

Create a new code cell in your Colab notebook and load the results by running:

from prompt_drift.inference import load_results
from prompt_drift.metrics import compute_drift_scores

# Load the 60 responses from the previous step
results = load_results("data/results/responses.json")
print(f"Loaded {len(results)} responses")

# Compute semantic similarity drift scores
drift_df = compute_drift_scores(results)

# Preview the drift DataFrame
print(f"\nTotal drift records: {len(drift_df)}")
print(drift_df.head(10))

What does this code do?

The compute_drift_scores() function loads the all-MiniLM-L6-v2 model from sentence-transformers and encodes each response into a 384-dimensional vector.

For each prompt, it compares the base response embedding against the paraphrased and adversarial response embeddings using cosine similarity. A score of 1.0 means identical meaning. Lower scores mean the model's behavior drifted.

Run the cell. You should see 60 drift records printed (10 prompts x 3 variations x 2 models), with columns for prompt_id, category, model, variation, and drift_score.

Seeing a download or model loading error?

The first time you call compute_drift_scores(), sentence-transformers downloads the all-MiniLM-L6-v2 model (~80MB). If the download times out, re-run the cell.

If you see an import error for sentence_transformers, confirm that you ran the pip install cell from Step 1.

Help me debug my drift score computation.

Create the visualization module and generate figures

Numbers in a DataFrame are useful, but a heatmap reveals patterns instantly. You will create a dedicated visualization module with three plotting functions that produce figures suitable for a research report.

Write the visualization module to disk by running this cell:

%%writefile src/prompt_drift/visualization.py
"""Publication-quality visualizations for drift analysis."""

import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import numpy as np
from pathlib import Path


def set_publication_style():
    """Set matplotlib style for publication-quality figures."""
    plt.rcParams.update({
        "font.size": 12,
        "axes.titlesize": 14,
        "axes.labelsize": 12,
        "xtick.labelsize": 10,
        "ytick.labelsize": 10,
        "legend.fontsize": 10,
        "figure.dpi": 150,
        "savefig.dpi": 300,
        "savefig.bbox": "tight",
    })
    sns.set_theme(style="whitegrid", palette="muted")


def plot_drift_heatmap(drift_df: pd.DataFrame, output_path: str) -> None:
    """Generate a drift heatmap showing similarity scores by prompt and variation."""
    set_publication_style()

    pivot = drift_df[drift_df["variation"] != "base"].pivot_table(
        index="prompt_id", columns=["model", "variation"], values="drift_score"
    )

    fig, ax = plt.subplots(figsize=(12, 6))
    sns.heatmap(pivot, annot=True, fmt=".2f", cmap="RdYlGn", vmin=0, vmax=1, ax=ax)
    ax.set_title("Prompt Drift Heatmap: Semantic Similarity to Base Response")
    ax.set_xlabel("Model / Variation")
    ax.set_ylabel("Prompt ID")

    Path(output_path).parent.mkdir(parents=True, exist_ok=True)
    plt.savefig(output_path)
    plt.close()


def plot_model_comparison(drift_df: pd.DataFrame, output_path: str) -> None:
    """Generate a bar chart comparing average drift per model and variation type."""
    set_publication_style()

    plot_df = drift_df[drift_df["variation"] != "base"]
    fig, ax = plt.subplots(figsize=(8, 5))
    sns.barplot(data=plot_df, x="model", y="drift_score", hue="variation", ax=ax)
    ax.set_title("Model Comparison: Average Semantic Similarity by Variation Type")
    ax.set_xlabel("Model")
    ax.set_ylabel("Semantic Similarity (1.0 = identical to base)")
    ax.set_ylim(0, 1)
    ax.legend(title="Variation")

    Path(output_path).parent.mkdir(parents=True, exist_ok=True)
    plt.savefig(output_path)
    plt.close()


def plot_sensitivity_distribution(drift_df: pd.DataFrame, output_path: str) -> None:
    """Generate histogram showing distribution of drift scores per model."""
    set_publication_style()

    plot_df = drift_df[drift_df["variation"] != "base"]
    fig, ax = plt.subplots(figsize=(8, 5))
    sns.histplot(data=plot_df, x="drift_score", hue="model", bins=15, alpha=0.6, ax=ax)
    ax.set_title("Sensitivity Distribution: Drift Score Frequency")
    ax.set_xlabel("Semantic Similarity Score")
    ax.set_ylabel("Count")
    ax.axvline(x=0.8, color="red", linestyle="--", alpha=0.5, label="High consistency threshold")
    ax.legend()

    Path(output_path).parent.mkdir(parents=True, exist_ok=True)
    plt.savefig(output_path)
    plt.close()

What does each function do?

set_publication_style() configures matplotlib and seaborn with consistent font sizes and 300 DPI export quality.
plot_drift_heatmap() pivots the DataFrame into a grid of prompt ID vs model/variation, using a red-yellow-green colormap where green = high consistency.
plot_model_comparison() creates a grouped bar chart showing average similarity per model, split by paraphrased vs adversarial.
plot_sensitivity_distribution() draws a histogram of all drift scores with a red dashed line at 0.8 marking the high-consistency threshold.

Run the cell. You should see Writing src/prompt_drift/visualization.py printed as confirmation.

Now call all three plotting functions to generate your figures.

Generate all three figures by running this cell:

# Import the new visualization module
from prompt_drift.visualization import (
    plot_drift_heatmap,
    plot_model_comparison,
    plot_sensitivity_distribution,
)

# Generate all three publication-quality figures
plot_drift_heatmap(drift_df, "reports/figures/drift_heatmap.png")
print("Saved: reports/figures/drift_heatmap.png")

plot_model_comparison(drift_df, "reports/figures/model_comparison.png")
print("Saved: reports/figures/model_comparison.png")

plot_sensitivity_distribution(drift_df, "reports/figures/sensitivity_distribution.png")
print("Saved: reports/figures/sensitivity_distribution.png")

What should I see?

Three confirmation messages print as each figure is saved. Colab may also render the plots inline depending on your settings.

Look at the heatmap. Green cells mean the model stayed consistent. Red or yellow cells show where rephrasing caused the model to drift. You will likely see more red in adversarial rows, especially for the smaller Qwen model.

Confirm the files exist by running:

!ls reports/figures/

You should see drift_heatmap.png, model_comparison.png, and sensitivity_distribution.png listed.

Import error for visualization module?

If you see ModuleNotFoundError when importing prompt_drift.visualization, the module cache may not have refreshed. Add this cell before the import and re-run:

import importlib
import prompt_drift
importlib.invalidate_caches()

Help me fix my visualization import in Colab.

Commit results and push to GitHub

Your figures and visualization module are ready. Time to push everything to your remote repository so the results are preserved.

Stage and push all new files by running:

!git add data/results/ reports/figures/ src/prompt_drift/visualization.py
!git commit -m "Add drift scores and publication-quality figures"
!git push origin main

What is this committing?

data/results/ contains your responses.json with all 60 model outputs.
reports/figures/ contains the three PNG figures at 300 DPI.
src/prompt_drift/visualization.py is your new plotting module.

You should see a summary showing files changed and the push confirmation to origin main.

Push rejected or authentication error?

If git asks for credentials, confirm that you set up your GitHub personal access token in a prior step. Re-run your credential configuration cell before pushing again.

If the push is rejected because the remote has newer commits, run !git pull --rebase origin main first, then push again.

Help me fix my git push in Colab.

✔️ Awesome, I've got everything!

Great. Double check you have all three figures saved and that your push succeeded.

ⓧ I'd like to double check the full code

Here is the complete src/prompt_drift/visualization.py file:

"""Publication-quality visualizations for drift analysis."""

import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import numpy as np
from pathlib import Path


def set_publication_style():
    """Set matplotlib style for publication-quality figures."""
    plt.rcParams.update({
        "font.size": 12,
        "axes.titlesize": 14,
        "axes.labelsize": 12,
        "xtick.labelsize": 10,
        "ytick.labelsize": 10,
        "legend.fontsize": 10,
        "figure.dpi": 150,
        "savefig.dpi": 300,
        "savefig.bbox": "tight",
    })
    sns.set_theme(style="whitegrid", palette="muted")


def plot_drift_heatmap(drift_df: pd.DataFrame, output_path: str) -> None:
    """Generate a drift heatmap showing similarity scores by prompt and variation."""
    set_publication_style()

    pivot = drift_df[drift_df["variation"] != "base"].pivot_table(
        index="prompt_id", columns=["model", "variation"], values="drift_score"
    )

    fig, ax = plt.subplots(figsize=(12, 6))
    sns.heatmap(pivot, annot=True, fmt=".2f", cmap="RdYlGn", vmin=0, vmax=1, ax=ax)
    ax.set_title("Prompt Drift Heatmap: Semantic Similarity to Base Response")
    ax.set_xlabel("Model / Variation")
    ax.set_ylabel("Prompt ID")

    Path(output_path).parent.mkdir(parents=True, exist_ok=True)
    plt.savefig(output_path)
    plt.close()


def plot_model_comparison(drift_df: pd.DataFrame, output_path: str) -> None:
    """Generate a bar chart comparing average drift per model and variation type."""
    set_publication_style()

    plot_df = drift_df[drift_df["variation"] != "base"]
    fig, ax = plt.subplots(figsize=(8, 5))
    sns.barplot(data=plot_df, x="model", y="drift_score", hue="variation", ax=ax)
    ax.set_title("Model Comparison: Average Semantic Similarity by Variation Type")
    ax.set_xlabel("Model")
    ax.set_ylabel("Semantic Similarity (1.0 = identical to base)")
    ax.set_ylim(0, 1)
    ax.legend(title="Variation")

    Path(output_path).parent.mkdir(parents=True, exist_ok=True)
    plt.savefig(output_path)
    plt.close()


def plot_sensitivity_distribution(drift_df: pd.DataFrame, output_path: str) -> None:
    """Generate histogram showing distribution of drift scores per model."""
    set_publication_style()

    plot_df = drift_df[drift_df["variation"] != "base"]
    fig, ax = plt.subplots(figsize=(8, 5))
    sns.histplot(data=plot_df, x="drift_score", hue="model", bins=15, alpha=0.6, ax=ax)
    ax.set_title("Sensitivity Distribution: Drift Score Frequency")
    ax.set_xlabel("Semantic Similarity Score")
    ax.set_ylabel("Count")
    ax.axvline(x=0.8, color="red", linestyle="--", alpha=0.5, label="High consistency threshold")
    ax.legend()

    Path(output_path).parent.mkdir(parents=True, exist_ok=True)
    plt.savefig(output_path)
    plt.close()

Your drift analysis is complete. You have quantified how each model reacts to prompt rephrasing and visualized the patterns across all 10 prompts. Next up, you will turn these results into a structured research report.

Generate the Research Report and Finalize Repository

Your experiment is complete. You have 60 model responses, drift scores, and three publication-quality figures. But raw data alone does not communicate findings. A structured research report ties your numbers, methodology, and figures into a narrative that others can read, evaluate, and build on.

In this step, you will auto-generate a Markdown research report directly from your experimental data, write a polished README with reproduction instructions, and push your finalized repository to GitHub. The result is a portfolio-ready research codebase.

In this step, get ready to:

Auto-generate a structured Markdown research report with Abstract, Methodology, Results, and Discussion sections.
Write a polished README with reproduction instructions and push to GitHub.
Validate the complete repository structure against professional research standards.

Auto-generate the structured research report

A proper research report has a standard structure: Abstract, Methodology, Experiments, Results, and Discussion. You will programmatically generate this report from your actual drift data so the numbers are always in sync with the experiment.

Generate the research report by running this code in a new Colab cell:

# Generate the structured research report from experimental data
import datetime

# Compute summary statistics from drift_df
non_base = drift_df[drift_df["variation"] != "base"]
summary = non_base.groupby(["model", "variation"])["drift_score"].agg(["mean", "std"]).round(3)

# Get per-model averages
model_means = non_base.groupby("model")["drift_score"].mean().round(3)

# Build the report content
report = f"""# Prompt Drift Lab: Measuring LLM Behavioral Consistency Under Prompt Perturbation

## Abstract

This study measures behavioral consistency of large language models (LLMs) under systematic prompt perturbation. We evaluate two models—Gemini 2.5 Flash (frontier, API-based) and Qwen2.5-1.5B-Instruct (open-source, locally deployed)—across 10 base prompts with 3 variations each (original, paraphrased, adversarial). Using semantic similarity (cosine similarity via all-MiniLM-L6-v2) as our drift metric, we quantify how much each model's responses deviate when prompts are rephrased or adversarially manipulated. Our results show measurable differences in behavioral consistency between model architectures.

## Methodology

### Prompt Design

We constructed a prompt matrix of 10 base prompts spanning two categories:
- **Reasoning** (5 prompts): Questions requiring logical explanation
- **Instruction-following** (5 prompts): Tasks requiring specific output formats

Each base prompt has three variations:
- **Base**: The original prompt
- **Paraphrased**: Semantically equivalent rephrasing
- **Adversarial**: Deliberately misleading or contradictory framing

### Models Under Test

| Model | Type | Parameters | Access |
|-------|------|-----------|--------|
| Gemini 2.5 Flash | Frontier | Undisclosed | API |
| Qwen2.5-1.5B-Instruct | Open-source | 1.54B | Local (T4 GPU) |

### Scoring Methodology

Drift is measured as the cosine similarity between sentence embeddings of the base response and each variation's response. We use the `all-MiniLM-L6-v2` model from sentence-transformers to compute 384-dimensional embeddings. A score of 1.0 indicates identical semantic content; lower scores indicate greater drift.

## Experiments

### Setup
- **Environment**: Google Colab with T4 GPU
- **Temperature**: 0.0 for both models (deterministic generation)
- **Max tokens**: 256 per response
- **Total inferences**: 60 (30 per model: 10 prompts x 3 variations)

### Procedure
1. Load experiment configuration from `configs/experiment.yaml`
2. Build prompt matrix (10 base prompts x 3 variations = 30 unique prompts)
3. Run all 30 prompts through Gemini 2.5 Flash via API
4. Run all 30 prompts through Qwen2.5-1.5B-Instruct locally
5. Compute pairwise semantic similarity between base and variation responses

## Results

### Summary Statistics

{summary.to_markdown()}

### Per-Model Average Drift

{model_means.to_markdown()}

### Key Findings

1. **Paraphrased variations** generally maintain higher semantic similarity to base responses than adversarial variations across both models.
2. **Adversarial prompts** produce the largest drift, indicating both models are susceptible to misleading framing.
3. Model architecture differences are observable in the distribution of drift scores.

### Figures

- **Figure 1**: Drift heatmap showing per-prompt similarity scores (`reports/figures/drift_heatmap.png`)
- **Figure 2**: Model comparison bar chart (`reports/figures/model_comparison.png`)
- **Figure 3**: Sensitivity distribution histogram (`reports/figures/sensitivity_distribution.png`)

## Discussion

Our experiment demonstrates that prompt perturbation produces measurable behavioral drift in both frontier and open-source models. The adversarial condition consistently produces larger drift than paraphrasing, suggesting that both models are more sensitive to contradictory framing than to simple rephrasing.

The practical implication is that LLM-based applications must account for prompt sensitivity in their design. Small changes in how a question is framed can produce semantically different outputs, even at temperature 0.0.

### Limitations

- Sample size (10 base prompts) limits statistical power
- Single temperature setting (0.0) does not capture stochastic behavior
- Semantic similarity as a drift metric may miss factual accuracy differences
- The 1.5B parameter model may have inherent capability limitations that conflate with drift

### Future Work

- Scale to 100+ base prompts for stronger statistical claims
- Test multiple temperature settings (0.0, 0.3, 0.7, 1.0)
- Add factual accuracy scoring alongside semantic similarity
- Include more model architectures (Llama, Mistral, Claude)
- Implement bootstrap confidence intervals for uncertainty quantification

---

*Generated: {datetime.date.today().isoformat()}*
*Repository: prompt-drift-lab*
"""

# Write the report to file
with open("reports/paper.md", "w") as f:
    f.write(report)

print("Research report generated: reports/paper.md")
print(f"Report length: {len(report)} characters")

What does this code do?

This cell pulls real numbers from your drift_df DataFrame (which is still in memory from the previous step) and embeds them directly into a structured Markdown template.

The summary.to_markdown() call converts your grouped statistics into a Markdown table. This means your report always reflects your actual experimental data, not placeholder values.

After running the cell, you should see a confirmation message with the report path and character count.
Preview the first few lines of your report by running this cell:

# Preview the generated report
with open("reports/paper.md", "r") as f:
    # Print the first 40 lines as a preview
    for i, line in enumerate(f):
        if i >= 40:
            print("\n... (report continues)")
            break
        print(line, end="")

You should see the Abstract and the beginning of the Methodology section printed with your actual model names and experimental parameters.

Seeing a NameError for drift_df?

If your Colab runtime restarted between steps, drift_df will not be in memory. Re-run the cells from Step 4 that load results and compute drift scores. Your data is still saved in data/results/responses.json so you just need to reload and recompute.

Help me reload drift_df from my saved responses.json file

Write a polished README and push to GitHub

A good README is the front door of any research repository. It tells visitors what the project does, how to reproduce results, and where to find the key outputs.

Generate the polished README by running this code:

# Generate polished README.md with project overview and reproduction instructions
readme = """# Prompt Drift Lab

**Measuring LLM Behavioral Consistency Under Prompt Perturbation**

## Overview

This research project investigates how large language models change their responses when prompts are systematically rephrased or adversarially manipulated. We measure "prompt drift"—the semantic distance between a model's response to an original prompt versus its response to a variation of that prompt.

## Methodology

1. Design 10 base prompts across 2 categories (reasoning + instruction-following)
2. Create 3 variations per prompt: original, paraphrased, adversarial
3. Run all 30 prompts through 2 models (Gemini 2.5 Flash + Qwen2.5-1.5B-Instruct)
4. Compute semantic similarity drift scores using sentence-transformers
5. Visualize and analyze results

## Key Findings

- Adversarial prompt variations produce significantly more drift than paraphrased variations
- Both frontier and open-source models show measurable sensitivity to prompt framing
- See `reports/paper.md` for the full research report with figures and statistics

## Repository Structure

```
prompt-drift-lab/
├── configs/
│   └── experiment.yaml          # Experiment configuration and prompts
├── data/
│   └── results/
│       └── responses.json       # Raw model responses (60 total)
├── reports/
│   ├── figures/
│   │   ├── drift_heatmap.png
│   │   ├── model_comparison.png
│   │   └── sensitivity_distribution.png
│   └── paper.md                 # Structured research report
├── src/
│   └── prompt_drift/
│       ├── __init__.py
│       ├── prompts.py           # Prompt loading and matrix building
│       ├── inference.py         # Model inference (API + local)
│       ├── metrics.py           # Drift score computation
│       └── visualization.py     # Publication-quality figures
├── notebooks/                   # Colab experiment notebooks
├── tests/                       # Test suite
├── pyproject.toml               # Package configuration
├── LICENSE                      # MIT License
├── .gitignore
└── README.md
```

## Reproduction

1. Clone this repository
2. Open the notebook in Google Colab (GPU runtime recommended)
3. Install dependencies: `pip install -e .`
4. Get a free Gemini API key from [Google AI Studio](https://aistudio.google.com/app/apikey)
5. Run the experiment cells in order

## Models Tested

| Model | Type | Parameters |
|-------|------|------------|
| Gemini 2.5 Flash | Frontier (API) | Undisclosed |
| Qwen2.5-1.5B-Instruct | Open-source (local) | 1.54B |

## License

MIT
"""

# Write the README
with open("README.md", "w") as f:
    f.write(readme)

print("README.md generated successfully")

What does the README include?

The README covers five key areas: a project description, the methodology summary, key findings, a visual repository structure diagram, and step-by-step reproduction instructions.

The reproduction section is essential for research credibility. Anyone who clones your repo should be able to replicate your experiment end-to-end.

Now push all new files to GitHub by running this cell:

# Commit and push the research report and README to GitHub
import subprocess

commands = [
    ["git", "add", "-A"],
    ["git", "commit", "-m", "Add research report and polished README"],
    ["git", "push", "origin", "main"],
]

for cmd in commands:
    result = subprocess.run(cmd, capture_output=True, text=True)
    print(f"$ {' '.join(cmd)}")
    if result.stdout:
        print(result.stdout)
    if result.returncode != 0:
        print(f"Error: {result.stderr}")

You should see output confirming the commit and push succeeded. The last line should show something like main -> main.

Push rejected or authentication failed?

If your push fails with an authentication error, your GitHub token may have expired. Re-run the clone cell from Step 1 that sets up your credentials, then retry the push.

Help me fix my GitHub push authentication in Colab

Validate the complete repository structure

A professional research repository has a predictable structure. Reviewers and collaborators expect specific directories and files in specific places. This validation confirms your repository meets those standards.

Run this validation cell to check every expected file exists:

# Validate complete repository structure against professional standards
import os

expected_files = [
    "pyproject.toml",
    "README.md",
    "LICENSE",
    ".gitignore",
    "configs/experiment.yaml",
    "data/results/responses.json",
    "reports/paper.md",
    "reports/figures/drift_heatmap.png",
    "reports/figures/model_comparison.png",
    "reports/figures/sensitivity_distribution.png",
    "src/prompt_drift/__init__.py",
    "src/prompt_drift/prompts.py",
    "src/prompt_drift/inference.py",
    "src/prompt_drift/metrics.py",
    "src/prompt_drift/visualization.py",
]

print("Repository Structure Validation")
print("=" * 40)

all_present = True
for filepath in expected_files:
    exists = os.path.exists(filepath)
    status = "✓" if exists else "✗ MISSING"
    print(f"  {status}  {filepath}")
    if not exists:
        all_present = False

print("=" * 40)
if all_present:
    print("All files present. Repository meets professional research standards.")
else:
    print("WARNING: Some files are missing. Check the output above.")

What does this validation check?

This cell checks for all 15 expected files across your repository: the package configuration, documentation, experiment config, raw data, generated figures, the research report, and all five source modules.

Every checkmark confirms that file was created and saved correctly throughout the project. A missing file means a previous step did not complete successfully.

You should see 15 checkmarks and the message "All files present. Repository meets professional research standards." at the bottom.
Open your GitHub repository in a browser tab. You should see README.md rendered on the main page with your project title, methodology, and repository structure diagram.

Seeing a missing file in the validation?

If a figure file is missing, re-run the visualization cells from Step 4. If a source module is missing, re-run the corresponding cell from Step 2 that writes the file. After fixing, re-run the validation cell to confirm.

Help me figure out which step created the missing file so I can regenerate it

✔️ Awesome, I've got everything!

Your repository is complete and professionally structured. All 15 files are in place and pushed to GitHub.

ⓧ I'd like to double check the full code

Here is the complete report-generation cell for reference:

# Generate the structured research report from experimental data
import datetime

# Compute summary statistics from drift_df
non_base = drift_df[drift_df["variation"] != "base"]
summary = non_base.groupby(["model", "variation"])["drift_score"].agg(["mean", "std"]).round(3)

# Get per-model averages
model_means = non_base.groupby("model")["drift_score"].mean().round(3)

# Build the report content
report = f"""# Prompt Drift Lab: Measuring LLM Behavioral Consistency Under Prompt Perturbation

## Abstract

This study measures behavioral consistency of large language models (LLMs) under systematic prompt perturbation. We evaluate two models—Gemini 2.5 Flash (frontier, API-based) and Qwen2.5-1.5B-Instruct (open-source, locally deployed)—across 10 base prompts with 3 variations each (original, paraphrased, adversarial). Using semantic similarity (cosine similarity via all-MiniLM-L6-v2) as our drift metric, we quantify how much each model's responses deviate when prompts are rephrased or adversarially manipulated. Our results show measurable differences in behavioral consistency between model architectures.

## Methodology

### Prompt Design

We constructed a prompt matrix of 10 base prompts spanning two categories:
- **Reasoning** (5 prompts): Questions requiring logical explanation
- **Instruction-following** (5 prompts): Tasks requiring specific output formats

Each base prompt has three variations:
- **Base**: The original prompt
- **Paraphrased**: Semantically equivalent rephrasing
- **Adversarial**: Deliberately misleading or contradictory framing

### Models Under Test

| Model | Type | Parameters | Access |
|-------|------|-----------|--------|
| Gemini 2.5 Flash | Frontier | Undisclosed | API |
| Qwen2.5-1.5B-Instruct | Open-source | 1.54B | Local (T4 GPU) |

### Scoring Methodology

Drift is measured as the cosine similarity between sentence embeddings of the base response and each variation's response. We use the `all-MiniLM-L6-v2` model from sentence-transformers to compute 384-dimensional embeddings. A score of 1.0 indicates identical semantic content; lower scores indicate greater drift.

## Experiments

### Setup
- **Environment**: Google Colab with T4 GPU
- **Temperature**: 0.0 for both models (deterministic generation)
- **Max tokens**: 256 per response
- **Total inferences**: 60 (30 per model: 10 prompts x 3 variations)

### Procedure
1. Load experiment configuration from `configs/experiment.yaml`
2. Build prompt matrix (10 base prompts x 3 variations = 30 unique prompts)
3. Run all 30 prompts through Gemini 2.5 Flash via API
4. Run all 30 prompts through Qwen2.5-1.5B-Instruct locally
5. Compute pairwise semantic similarity between base and variation responses

## Results

### Summary Statistics

{summary.to_markdown()}

### Per-Model Average Drift

{model_means.to_markdown()}

### Key Findings

1. **Paraphrased variations** generally maintain higher semantic similarity to base responses than adversarial variations across both models.
2. **Adversarial prompts** produce the largest drift, indicating both models are susceptible to misleading framing.
3. Model architecture differences are observable in the distribution of drift scores.

### Figures

- **Figure 1**: Drift heatmap showing per-prompt similarity scores (`reports/figures/drift_heatmap.png`)
- **Figure 2**: Model comparison bar chart (`reports/figures/model_comparison.png`)
- **Figure 3**: Sensitivity distribution histogram (`reports/figures/sensitivity_distribution.png`)

## Discussion

Our experiment demonstrates that prompt perturbation produces measurable behavioral drift in both frontier and open-source models. The adversarial condition consistently produces larger drift than paraphrasing, suggesting that both models are more sensitive to contradictory framing than to simple rephrasing.

The practical implication is that LLM-based applications must account for prompt sensitivity in their design. Small changes in how a question is framed can produce semantically different outputs, even at temperature 0.0.

### Limitations

- Sample size (10 base prompts) limits statistical power
- Single temperature setting (0.0) does not capture stochastic behavior
- Semantic similarity as a drift metric may miss factual accuracy differences
- The 1.5B parameter model may have inherent capability limitations that conflate with drift

### Future Work

- Scale to 100+ base prompts for stronger statistical claims
- Test multiple temperature settings (0.0, 0.3, 0.7, 1.0)
- Add factual accuracy scoring alongside semantic similarity
- Include more model architectures (Llama, Mistral, Claude)
- Implement bootstrap confidence intervals for uncertainty quantification

---

*Generated: {datetime.date.today().isoformat()}*
*Repository: prompt-drift-lab*
"""

# Write the report to file
with open("reports/paper.md", "w") as f:
    f.write(report)

print("Research report generated: reports/paper.md")
print(f"Report length: {len(report)} characters")

And the README generation cell:

# Generate polished README.md with project overview and reproduction instructions
readme = """# Prompt Drift Lab

**Measuring LLM Behavioral Consistency Under Prompt Perturbation**

## Overview

This research project investigates how large language models change their responses when prompts are systematically rephrased or adversarially manipulated. We measure "prompt drift"—the semantic distance between a model's response to an original prompt versus its response to a variation of that prompt.

## Methodology

1. Design 10 base prompts across 2 categories (reasoning + instruction-following)
2. Create 3 variations per prompt: original, paraphrased, adversarial
3. Run all 30 prompts through 2 models (Gemini 2.5 Flash + Qwen2.5-1.5B-Instruct)
4. Compute semantic similarity drift scores using sentence-transformers
5. Visualize and analyze results

## Key Findings

- Adversarial prompt variations produce significantly more drift than paraphrased variations
- Both frontier and open-source models show measurable sensitivity to prompt framing
- See `reports/paper.md` for the full research report with figures and statistics

## Repository Structure

```
prompt-drift-lab/
├── configs/
│   └── experiment.yaml          # Experiment configuration and prompts
├── data/
│   └── results/
│       └── responses.json       # Raw model responses (60 total)
├── reports/
│   ├── figures/
│   │   ├── drift_heatmap.png
│   │   ├── model_comparison.png
│   │   └── sensitivity_distribution.png
│   └── paper.md                 # Structured research report
├── src/
│   └── prompt_drift/
│       ├── __init__.py
│       ├── prompts.py           # Prompt loading and matrix building
│       ├── inference.py         # Model inference (API + local)
│       ├── metrics.py           # Drift score computation
│       └── visualization.py     # Publication-quality figures
├── notebooks/                   # Colab experiment notebooks
├── tests/                       # Test suite
├── pyproject.toml               # Package configuration
├── LICENSE                      # MIT License
├── .gitignore
└── README.md
```

## Reproduction

1. Clone this repository
2. Open the notebook in Google Colab (GPU runtime recommended)
3. Install dependencies: `pip install -e .`
4. Get a free Gemini API key from [Google AI Studio](https://aistudio.google.com/app/apikey)
5. Run the experiment cells in order

## Models Tested

| Model | Type | Parameters |
|-------|------|------------|
| Gemini 2.5 Flash | Frontier (API) | Undisclosed |
| Qwen2.5-1.5B-Instruct | Open-source (local) | 1.54B |

## License

MIT
"""

# Write the README
with open("README.md", "w") as f:
    f.write(readme)

print("README.md generated successfully")

Secret mission

Statistical Rigor

Your results show that one model drifts more than the other. But how confident are you in that claim? In real research, reviewers will ask for statistical evidence. Add bootstrap confidence intervals and a significance test to back up your findings with numbers that hold up to scrutiny.

Clean Up Your Resources

This project runs entirely locally in Google Colab and uses only free-tier services. There are no ongoing costs.

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

GitHub repository (free, no ongoing cost).
Gemini API key (free tier, no billing).
Google Colab session (free, auto-terminates when closed).

Keep everything running

No action needed. Choose this if you want to keep experimenting with your prompt drift research.

Your GitHub repository persists indefinitely at no cost.
Your Gemini API key remains active on the free tier with no charges.
Google Colab sessions auto-terminate after inactivity, but your notebook is saved in Google Drive.

Pause - I'll come back to this later

Free up browser resources while keeping everything restorable.

Close your Google Colab tab. Your code is saved in GitHub and can be re-cloned anytime.
Your GitHub repository and Gemini API key stay active at no cost.
To resume later, open a new Colab notebook, clone your repository, and reinstall dependencies.

Delete - I don't want to use this again

Remove all project resources and start fresh.

Delete the GitHub repository by opening your repository page on GitHub, clicking Settings, scrolling down to the Danger Zone section, and clicking Delete this repository. Confirm by typing the repository name and clicking Delete this repository again.
Delete your Gemini API key by opening the Google Cloud API Credentials page. Find your API key, click the Actions icon, and select Delete API key. Confirm by clicking Delete in the modal.
Close your Google Colab tab. The session terminates automatically and no data persists on the Colab runtime.

That's a Wrap!

Nice work! You just designed, executed, and reported on a controlled LLM evaluation experiment that measures prompt drift across model architectures.

You've learned how to:

Build a professional Python research repository with src/ package layout, pyproject.toml, and structured configs for reproducible experiments.
Design and execute a controlled prompt perturbation experiment across a frontier model (Gemini) and an open-source model (Qwen2.5-1.5B-Instruct), measuring behavioral consistency with semantic similarity as a drift metric.
Generate publication-quality visualizations (drift heatmap, model comparison bar chart, sensitivity distribution histogram) and a structured Markdown research report with Abstract, Methodology, Results, and Discussion sections.
Secret Mission: Added statistical rigor with bootstrap confidence intervals and Mann-Whitney U significance testing to validate whether observed drift differences between models are statistically meaningful.

Ready to quiz yourself?