Test Claude's Code Ownership Prompt

Compare Claude refactors with and without a code ownership prompt.

Introduction

30 Second Summary

A single sentence in a request can change how an AI assistant handles the same job. Without a side-by-side test, even a confident recommendation is still just a claim.

In this project, you will test whether adding This codebase is yours changes how Claude Fable 5.2 refactors repeated Python code. You'll use the Anthropic API to run both prompt versions several times. You'll save every result before turning your observations into a table and bar chart.

What You'll Build

Your finished experiment shows the effect of one ownership sentence through saved code, passing tests, side-by-side diffs, merge counts, and a bar chart.

By the end of this project, you'll have:

  • A deliberately repetitive Python test case with three near-identical functions. Its passing tests give every refactor the same behavioural target.
  • A side-by-side comparison showing how Claude handles the same coding task with and without the ownership sentence.
  • A repeatable 20-run experiment with every output saved and tested. A results table and bar chart show how often each prompt version merged the repeated functions.
  • Secret Mission: Add a third experiment arm that places the ownership sentence at the end of the task. Compare its merge count with the two original arms.

Are there any prerequisites?

You'll need Python and an Anthropic Console account with a card on file. You'll also need an API key for the experiment.

The model calls use a paid API. The project keeps the experiment small by running ten trials for each prompt version.

Before We Start

A fair prompt experiment needs the same tools for every run. A stable local setup keeps environment differences out of your results.

In this step, you will prepare Python 3, pip, pytest, and Git. You will also create an Anthropic API key without sending an API request.

In this step, get ready to:
  • Confirm that Python 3, pip, and Git are available.
  • Initialize the local experiment repository and install pytest.
  • Create an Anthropic API key for later requests.
Check your local tools

The experiment uses Python to run scripts. Pip installs the testing dependency. Git records the files and diffs you compare.

  • Press Cmd+Space on macOS or the Windows key on Windows to open system search.
  • Type Terminal on macOS or PowerShell on Windows.
  • Press Enter to open the terminal.

Choose your operating system to check the three tools.

macOS

  • Check Python, pip, and Git by running these commands:
python3 --version
python3 -m pip --version
git --version

What does this check prove?

  • The first line confirms that your terminal can run Python 3.
  • The second line confirms that pip belongs to the same Python installation.
  • The third line confirms that Git is available from your terminal.

Windows

  • Check Python, pip, and Git by running these commands:
python --version
python -m pip --version
git --version

What does this check prove?

  • The first line confirms that your terminal can run Python 3.
  • The second line confirms that pip belongs to the same Python installation.
  • The third line confirms that Git is available from your terminal.

✔️ I see three version numbers

Your terminal can reach every system tool needed for the experiment. Continue to the repository setup below.

ⓧ Python, pip, or Git is missing

A missing command means the tool is absent or unavailable through your terminal. Install the missing tool before you create the repository.

macOS

  • Visit the official Python downloads page if the Python or pip check failed.
  • Follow the macOS installation guide to install Python with pip.
  • Visit the official Git installation page if the Git check failed.
  • Follow the macOS installation guide to install Git.
  • Close the current terminal after the installation finishes.
  • Open a fresh terminal through system search.
  • Rerun the three version checks from the macOS tab above.

Windows

  • Visit the official Python downloads page if the Python or pip check failed.
  • Follow the Windows installation guide to install Python with pip.
  • Visit the official Git installation page if the Git check failed.
  • Follow the Windows installation guide to install Git.
  • Close the current PowerShell window after the installation finishes.
  • Open a fresh PowerShell window through system search.
  • Rerun the three version checks from the Windows tab above.

Still missing a command?

A newly installed tool may remain unavailable in a terminal that was already open. Close every terminal window before trying the version checks again.

Still stuck? Help me diagnose which local tool is missing.

Initialize the experiment repository

A local repository gives every generated file one consistent home. Git also makes the later refactoring diffs easy to save and compare.

Choose your operating system to create the repository on your Desktop. The commands leave your terminal inside the new repository.

macOS

  • Create the claude-refactoring-prompt-experiment repository on your Desktop by running these commands:
cd ~/Desktop
mkdir claude-refactoring-prompt-experiment
cd claude-refactoring-prompt-experiment
git init

What do these commands do?

  • The first command moves your terminal to the Desktop.
  • The next two commands create the experiment folder and make it your current location.
  • The final command creates the hidden .git directory that stores the repository history.
  • Create an isolated test environment and install pytest by running these commands:
python3 -m venv .venv
source .venv/bin/activate
pip install -U pytest
pytest --version

Why use an isolated environment?

A virtual environment keeps the experiment dependency inside .venv. This protects the packages used by other Python projects.

The final command prints the installed pytest version. That output confirms the test runner is available.

Windows

  • Create the claude-refactoring-prompt-experiment repository on your Desktop by running these commands:
Set-Location -Path "$HOME\Desktop"
New-Item -ItemType "Directory" -Path "claude-refactoring-prompt-experiment"
Set-Location -Path "claude-refactoring-prompt-experiment"
git init

What do these commands do?

  • The first command moves your PowerShell session to the Desktop.
  • The next two commands create the experiment folder and make it your current location.
  • The final command creates the hidden .git directory that stores the repository history.
  • Create an isolated test environment and install pytest by running these commands:
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -U pytest
pytest --version

Why use an isolated environment?

A virtual environment keeps the experiment dependency inside .venv. This protects the packages used by other Python projects.

The final command prints the installed pytest version. That output confirms the test runner is available.

That is the local foundation in place. Your repository now has its own test environment and version history.

Repository or pytest setup failed?

  • Check that your terminal is inside claude-refactoring-prompt-experiment before creating .venv.
  • Close the terminal if your operating system blocks environment activation.
  • Open a fresh terminal through system search before retrying the platform instructions.

Still stuck? Help me fix my local repository setup.

Create your Anthropic API key

The later experiment sends repeated prompts through the API. A personal key gives your local script permission to make those requests.

Adding a payment method can feel risky. No API request leaves your computer in this step.

  • Open the official Claude Console Billing page in your browser.
  • Complete the Claude Console sign-in flow.
  • Use the Billing page to add a payment method if one is not already listed.
  • Open the official API keys page.
  • Click Create key.
  • Enter refactor-prompt-experiment as the key name.
  • Select 30 days as the expiration.
  • Set Linked account to yourself.

Why use an expiring personal key?

A personal key fits a local experiment that you run yourself. The expiration limits how long the credential remains usable.

The Console reveals the full key only once. Its value starts with sk-ant-.

  • Submit the key creation form.
  • Copy the full key while the Console displays it.
  • Store the key in a password manager or secrets manager.
  • Close the key view after confirming that the credential is stored.

Keep the key out of your repository

Your API key grants access to paid API usage. Keep it out of screenshots and project files.

You have created the credential for later use. You have not sent it in an API request.

Before you run this final check, which four tool versions do you expect the terminal to print?

macOS

  • Verify the complete local setup from the repository terminal by running these commands:
python3 --version
python3 -m pip --version
pytest --version
git --version
git status

What proves the setup is ready?

  • The first four lines report versions for Python, pip, pytest, and Git.
  • The final output reports the state of your initialized Git repository.
  • Your terminal remains inside claude-refactoring-prompt-experiment.

Windows

  • Verify the complete local setup from the repository terminal by running these commands:
python --version
python -m pip --version
pytest --version
git --version
git status

What proves the setup is ready?

  • The first four lines report versions for Python, pip, pytest, and Git.
  • The final output reports the state of your initialized Git repository.
  • Your PowerShell session remains inside claude-refactoring-prompt-experiment.

✔️ Awesome, I've got everything!

You are ready to run the experiment. The local repository, test runner, and API credential are all prepared.

ⓧ I'd like to double check the full setup

  • Tooling: Python 3, pip, pytest, and Git are installed and available from the terminal.
  • Repository: Initialized local Git repository for the Claude refactoring-prompt experiment.
  • Anthropic API key: Available for later use and not yet sent in an API request.

Your experiment workspace is ready without consuming any API usage. Next, you will write the repetitive Python file that gives both prompts the same refactoring target.

Create the Repetitive Python Example

Any refactoring comparison needs a stable piece of code to start from. Your baseline needs visible repetition plus behavior worth preserving.

This step creates that baseline in a small Python module. You'll use pytest tests to protect its behavior during later refactors.

In this step, get ready to:
  • Create a Python module with three deliberately repetitive formatters.
  • Write pytest tests for all three formatters.
  • Run the test suite to lock in a passing baseline.
Create the repetitive module

The starting module needs three separate paths that perform the same cleanup. That repetition gives the later refactoring prompt something concrete to reorganize.

  • Create repetitive_code.py in the repository root using your code editor's new-file control.
  • Add the three role formatters to repetitive_code.py by pasting this code:
def format_admin_name(name):
    # Keep the full cleanup inside each role function for the experiment.
    return f"Admin: {name.strip().title()}"

def format_customer_name(name):
    # Keep the full cleanup inside each role function for the experiment.
    return f"Customer: {name.strip().title()}"

def format_guest_name(name):
    # Keep the full cleanup inside each role function for the experiment.
    return f"Guest: {name.strip().title()}"

What does this code do?

  • Each function removes outer whitespace with strip().
  • Each function converts the remaining words with title().
  • Each function adds its own role label to the returned string.
  • The repeated cleanup pipeline is deliberate. It creates the pattern your later prompts may choose to merge.
  • Save repetitive_code.py.

Your editor's file list should now show repetitive_code.py inside the repository.

✔️ Awesome, I've got everything!

Great. Confirm that repetitive_code.py is saved in the repository.

ⓧ I'd like to double check the full code

def format_admin_name(name):
    # Keep the full cleanup inside each role function for the experiment.
    return f"Admin: {name.strip().title()}"

def format_customer_name(name):
    # Keep the full cleanup inside each role function for the experiment.
    return f"Customer: {name.strip().title()}"

def format_guest_name(name):
    # Keep the full cleanup inside each role function for the experiment.
    return f"Guest: {name.strip().title()}"

Cannot find the module?

  • Check that the file name is exactly repetitive_code.py.
  • Confirm that the file sits beside the repository's Git metadata.

Still stuck? Help me check where I saved my Python module.

Protect the original behavior with tests

Tests turn the current behavior into a contract. A later refactor only counts if these same checks still pass.

  • Create test_repetitive_code.py beside repetitive_code.py using your code editor's new-file control.
  • Add one behavior check for each formatter by pasting this code:
# Import every formatter so each repeated path gets its own check.
from repetitive_code import (
    format_admin_name,
    format_customer_name,
    format_guest_name,
)

def test_format_admin_name():
    # Protect the exact cleaned name and role label.
    assert format_admin_name("  ada lovelace  ") == "Admin: Ada Lovelace"

def test_format_customer_name():
    # Protect the exact cleaned name and role label.
    assert format_customer_name("  grace hopper  ") == "Customer: Grace Hopper"

def test_format_guest_name():
    # Protect the exact cleaned name and role label.
    assert format_guest_name("  alan turing  ") == "Guest: Alan Turing"

What do these tests protect?

  • The import group gives the test module access to every formatter.
  • Each test supplies a name with extra whitespace.
  • Each assertion requires the precise cleaned name.
  • Each assertion also protects its role label.
  • Save test_repetitive_code.py.

Your editor's file list should now show both Python files inside the repository.

✔️ Awesome, I've got everything!

Good. Confirm that both Python files are saved before running the test suite.

ⓧ I'd like to double check the full code

# Import every formatter so each repeated path gets its own check.
from repetitive_code import (
    format_admin_name,
    format_customer_name,
    format_guest_name,
)

def test_format_admin_name():
    # Protect the exact cleaned name and role label.
    assert format_admin_name("  ada lovelace  ") == "Admin: Ada Lovelace"

def test_format_customer_name():
    # Protect the exact cleaned name and role label.
    assert format_customer_name("  grace hopper  ") == "Customer: Grace Hopper"

def test_format_guest_name():
    # Protect the exact cleaned name and role label.
    assert format_guest_name("  alan turing  ") == "Guest: Alan Turing"

Before you run the suite, predict how many tests the original module passes.

  • Switch back to the terminal from earlier.
  • Run the complete test suite by entering:
pytest

What should you see?

The runner discovers three tests. The final summary reports that all three pass.

This result proves that every role formatter produces its expected output before any refactoring begins.

That's your baseline locked in. All three repetitive formatters now have passing behavior checks.

Tests not passing?

  • Check that repetitive_code.py and test_repetitive_code.py are saved in the same repository.
  • Compare each imported function name with the matching definition in repetitive_code.py.
  • Check every expected role label for matching capitalization.

Still stuck? Help me diagnose why my three formatter tests are failing.

Your repetitive example is ready. Next, you'll ask Claude to refactor this exact module and save the first result.

Run the Baseline Prompt

Your original module now has passing pytest tests. A fair experiment also needs a control that shows how the model refactors the file without an ownership cue.

In this step, you'll use the Anthropic API to send the baseline task to Claude Fable 5.2. Your script saves the returned module and creates a unified diff against the original.

In this step, get ready to:
  • Install the Anthropic Python SDK and configure your API key.
  • Send the fixed refactoring task without the ownership sentence.
  • Save the candidate module and verify its unified diff.
Prepare the API connection

The experiment script uses Anthropic's Python SDK to authenticate with the key from the previous step. The SDK reads that key from the ANTHROPIC_API_KEY environment variable.

  • Install the Anthropic Python SDK by running:
pip install anthropic

What does this command do?

This installs the anthropic package that provides the Python client. Your script uses that client to send the refactoring request.

You should see a successful installation message or a message confirming that the package is already installed.

Having trouble installing the SDK?

Confirm that the terminal is still at the repository root. Check that pip uses the same Python installation that ran your tests.

Still stuck? Help me diagnose why the Anthropic Python SDK will not install with pip.

Your API key is a live credential. The next command keeps it in the current terminal session without writing it into the repository.

  • Select the tab that matches your terminal.
  • Replace your-api-key-here with the API key you prepared earlier.

macOS

  • Configure the API key in your current terminal session by running:
export ANTHROPIC_API_KEY="your-api-key-here"

What does this command do?

This sets ANTHROPIC_API_KEY for commands launched from the current terminal. The value stays outside your Python files.

Windows

  • Configure the API key in your current PowerShell session by running:
$env:ANTHROPIC_API_KEY="your-api-key-here"

What does this command do?

This sets ANTHROPIC_API_KEY for commands launched from the current PowerShell session. The value stays outside your Python files.

API key missing later?

Environment variables set this way last for the current terminal session. Return to this terminal when you run the experiment.

Still stuck? Help me check whether ANTHROPIC_API_KEY is available without printing its value.

Send the baseline request

The baseline prompt contains the refactoring task and the original module. Its ownership prefix is empty, which makes this run the control for your later comparison.

  • Create run_experiment.py beside repetitive_code.py in your repository using your code editor.
  • Paste this first section into the empty file:
import argparse, difflib, shutil, subprocess
from pathlib import Path
from anthropic import Anthropic
# Select one prompt variant and one numbered output.
parser = argparse.ArgumentParser()
parser.add_argument("variant", choices=("baseline", "ownership"))
parser.add_argument("--run", type=int, default=1)
args = parser.parse_args()
source_path = Path("repetitive_code.py")
original = source_path.read_text()
result_dir = Path("results") / args.variant
result_dir.mkdir(parents=True, exist_ok=True)
output_path = result_dir / f"run_{args.run:02d}.py"
# Reuse a saved response so later checks do not spend another API request.
if output_path.exists():
    candidate = output_path.read_text()
else:
    ownership = "This codebase is yours\n\n" if args.variant == "ownership" else ""
    task = "Refactor repetitive_code.py to merge repeated code where appropriate. Preserve behavior. Return only the complete Python module."
    prompt = f"{ownership}{task}\n\nFile: repetitive_code.py\n```python\n{original}\n```"
    response = Anthropic().messages.create(model="claude-fable-5-2", max_tokens=4096, messages=[{"role": "user", "content": prompt}], output_config={"effort": "medium"})
    candidate = "".join(block.text for block in response.content if block.type == "text").strip()
    candidate = candidate.removeprefix("```python").removeprefix("```").removesuffix("```").strip() + "\n"
    output_path.write_text(candidate)
print(f"Saved {output_path}")

What does this section do?

  • The command-line arguments let the same script run the baseline and ownership variants.
  • The baseline variant leaves ownership empty. The fixed task therefore reaches the model without This codebase is yours.
  • The API request uses model ID claude-fable-5-2 with effort set to medium. The script omits sampling parameters so both experiment arms use the same model settings.
  • The response is cleaned of optional Markdown fences before it is saved as a Python module.
  • Save run_experiment.py.

This next run sends one paid API request. You are limiting it to one small file at medium effort, which keeps the baseline tightly scoped.

Before you run it, do you expect the saved candidate to keep all three repeated functions or merge their shared logic?

  • Send baseline run 1 by running:
python run_experiment.py baseline --run 1

What should you see?

The request may pause while Claude processes the module. When it finishes, you should see Saved results/baseline/run_01.py in the terminal.

That message confirms the control response is now stored separately from your original module.

Baseline request failed?

  • Return to the terminal where you configured ANTHROPIC_API_KEY.
  • Check that run_experiment.py contains the model ID claude-fable-5-2 exactly.
  • Confirm that your Anthropic account can make API requests.

Still stuck? Help me debug my failed baseline Anthropic API request without exposing my API key.

Generate the diff and test the candidate

A refactoring result only matters when you can inspect the structural changes and check the existing behavior. The next section creates the diff and temporarily tests the saved candidate in place of the original module.

  • Find the final print(f"Saved {output_path}") line in run_experiment.py.
  • Add a blank line below it.
  • Paste this final section below the blank line:
# Save a line-by-line comparison with stable file labels.
diff_path = result_dir / f"run_{args.run:02d}.diff"
diff_text = "".join(difflib.unified_diff(
    original.splitlines(keepends=True),
    candidate.splitlines(keepends=True),
    fromfile=str(source_path),
    tofile=str(output_path),
))
diff_path.write_text(diff_text)
# Test the candidate while guaranteeing that the original file is restored.
try:
    shutil.copyfile(output_path, source_path)
    test_result = subprocess.run(["pytest", "-q"])
finally:
    source_path.write_text(original)
print(f"Wrote {diff_path}")
print(diff_text or "No differences from the original module.")
print("Candidate tests passed." if test_result.returncode == 0 else "Candidate tests failed.")

What does this section do?

  • The unified_diff() call compares the original text with the saved candidate. It writes the result to the matching .diff file.
  • The try block temporarily copies the candidate over repetitive_code.py so the existing tests exercise Claude's version.
  • The finally block restores the original module even when a test fails.
  • The final messages expose the saved diff and the candidate's test result in the terminal.
  • Save run_experiment.py.

Before you run the completed script, which file do you think the tests exercise during the temporary swap?

  • Generate the baseline diff and test the saved candidate by running:
python run_experiment.py baseline --run 1

What should you see?

The script reuses results/baseline/run_01.py instead of sending another API request. You should see a pytest summary followed by Wrote results/baseline/run_01.diff.

The terminal then shows the unified diff and reports whether the candidate passed its tests. Your original repetitive_code.py is restored afterward.

Candidate tests failed?

A failed candidate means the response changed behavior during the refactor. Keep the saved module and diff because that failure is part of the baseline observation.

Confirm that the original module still passes by running your existing pytest check from the previous step. Help me compare the failing candidate with the original module.

✔️ Awesome, I've got everything!

Great. Confirm that your terminal shows the candidate's test result and the path to results/baseline/run_01.diff.

  • Keep repetitive_code.py unchanged as the experiment's original module.
  • Keep results/baseline/run_01.py as the first control response.
  • Keep results/baseline/run_01.diff as the saved comparison.

ⓧ I'd like to double check the full code

Compare your run_experiment.py file with this complete version:

import argparse, difflib, shutil, subprocess
from pathlib import Path
from anthropic import Anthropic
# Select one prompt variant and one numbered output.
parser = argparse.ArgumentParser()
parser.add_argument("variant", choices=("baseline", "ownership"))
parser.add_argument("--run", type=int, default=1)
args = parser.parse_args()
source_path = Path("repetitive_code.py")
original = source_path.read_text()
result_dir = Path("results") / args.variant
result_dir.mkdir(parents=True, exist_ok=True)
output_path = result_dir / f"run_{args.run:02d}.py"
# Reuse a saved response so later checks do not spend another API request.
if output_path.exists():
    candidate = output_path.read_text()
else:
    ownership = "This codebase is yours\n\n" if args.variant == "ownership" else ""
    task = "Refactor repetitive_code.py to merge repeated code where appropriate. Preserve behavior. Return only the complete Python module."
    prompt = f"{ownership}{task}\n\nFile: repetitive_code.py\n```python\n{original}\n```"
    response = Anthropic().messages.create(model="claude-fable-5-2", max_tokens=4096, messages=[{"role": "user", "content": prompt}], output_config={"effort": "medium"})
    candidate = "".join(block.text for block in response.content if block.type == "text").strip()
    candidate = candidate.removeprefix("```python").removeprefix("```").removesuffix("```").strip() + "\n"
    output_path.write_text(candidate)
print(f"Saved {output_path}")

# Save a line-by-line comparison with stable file labels.
diff_path = result_dir / f"run_{args.run:02d}.diff"
diff_text = "".join(difflib.unified_diff(
    original.splitlines(keepends=True),
    candidate.splitlines(keepends=True),
    fromfile=str(source_path),
    tofile=str(output_path),
))
diff_path.write_text(diff_text)
# Test the candidate while guaranteeing that the original file is restored.
try:
    shutil.copyfile(output_path, source_path)
    test_result = subprocess.run(["pytest", "-q"])
finally:
    source_path.write_text(original)
print(f"Wrote {diff_path}")
print(diff_text or "No differences from the original module.")
print("Candidate tests passed." if test_result.returncode == 0 else "Candidate tests failed.")

Your baseline is captured. You now have the control response and its diff, ready for a side-by-side comparison with the ownership prompt.

  • Scroll the terminal until the API key command is no longer visible before taking your checkpoint screenshot.

The control arm is ready. Next, you'll add the ownership sentence and compare what Claude changes.

Compare the Ownership Prompt

Your baseline request to Claude Fable 5.2 produced one candidate file. Its unified diff captures every change against the original module.

A fair comparison changes one input. This time, that input is the ownership sentence in the prompt.

In this step, get ready to:
  • Run the ownership prompt against the same repetitive module.
  • Confirm the ownership response has its own saved diff.
  • Compare the baseline and ownership diffs side by side.
Run the ownership variant

The ownership variant keeps the fixed refactoring task unchanged. It adds only This codebase is yours to the prompt.

One more request goes to the Anthropic API. That request can use your paid balance.

The API key stays in the environment variable you configured earlier. It does not appear in the recalled command.

  • Press the Up Arrow until your earlier baseline command appears in the terminal.
  • Replace the baseline variant value with ownership.
  • Check that every other command argument still matches the baseline run.
  • Press Enter to send the ownership request.

After the request finishes, you should see results/ownership/run_01.py plus results/ownership/run_01.diff created.

That completes the matched pair. Both variants have now processed the same module with the same refactoring task.

Ownership files missing?

  • Recall the baseline command again from your terminal history.
  • Check that you replaced only baseline with ownership.
  • Confirm that the API key environment variable is still configured in this terminal.

Still stuck? Help me diagnose why my ownership experiment files were not created.

Compare the two diffs

Each diff measures one candidate against the same original module. Removed lines reveal what Claude discarded.

Added lines reveal the replacement structure. Placing both diffs side by side makes the refactoring choices easier to compare.

Before you compare them, do you expect the ownership sentence to change how much repeated code Claude combines?

  • Switch to the local editor you used for the repository files.
  • Open results/baseline/run_01.diff from the repository file tree.
  • Open results/ownership/run_01.diff in a second editor tab.
  • Move the baseline diff into the left editor pane using the editor's split-view control.
  • Move the ownership diff into the right editor pane.
  • Compare how each response handles the three repeated function bodies.
  • Look for shared logic introduced by either response.

The left pane now shows the baseline changes against repetitive_code.py. The right pane shows the ownership changes against the same file.

Any difference in consolidation is now visible. Matching structures are also a valid result from this single pair.

What Counts as a Meaningful Difference?

A merged result replaces repeated function bodies with shared logic. The original behaviors should remain available through the refactored module.

One pair can illustrate this behavior. It cannot establish how often the behavior occurs.

Can't Split the Editor?

  • Open the baseline diff in one editor window.
  • Open the ownership diff in a second editor window.
  • Place the two windows next to each other using your operating system's window controls.

Need a hand? Help me place these two diff files side by side for comparison.

That's your first controlled pair complete. Next, you'll repeat each variant ten times to see whether the pattern survives more runs.

Run Ten Trials Per Prompt

Your first baseline diff is saved. Your first ownership diff is saved too.

Now you'll use the experiment script to build ten runs for each prompt variant. The repeated runs reveal how much the model's response changes from trial to trial.

One pair can be a fluke. Every saved candidate needs a test result before it counts as useful evidence.

In this step, get ready to:
  • Complete ten baseline trials without the ownership sentence.
  • Complete ten ownership trials with the ownership sentence.
  • Record a test result for every saved candidate.
Complete the baseline batch

Claude Fable 5.2 can produce different refactors from the same request. Repeating the baseline prompt shows the variation that appears without the ownership sentence.

Before you run the batch

These requests consume paid Anthropic API credit. You stay in control by stopping the batch whenever you choose.

  • Return to the terminal from earlier.
  • Use the terminal history to recall the baseline command that created results/baseline/run_01.py.

Before you start run 02, do you expect every baseline response to choose the same refactor?

  • Press Enter to start baseline run 02.

The script sends the fixed baseline task. Its completion summary should identify the new candidate after the response returns.

  • Confirm the candidate was saved as results/baseline/run_02.py.
  • Confirm the matching diff was saved as results/baseline/run_02.diff.
  • Confirm the trial log records the pytest result for run 02.
  • Confirm the trial log records whether run 02 merged the repeated functions.
  • Recall the same baseline command from the terminal history.
  • Run the recalled command again.
  • Repeat this cycle until the baseline directory reaches run_10.py.

Each completed cycle should advance the run number by one. The baseline directory should finish with ten candidates plus ten matching diff files.

Why keep the prompt fixed?

The prompt variant is the condition you are testing. Keeping the task text fixed makes the ownership sentence the only planned difference between the two batches.

The model can still vary between runs. That variation is the reason you are collecting ten trials.

Did a baseline trial stop early?

  • Check that the API key environment variable is still available to the experiment script.
  • Keep every candidate that completed successfully.
  • Run only the missing trial again from the terminal history.

Still stuck? Why did one of my repeated baseline trials stop before saving its files?

Good work. Your baseline condition now has ten saved attempts instead of a single anecdote.

Complete the ownership batch

The ownership batch uses the same fixed refactoring task. Its only prompt change is the added sentence "This codebase is yours".

  • Use the terminal history to find the ownership command that created results/ownership/run_01.py.
  • Press Enter to start ownership run 02.

The completion summary should identify a new candidate inside the ownership directory. Its run number should match the new diff and trial-log entry.

  • Confirm the candidate was saved as results/ownership/run_02.py.
  • Confirm the matching diff was saved as results/ownership/run_02.diff.
  • Confirm the trial log records the pytest result for ownership run 02.
  • Confirm the trial log records whether ownership run 02 merged the repeated functions.
  • Recall the same ownership command from the terminal history.
  • Run the recalled command again.
  • Repeat this cycle until the ownership directory reaches run_10.py.

That's the full batch complete. You now have ten baseline responses plus ten ownership responses from the same coding task.

Is a run number missing or duplicated?

  • Compare the candidate filenames with their matching diff filenames.
  • Rerun only a missing number.
  • Keep duplicate output until you confirm which trial-log row points to it.

Need help? How do I reconcile a missing or duplicated run number in my experiment results?

Audit all trial records

A refactor only counts as useful when it preserves the original behavior. The pytest result gives every candidate the same behavioral check.

The trial log also preserves unsuccessful outcomes. Keeping those records prevents the final comparison from quietly excluding inconvenient results.

  • Review the trial log maintained by run_experiment.py.
  • Confirm the log contains twenty trial rows.
  • Confirm ten rows use the baseline variant.
  • Confirm ten rows use the ownership variant.
  • Confirm every row includes its saved output path.
  • Confirm every row includes its diff path.
  • Confirm every row includes a merge classification.

Before you inspect the final column, do you think every candidate preserved the behavior covered by the original tests?

  • Inspect the recorded pytest result for every trial.

For the successful checkpoint, you should see a recorded passing result for all twenty candidates. Each passing row proves that its refactor preserved the tested behavior.

  • Confirm results/baseline/ contains run_01.py through run_10.py.
  • Confirm results/ownership/ contains run_01.py through run_10.py.
  • Confirm every candidate has a corresponding unified diff.

Does a candidate fail its tests?

Keep the failed candidate because it is part of the experiment. Check that its trial-log row records the failure.

  • Compare the failed candidate with its saved diff.
  • Check whether the response removed behavior covered by test_repetitive_code.py.
  • Avoid replacing the failed candidate with a fresh response.

Need a second pair of eyes? Why did this saved refactor fail the original pytest tests?

Your evidence set is ready. Next up, you'll count the merged refactors and turn those observations into a table and bar chart.

Summarize the Refactoring Results

You now have twenty saved candidates. Every run has a diff plus a test result.

Raw files still leave the experiment's main question unanswered. You need a compact comparison of the two prompt variants.

You'll reduce the trial log to two counts. A results table plus a bar chart will make the observed difference visible.

In this step, get ready to:
  • Count the merged-function outcomes for both prompt variants.
  • Create a Markdown results table from the trial log.
  • Draw a bar chart that visualizes the two counts.
Add the results table

A reproducible summary calculates its counts from the saved trial log. This keeps the report tied to the evidence from all twenty runs.

What should the summary calculate?

  • The baseline total comes from the ten records created without the ownership sentence.
  • The ownership total comes from the ten records created with the ownership sentence.
  • Each merged count comes from the trial log value that records whether the repeated functions were merged.
  • The table contains one row for each prompt variant.
  • Each row reports 10 runs plus its merged-function count.
  • In the trial log from earlier, count the baseline records marked as merged.
  • Record that value here: your baseline merged-run count.
  • In the same trial log, count the ownership records marked as merged.
  • Record that value here: your ownership merged-run count.

How should the script build the table?

The new summary logic reads the completed trial log. It groups the records by prompt variant.

It counts the merged outcomes in each group. It writes those counts to results/summary.md.

  • The table headings identify the prompt variant.
  • The table headings identify the number of runs.
  • The table headings identify the merged-function count.
  • The row labels are Baseline and Ownership.
  • The existing twenty trial records remain the inputs for the summary.
  • Use your editor to open run_experiment.py from the repository root.
  • Add the summary logic after the script finishes recording all trial outcomes.
  • Save run_experiment.py.

Before you run the summary, do you expect its two merged counts to match the values you recorded manually?

  • Run run_experiment.py using the same method from earlier.

You'll see results/summary.md in the existing results folder. The file contains one row for each prompt variant.

  • Open results/summary.md in your editor.

Good progress. Your twenty trial records now resolve into a two-row comparison that another person can inspect.

Missing the results table?

  • Confirm that the summary logic runs after the twentieth trial record is available.
  • Check that the output path is exactly results/summary.md.
  • Check whether an earlier exit stops the script before the summary logic runs.

Still stuck? Help me debug why my experiment script does not create its Markdown summary.

Draw the comparison chart

The table preserves the exact counts. A bar chart makes the gap between the prompt variants easier to recognize at a glance.

What should the chart show?

  • The chart contains one bar labelled Baseline.
  • The chart contains one bar labelled Ownership.
  • Each bar height equals the corresponding merged-function count from the table.
  • The vertical scale spans 0 through 10.
  • The script saves the finished chart as results/refactoring_counts.png.
  • Switch back to run_experiment.py from earlier.
  • Extend the summary logic so it renders the two counts as the specified bar chart.
  • Save run_experiment.py.

Before you run the updated summary, which bar do you expect to be taller based on your manual counts?

  • Run run_experiment.py again using the same method from earlier.

You'll see results/refactoring_counts.png beside the table. The image contains two labelled bars.

  • Open results/refactoring_counts.png from the results folder.

That comparison is now visible. The height of each bar shows how often its prompt variant merged the repeated functions in your ten trials.

Chart missing or empty?

  • Confirm that the chart uses the same two counts written to results/summary.md.
  • Check that the image path is exactly results/refactoring_counts.png.
  • Confirm that the chart is saved before the script finishes.

Need another pair of eyes? Help me debug why my two-count bar chart is missing or empty.

✔️ Awesome, I've got everything!

Great. Confirm that you saved run_experiment.py after adding the table plus chart logic.

ⓧ I'd like to double check the full code

Compare the complete run_experiment.py file against these cumulative requirements:

  • The existing trial loop still saves every returned Python candidate.
  • The existing trial loop still saves every unified diff.
  • The trial log still records each merge classification.
  • The trial log still records each pytest result.
  • The summary logic reads all twenty completed trial records.
  • The summary logic writes results/summary.md.
  • The chart logic writes results/refactoring_counts.png.
  • Both outputs use the same baseline count plus ownership count.
Verify the table against the chart

Twenty runs can show what happened in this experiment. They cannot establish a general benchmark for every codebase or coding task.

Your final check proves that the two output files report the same evidence. It also keeps the conclusion tied to this specific set of trials.

Before you compare the files, should each bar reach the same number shown in its table row?

  • Return to results/summary.md from earlier.
  • Confirm that the baseline row reports 10 runs.
  • Confirm that the ownership row reports 10 runs.
  • Compare the baseline table count with your baseline merged-run count.
  • Compare the ownership table count with your ownership merged-run count.
  • Switch back to results/refactoring_counts.png.
  • Match the baseline bar height to the baseline table count.
  • Match the ownership bar height to the ownership table count.

You'll see two table rows plus two chart bars. Each bar reaches the count reported for the same prompt variant.

How should you describe the result?

Report which prompt variant merged the repeated functions more often in these twenty trials. Include the two counts so the observation stays concrete.

Describe the result as evidence from your experiment. The small sample leaves room for a different result in another set of runs.

That is the experiment complete. Your table plus chart now show what the ownership sentence changed in this set of trials.

Secret mission

Audit the Refactors That Count

Turn the raw merge totals into quality-adjusted counts. Review the saved diffs to reject broken or harder-to-maintain consolidations. Decide whether the ownership sentence changed useful refactoring behavior.

Clean Up Your Resources

Clean Up Your Resources

Your local Git repository costs nothing to keep on your computer. An active Anthropic API key can still authorize paid requests.

Resources you used:

  • The local Git repository for the Claude refactoring prompt experiment.
  • The Anthropic API key used for twenty trial requests.

Keep everything running

No action is needed for your local files. Choose this option if you want to rerun the experiment.

  • Keep the repository folder that contains run_experiment.py.
  • Keep results/summary.md as the experiment's compact record.
  • Keep results/refactoring_counts.png as the visual comparison.
  • Retain the experiment key only if you expect to make more Anthropic API requests.
  • Limit the key in the Anthropic console if you want tighter spending control.

Your experiment remains ready for more trials or closer inspection.

Pause - I'll come back to this later

No experiment process needs to keep running while you are away. Your saved files remain available on disk.

  • Leave the repository folder in place.
  • Limit the experiment key through the Anthropic console.
  • Confirm the selected limit is visible in the console.
  • Close the terminal session that contains the key environment variable.

The twenty trial outputs remain ready for your return.

Delete - I don't want to use this again

Deleting a credential can feel final. The twenty saved outputs remain readable without an active key.

Delete the Anthropic API key:

  • Sign in to the Anthropic console.
  • Open the API key management area.
  • Select the key used for this experiment.
  • Delete the selected key.
  • Confirm the key no longer appears as active.
  • Close the terminal session that contains the key environment variable.

Deleting the repository folder is permanent. Its hidden Git history disappears with the saved candidates.

Delete the local repository:

  • Open Finder on macOS or File Explorer on Windows.
  • Search your computer for run_experiment.py.
  • Open the folder containing run_experiment.py.
  • Delete that repository folder.
  • Search your computer for run_experiment.py again.

You should see no result for the deleted repository copy. The results/baseline/, results/ownership/, results/summary.md, and results/refactoring_counts.png artifacts are gone with it.

Nice Work!

Nice Work!

You did it! You ran a controlled refactoring experiment that tested whether one ownership sentence changed Claude's treatment of repeated Python code. Your repository now preserves a complete evidence trail for all twenty trials.

You've learned how to:

  • Ran the same fixed refactoring task across both prompt variants.
  • Protected behavior by running the pytest suite against every saved candidate.
  • Reported the merge counts in a results table. The bar chart makes both prompt variants easy to compare. Your conclusion stays grounded as an observation from ten runs per variant.
  • Completed the optional Secret Mission to apply the same evidence-first method to a further challenge.

Ready to quiz yourself?