Claude Support Ticket Classifier

Build a Claude-powered classifier with an evaluation framework measuring accuracy.

Introduction

30 Second Summary

Every support team deals with the same problem: hundreds of incoming tickets that need to be routed to the right person. Manually reading and categorizing each one is slow and error-prone.

In this project, you will build a Claude-powered ticket classifier that automatically sorts support tickets into categories. Then you will create an evaluation framework that measures exactly how well it performs.

What You'll Build

You run a single command, Claude classifies 20 support tickets into categories like billing and technical, and your eval framework prints a confusion matrix showing exactly where the classifier nails it and where it stumbles.

By the end of this project, you'll have:

  • A classifier that takes raw ticket text and returns structured JSON with a category, confidence score, and reasoning for every classification.
  • A labeled test dataset of 20 support tickets spanning 4 categories, including deliberately ambiguous edge cases designed to stress-test the model.
  • An evaluation harness that computes accuracy, precision, and recall per category, with a printed confusion matrix showing where the classifier succeeds and struggles.
  • Secret Mission: Build an automated prompt optimization loop that tests multiple system prompts and compares their eval scores to find the best-performing configuration.

Are there any prerequisites?

This project assumes you are comfortable writing Python. You do not need prior experience with the Anthropic API. You will set up your API key from scratch in Step 1.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Set Up Your Environment and Claude API Access

Before you can build anything with Claude, you need two things: the Anthropic Python SDK installed locally, and a valid API key that authenticates your requests.

In this step, you'll create your project environment, grab an API key from Anthropic's platform, and confirm everything connects with a quick test call.

In this step, get ready to:
  • Create a project directory with a Python virtual environment and install the anthropic package.
  • Sign up at platform.claude.com, create an API key, and configure it as an environment variable.
  • Run a test API call to confirm Claude responds.
Create your project directory and install the SDK

You need an isolated Python environment so your project dependencies stay separate from your system packages.

  • Open your terminal. Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or PowerShell (Windows) and press Enter to open it.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and set up a virtual environment by running these commands:
mkdir ticket-classifier
cd ticket-classifier
python3 -m venv venv
  • Confirm the virtual environment was created by running:
ls

You should see a venv folder listed in the output.

Don't see the venv folder?

Make sure you have Python 3.9 or higher installed. Run python3 --version to check. If the command isn't found, you need to install Python from python.org first.

On Windows, use python instead of python3 if the latter isn't recognized.

help me set up Python and a virtual environment for my project

  • Activate the virtual environment:

macOS

source venv/bin/activate

Windows

venv\Scripts\Activate.ps1

Your terminal prompt now shows (venv) at the beginning, confirming the virtual environment is active.

  • Install the Anthropic SDK by running:
pip install anthropic
  • Confirm the package installed correctly by running:
python -c "import anthropic; print(anthropic.__version__)"

You should see a version number printed, such as 0.121.0.

Seeing a ModuleNotFoundError?

Make sure you activated the virtual environment before running pip install. Your prompt should start with (venv). If it doesn't, run the activation command again and re-install.

help me fix my anthropic SDK installation

Create your API key on the Anthropic platform

The SDK needs an API key to authenticate your requests to Claude. You'll create one on Anthropic's developer platform.

  • Open platform.claude.com in your browser.
  • Sign up for a new account or log in if you already have one.
  • Navigate to Settings, then select API Keys.
  • Click the button to create a new key.
  • Choose an expiration (Never is fine for a learning project).
  • Copy the key immediately. You will not be able to see it again after closing the dialog.

What about the cost?

New users receive a small amount of free credits to test the API. This entire project will cost approximately $0.30 to $0.50 in API usage using Claude Sonnet 5 at introductory pricing ($2 per million input tokens, $10 per million output tokens).

Now set the key as an environment variable so the SDK can find it automatically.

  • Set the ANTHROPIC_API_KEY environment variable by running this command in your terminal (replace the placeholder with your actual key):

macOS

export ANTHROPIC_API_KEY=your-api-key-here

Windows

$env:ANTHROPIC_API_KEY = "your-api-key-here"

Why an environment variable?

The Anthropic SDK automatically reads your key from the ANTHROPIC_API_KEY environment variable. This keeps your secret out of your source code, which is critical if you ever push to a public repository.

Test your Claude connection

Everything is wired up. Time to confirm Claude actually responds to your requests.

Before you run this, what do you expect to see? A greeting from Claude, or an error?

  • Send a test message to Claude by running this command:

macOS

python -c "
import anthropic
client = anthropic.Anthropic()
msg = client.messages.create(
    model='claude-sonnet-5',
    max_tokens=100,
    messages=[{'role': 'user', 'content': 'Hello, Claude'}]
)
print(msg.content[0].text)
"

Windows

python -c "import anthropic; client = anthropic.Anthropic(); msg = client.messages.create(model='claude-sonnet-5', max_tokens=100, messages=[{'role': 'user', 'content': 'Hello, Claude'}]); print(msg.content[0].text)"

You should see a friendly text response from Claude printed in your terminal. The exact wording varies, but any coherent reply confirms your SDK, API key, and network connection are all working.

Getting an authentication error?

If you see a 401 error, your API key is not being read correctly. Make sure you set the environment variable in the same terminal session where you're running the command. Opening a new terminal window requires you to export the variable again.

Double-check that you copied the full key from platform.claude.com with no trailing spaces.

help me fix my Anthropic API authentication error

Your environment is fully configured. Next up, you'll build the ticket classifier that turns raw support text into structured JSON classifications.

Build the Ticket Classifier with Structured Output

Your Claude connection is live. Now you need Claude to do something useful with it: classify support tickets into categories.

The challenge is getting Claude to return classifications in a consistent, parseable format. Without structured output, you would get free-text responses that break any downstream code trying to process the results. Tool use solves this by forcing Claude to return data that matches an exact schema you define.

In this step, get ready to:
  • Define a classification tool schema with four ticket categories and their JSON structure.
  • Write the classify_ticket function that sends tickets to Claude and receives structured JSON.
  • Test the classifier on sample tickets and confirm valid structured output.
Define the classification tool schema

Tool use lets you define a JSON schema that Claude must fill in. Instead of parsing free text, you get a guaranteed structure back every time.

  • Create a new file called classifier.py in your project directory and add the following code:
import anthropic
import json


client = anthropic.Anthropic()

CLASSIFICATION_TOOL = {
    "name": "classify_ticket",
    "description": "Classify a support ticket into exactly one category with a confidence score and reasoning.",
    "input_schema": {
        "type": "object",
        "properties": {
            "category": {
                "type": "string",
                "enum": ["billing", "technical", "account", "feature_request"],
                "description": "The primary category for this support ticket.",
            },
            "confidence": {
                "type": "number",
                "description": "Confidence score between 0.0 and 1.0.",
            },
            "reasoning": {
                "type": "string",
                "description": "Brief explanation of why this category was chosen.",
            },
        },
        "required": ["category", "confidence", "reasoning"],
    },
}

What does this code do?

The CLASSIFICATION_TOOL dictionary defines the exact JSON shape Claude must return. The input_schema uses standard JSON Schema notation.

  • The enum on category constrains Claude to exactly four valid values. It cannot invent new categories.
  • The confidence field gives you a numeric signal for how certain the classification is.
  • The reasoning field forces Claude to explain its decision, which helps you debug misclassifications later.
  • Save classifier.py.
  • Confirm the file was created by running:
ls classifier.py

You should see classifier.py listed in the output.

File not found?

  • Make sure your terminal is in the project directory you created in Step 1.
  • Confirm you saved the file in Cursor (check for the dot indicator on the file tab).

Why isn't my classifier.py showing up?

Write the system prompts and classify function

The tool schema defines the output shape. Now you need the instructions that tell Claude how to make classification decisions. You will write two prompt versions: a simple one and a detailed one with explicit category boundaries.

  • In classifier.py, add both system prompts below the CLASSIFICATION_TOOL dictionary:
SYSTEM_PROMPT_V1 = """You are a support ticket classifier. Classify each ticket into exactly one category:
- billing: Payment issues, charges, invoices, subscription changes, refunds
- technical: Bugs, errors, performance issues, integration problems, API failures
- account: Login issues, password resets, account settings, profile changes, permissions
- feature_request: Suggestions for new features, improvements, or enhancements"""

SYSTEM_PROMPT_V2 = """You are a support ticket classifier. Classify each ticket into exactly one category.

CATEGORY DEFINITIONS WITH BOUNDARIES:
- billing: Issues involving money, payments, charges, invoices, subscription tier changes, refunds, pricing questions. If someone mentions cost or payment, it is billing even if they also mention a feature.
- technical: Bugs, errors, crashes, slow performance, integration failures, API errors, broken functionality. The key signal is something that USED TO WORK or SHOULD WORK but does not.
- account: Login problems, password resets, account creation, profile updates, permission/role changes, account deletion. The key signal is access to or configuration of the account itself.
- feature_request: Requests for NEW capabilities that do not exist yet, suggestions for improvements, wishlists. The key signal is "I wish..." or "It would be nice if..." or "Can you add..."

DISAMBIGUATION RULES:
- "I can't log in" -> account (access issue, not a bug unless they say it was working before)
- "The login page is showing an error 500" -> technical (server error, not account access)
- "I want to upgrade my plan" -> billing (subscription change involves payment)
- "I want to add team members" -> account (permission/access management)
- "Can you add dark mode?" -> feature_request (new capability)
- "Dark mode is broken" -> technical (existing feature not working)"""

Why two prompts?

V1 is a minimal prompt with just category names and brief descriptions. V2 adds explicit boundary definitions and disambiguation rules for tricky edge cases.

You will use both later when evaluating classifier performance. Having them side by side lets you measure exactly how much prompt detail affects accuracy.

  • Now add the classify_ticket function and a test block below the system prompts:
def classify_ticket(ticket_text: str, system_prompt: str = SYSTEM_PROMPT_V2) -> dict:
    """Classify a single support ticket using Claude with tool use."""
    response = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=1024,
        system=system_prompt,
        tools=[CLASSIFICATION_TOOL],
        tool_choice={"type": "tool", "name": "classify_ticket"},
        messages=[{"role": "user", "content": f"Classify this support ticket:\n\n{ticket_text}"}],
    )

    # Extract the structured classification from the tool use response
    for block in response.content:
        if block.type == "tool_use":
            return {
                "category": block.input["category"],
                "confidence": block.input["confidence"],
                "reasoning": block.input["reasoning"],
                "usage": {
                    "input_tokens": response.usage.input_tokens,
                    "output_tokens": response.usage.output_tokens,
                },
            }

    raise ValueError("No tool use block found in response")


if __name__ == "__main__":
    test_ticket = "I can't log into my account since this morning"
    result = classify_ticket(test_ticket)
    print(json.dumps(result, indent=2))

What does this function do?

  • The tool_choice={"type": "tool", "name": "classify_ticket"} parameter forces Claude to use the classification tool. It cannot respond with free text.
  • The function iterates through response.content looking for a tool_use block, which contains the structured JSON that matches your schema.
  • Token usage is tracked in every response so you can compute costs later in the evaluation.
  • The if __name__ block at the bottom lets you quickly test by running the file directly.
  • Save classifier.py.
  • Before you run this, what do you think the output will look like? Will you get a plain text sentence from Claude, or something else?
  • Run the classifier by executing:
python classifier.py

You should see a JSON object printed with category, confidence, reasoning, and usage fields. The category should be account since the ticket is about a login issue.

Seeing an error?

  • If you see an authentication error, confirm your ANTHROPIC_API_KEY environment variable is still set in this terminal session.
  • If you see a model error, double-check that model="claude-sonnet-5" is spelled exactly as shown.
  • If you see a syntax error, make sure the triple-quoted strings for SYSTEM_PROMPT_V1 and SYSTEM_PROMPT_V2 each have matching """ at start and end.

Why is my classifier.py throwing an error when I run it?

Test the classifier on additional sample tickets

One successful test is a start. You need to confirm the classifier handles different categories reliably before building an evaluation framework on top of it.

  • Test a billing ticket by running:
python -c "from classifier import classify_ticket; import json; print(json.dumps(classify_ticket('I was charged twice for my subscription this month'), indent=2))"

You should see "category": "billing" in the output.

  • Test a feature request ticket by running:
python -c "from classifier import classify_ticket; import json; print(json.dumps(classify_ticket('It would be great if you could add dark mode'), indent=2))"

You should see "category": "feature_request" in the output. All three tickets returned valid structured JSON with the correct categories.

Why does this matter for evaluation?

Every classification comes back in the same predictable shape. This means your evaluation code can compare result["category"] against a ground-truth label without any string parsing or regex. Structured output turns a messy NLP problem into a clean data comparison.

Getting a wrong category?

  • Claude's classifications are probabilistic. If you occasionally see a different category, that is expected behavior. The evaluation framework you build next will measure this systematically.
  • If every test returns the same category regardless of input, check that your messages parameter includes the actual ticket text in the f-string.

Why is my classifier returning unexpected categories?

✔️ Awesome, I've got everything!

Great. Double check you have saved classifier.py with all your changes.

ⓧ I'd like to double check the full code

Here is the complete classifier.py as it should look now. Scroll up and confirm your file contains the following sections in order:

  • Imports and client initialization (the first chunk you added).
  • CLASSIFICATION_TOOL dictionary with the JSON schema.
  • SYSTEM_PROMPT_V1 (basic prompt) and SYSTEM_PROMPT_V2 (detailed boundaries with disambiguation rules).
  • The classify_ticket function.
  • The if __name__ == "__main__" test block at the bottom.

Your classifier is working and returning structured data. Next, you will build a labeled test dataset so you can measure exactly how accurate these classifications are across many tickets at once.

Create the Labeled Test Dataset

Your classifier returns structured JSON for individual tickets. But how do you know it's actually good? You need a dataset of tickets where you already know the correct answer, so you can measure how often the classifier gets it right.

In this step, you'll build a labeled test set with clear-cut examples for each category and deliberately ambiguous tickets designed to trip up the classifier. These tricky edge cases become the designed obstacles that surface weaknesses in the next step.

In this step, get ready to:
  • Create test_data.py with 20 labeled support tickets spanning all 4 categories.
  • Add 4 ambiguous edge-case tickets designed to stress-test the classifier.
  • Verify the dataset loads correctly in Python.
Create the baseline ticket data

A good evaluation dataset needs balanced representation across all categories. Five tickets per category gives you enough signal to compute meaningful precision and recall scores without spending too much on API calls.

  • Create a new file called test_data.py in your project folder and paste the following code:
TEST_TICKETS = [
    # Billing - Easy
    {"text": "I was charged twice for my subscription this month. Please refund the duplicate charge.", "expected": "billing", "difficulty": "easy"},
    {"text": "How do I update my credit card on file? The current one expired.", "expected": "billing", "difficulty": "easy"},
    {"text": "I want to downgrade from the Pro plan to the Basic plan.", "expected": "billing", "difficulty": "easy"},
    {"text": "Can you send me a copy of my invoice from last month?", "expected": "billing", "difficulty": "easy"},
    {"text": "I cancelled my subscription but I'm still being charged.", "expected": "billing", "difficulty": "easy"},
    # Technical - Easy
    {"text": "The app crashes every time I try to export a PDF. I get error code 5012.", "expected": "technical", "difficulty": "easy"},
    {"text": "API endpoint /v2/users returns a 500 error since your last deployment.", "expected": "technical", "difficulty": "easy"},
    {"text": "The dashboard is loading very slowly, taking over 30 seconds.", "expected": "technical", "difficulty": "easy"},
    {"text": "Images are not rendering correctly in the mobile app on iOS 18.", "expected": "technical", "difficulty": "easy"},
    {"text": "The webhook integration with Slack stopped working yesterday.", "expected": "technical", "difficulty": "easy"},
    # Account - Easy
    {"text": "I forgot my password and the reset email never arrives.", "expected": "account", "difficulty": "easy"},
    {"text": "I need to change the email address associated with my account.", "expected": "account", "difficulty": "easy"},
    {"text": "How do I enable two-factor authentication on my account?", "expected": "account", "difficulty": "easy"},
    {"text": "Please delete my account and all associated data.", "expected": "account", "difficulty": "easy"},
    {"text": "I need to transfer ownership of our team account to a new admin.", "expected": "account", "difficulty": "easy"},
    # Feature Request - Easy
    {"text": "It would be great if you could add a dark mode option to the dashboard.", "expected": "feature_request", "difficulty": "easy"},
    {"text": "Can you add support for exporting data in CSV format?", "expected": "feature_request", "difficulty": "easy"},
    {"text": "I wish there was a way to set up recurring reports automatically.", "expected": "feature_request", "difficulty": "easy"},
    {"text": "Please add keyboard shortcuts for common actions in the editor.", "expected": "feature_request", "difficulty": "easy"},
    {"text": "Would it be possible to integrate with Google Calendar?", "expected": "feature_request", "difficulty": "easy"},
]

What does this code do?

Each entry in TEST_TICKETS is a dictionary with three keys:

  • text holds the raw support ticket message, exactly what a customer would write.
  • expected is the ground-truth label you assigned by hand. This is what you compare the classifier's prediction against.
  • difficulty tags whether this ticket is straightforward or ambiguous. This helps you diagnose if the classifier only fails on edge cases or also on easy ones.
  • Save test_data.py.
  • Confirm the file loads by running this command:
python -c "from test_data import TEST_TICKETS; print(len(TEST_TICKETS))"

You should see 20 printed in the terminal.

Seeing a ModuleNotFoundError?

  • Make sure your terminal is in the same directory as test_data.py. Run ls and confirm you see both classifier.py and test_data.py listed.
  • Check that the file is named exactly test_data.py (with an underscore, not a hyphen).

Still stuck? Help me debug why Python can't import test_data

Add ambiguous edge cases

The easy tickets are clear-cut. A classifier that nails those but fails on borderline cases gives you a false sense of confidence. The next 4 tickets are deliberately ambiguous. They could reasonably belong to multiple categories.

  • In test_data.py, find the closing ] at the bottom of the file.
  • Add the following entries just above that closing bracket:
    # Ambiguous / Hard cases
    {"text": "I can't log in and I think it might be a bug because it was working fine yesterday.", "expected": "technical", "difficulty": "hard"},
    {"text": "I want to upgrade my plan but the upgrade button gives me an error.", "expected": "technical", "difficulty": "hard"},
    {"text": "Can you add the ability to share my login with my team members?", "expected": "feature_request", "difficulty": "hard"},
    {"text": "My API key seems to have been revoked but I didn't do it.", "expected": "account", "difficulty": "hard"},

Why are these tickets tricky?

  • "I can't log in... it was working fine yesterday" could be account (login issue) or technical (something broke). The "working yesterday" signal makes it technical.
  • "Upgrade my plan but the button gives an error" mixes billing intent with a technical symptom. The broken functionality wins.
  • "Share my login with team members" looks like account management but it's requesting a capability that doesn't exist yet.
  • "API key revoked but I didn't do it" could be technical (system error) or account (access management). The key belongs to the account.

These are the tickets that will expose weaknesses in your classifier's prompt in the next step. Without clear category boundary definitions, the classifier will guess wrong on several of these.

  • Save test_data.py.

Before you run the verification command, predict: how many total tickets do you expect in the list now?

  • Verify the full dataset by running:
python -c "from test_data import TEST_TICKETS; print(f'{len(TEST_TICKETS)} tickets, {len(set(t[\"expected\"] for t in TEST_TICKETS))} categories')"

You should see 24 tickets, 4 categories printed. That's 20 easy tickets plus your 4 hard edge cases, spread across all 4 classification categories.

Seeing a SyntaxError?

  • Check that every dict entry ends with a comma, including the last hard ticket before the closing ].
  • Make sure the closing ] is still present at the very bottom of the file after your additions.
  • Confirm all quote marks inside ticket text use straight quotes, not curly/smart quotes.

Need help? Help me fix a SyntaxError in my test_data.py file

✔️ Awesome, I've got everything!

Great. Double check you've saved test_data.py and that the import command returned 24 tickets, 4 categories.

ⓧ I'd like to double check the full code

Your test_data.py file should have these sections in order. Scroll up and confirm each is present:

  • TEST_TICKETS = [ opening on line 1.
  • 5 billing entries under the # Billing - Easy comment.
  • 5 technical entries under the # Technical - Easy comment.
  • 5 account entries under the # Account - Easy comment.
  • 5 feature request entries under the # Feature Request - Easy comment.
  • 4 ambiguous entries under the # Ambiguous / Hard cases comment.
  • A closing ] on the final line.

Your test dataset is locked and loaded. Next up, you'll run the classifier against all 24 tickets and discover exactly where it stumbles.

Run the Evaluation and Hit the Wall

Your classifier works on individual tickets, but you have no idea how well it performs across the full range of inputs.

In this step, you'll build an evaluation harness that runs every test ticket through the classifier and reveals exactly where it struggles. The ambiguous tickets you designed in Step 3 are about to expose gaps in your basic prompt.

In this step, get ready to:
  • Write an evaluation loop that processes all test tickets and collects predictions.
  • Compute accuracy, precision, recall, and a confusion matrix from the results.
  • Run the evaluation and identify where the classifier misclassifies.
Write the evaluation loop

The evaluation harness needs to iterate through every test ticket, call your classifier, and compare the prediction against the ground-truth label. It also tracks token usage so you know exactly what the evaluation costs.

  • Create a new file called run_eval.py in your project folder.
  • Add the imports at the top of the file:
import json
import time
from collections import defaultdict

from classifier import classify_ticket, SYSTEM_PROMPT_V1, SYSTEM_PROMPT_V2
from test_data import TEST_TICKETS

What do these imports do?

  • time provides sleep() for rate limiting between API calls.
  • defaultdict from collections creates the confusion matrix without needing to pre-initialize every cell.
  • You import both SYSTEM_PROMPT_V1 and SYSTEM_PROMPT_V2 so you can compare them head-to-head.
  • Below the imports, add the run_evaluation function that loops through all tickets and collects results:
def run_evaluation(tickets: list, system_prompt: str, label: str = "Evaluation") -> dict:
    """Run classifier against all tickets and compute metrics."""
    print(f"\n{'='*60}")
    print(f"  {label}")
    print(f"{'='*60}\n")

    results = []
    total_input_tokens = 0
    total_output_tokens = 0

    for i, ticket in enumerate(tickets):
        result = classify_ticket(ticket["text"], system_prompt=system_prompt)
        results.append({
            "text": ticket["text"][:50] + "...",
            "expected": ticket["expected"],
            "predicted": result["category"],
            "confidence": result["confidence"],
            "correct": result["category"] == ticket["expected"],
            "difficulty": ticket["difficulty"],
        })
        total_input_tokens += result["usage"]["input_tokens"]
        total_output_tokens += result["usage"]["output_tokens"]
        time.sleep(0.5)  # Rate limiting courtesy

    # Compute metrics
    categories = ["billing", "technical", "account", "feature_request"]
    metrics = compute_metrics(results, categories)
    metrics["total_input_tokens"] = total_input_tokens
    metrics["total_output_tokens"] = total_output_tokens
    metrics["estimated_cost_usd"] = (total_input_tokens / 1_000_000 * 2) + (total_output_tokens / 1_000_000 * 10)

    # Print results
    print_results(results, metrics, categories)
    return metrics

What does this function do?

  • It loops through every ticket, calls classify_ticket with the given system prompt, and stores both the prediction and whether it matched the expected label.
  • time.sleep(0.5) adds a half-second pause between API calls to avoid hitting rate limits.
  • After the loop, it delegates to compute_metrics and print_results (which you'll write next) and appends cost tracking to the metrics.
Compute precision, recall, and the confusion matrix

Raw accuracy tells you how many tickets were classified correctly overall. But per-category precision and recall reveal which categories the classifier is over-predicting or missing entirely.

  • Below run_evaluation, add the compute_metrics function:
def compute_metrics(results: list, categories: list) -> dict:
    """Compute accuracy, precision, recall, and confusion matrix."""
    correct = sum(1 for r in results if r["correct"])
    total = len(results)

    # Per-category precision and recall
    category_metrics = {}
    for cat in categories:
        true_positives = sum(1 for r in results if r["predicted"] == cat and r["expected"] == cat)
        false_positives = sum(1 for r in results if r["predicted"] == cat and r["expected"] != cat)
        false_negatives = sum(1 for r in results if r["predicted"] != cat and r["expected"] == cat)

        precision = true_positives / (true_positives + false_positives) if (true_positives + false_positives) > 0 else 0
        recall = true_positives / (true_positives + false_negatives) if (true_positives + false_negatives) > 0 else 0

        category_metrics[cat] = {"precision": precision, "recall": recall, "tp": true_positives, "fp": false_positives, "fn": false_negatives}

    # Confusion matrix
    confusion = defaultdict(lambda: defaultdict(int))
    for r in results:
        confusion[r["expected"]][r["predicted"]] += 1

    return {
        "accuracy": correct / total,
        "correct": correct,
        "total": total,
        "category_metrics": category_metrics,
        "confusion_matrix": dict(confusion),
    }

What does this function do?

  • For each category, it counts true positives (correctly predicted), false positives (predicted this category but wrong), and false negatives (should have been this category but wasn't).
  • Precision answers: "Of all tickets I predicted as billing, how many actually were billing?"
  • Recall answers: "Of all tickets that actually were billing, how many did I catch?"
  • The confusion matrix tracks every expected-vs-predicted pair so you can see exactly which categories get confused with each other.
Print the results and run the evaluation

The final piece formats the metrics into a readable report. It highlights misclassifications first so you immediately see what went wrong.

  • Below compute_metrics, add the print_results function:
def print_results(results: list, metrics: dict, categories: list):
    """Print formatted evaluation results."""
    # Misclassifications
    misclassified = [r for r in results if not r["correct"]]
    if misclassified:
        print("MISCLASSIFICATIONS:")
        for r in misclassified:
            print(f"  [{r['difficulty']}] {r['text']}")
            print(f"    Expected: {r['expected']} | Predicted: {r['predicted']} (conf: {r['confidence']:.2f})")
        print()

    # Per-category metrics
    print(f"{'Category':<16} {'Precision':<12} {'Recall':<12} {'TP':<5} {'FP':<5} {'FN':<5}")
    print("-" * 55)
    for cat in categories:
        m = metrics["category_metrics"][cat]
        print(f"{cat:<16} {m['precision']:<12.2f} {m['recall']:<12.2f} {m['tp']:<5} {m['fp']:<5} {m['fn']:<5}")

    # Confusion matrix
    print(f"\nConfusion Matrix (rows=expected, cols=predicted):")
    print(f"{'':>16}", end="")
    for cat in categories:
        print(f"{cat[:8]:>10}", end="")
    print()
    for expected in categories:
        print(f"{expected:<16}", end="")
        for predicted in categories:
            count = metrics["confusion_matrix"].get(expected, {}).get(predicted, 0)
            print(f"{count:>10}", end="")
        print()

    # Summary
    print(f"\nOVERALL ACCURACY: {metrics['accuracy']:.1%} ({metrics['correct']}/{metrics['total']})")
    print(f"API Usage: {metrics['total_input_tokens']} input + {metrics['total_output_tokens']} output tokens")
    print(f"Estimated Cost: ${metrics['estimated_cost_usd']:.4f}")

What does this function do?

  • It prints misclassifications first so you can see exactly which tickets tripped up the classifier and whether they were marked as easy or hard.
  • The per-category table shows precision and recall alongside raw TP/FP/FN counts.
  • The confusion matrix shows which categories get confused with which. Rows are expected labels, columns are what was predicted.
  • Finally, add the __main__ block at the bottom of run_eval.py to run both prompt variants and compare them:
if __name__ == "__main__":
    print("Running evaluation with initial prompt (V1)...")
    metrics_v1 = run_evaluation(TEST_TICKETS, SYSTEM_PROMPT_V1, label="V1: Basic Prompt")

    print("\n\nRunning evaluation with improved prompt (V2)...")
    metrics_v2 = run_evaluation(TEST_TICKETS, SYSTEM_PROMPT_V2, label="V2: Detailed Boundaries")

    # Comparison
    print(f"\n{'='*60}")
    print(f"  COMPARISON: V1 vs V2")
    print(f"{'='*60}")
    print(f"  V1 Accuracy: {metrics_v1['accuracy']:.1%}")
    print(f"  V2 Accuracy: {metrics_v2['accuracy']:.1%}")
    print(f"  Improvement: {(metrics_v2['accuracy'] - metrics_v1['accuracy']):.1%}")

What does this block do?

It runs the full 24-ticket evaluation twice. First with SYSTEM_PROMPT_V1 (the basic prompt), then with SYSTEM_PROMPT_V2 (the detailed boundaries prompt you already defined in classifier.py). Finally, it prints a head-to-head accuracy comparison.

  • Save run_eval.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved the file before running it.

ⓧ I'd like to double check the full code

Scroll up and confirm each of these sections is present in your run_eval.py, in this order:

  • Imports (6 lines): json, time, defaultdict, then your classifier and test_data imports.
  • run_evaluation function: loops tickets, calls classify_ticket, tracks tokens, returns metrics.
  • compute_metrics function: calculates TP/FP/FN, precision, recall, and builds the confusion matrix.
  • print_results function: prints misclassifications, category table, confusion matrix, and summary.
  • if __name__ == "__main__": block: runs V1, runs V2, prints comparison.

Now for the moment of truth. Before you run this, think about what you expect to happen. The V1 prompt gives Claude only brief category descriptions with no disambiguation guidance. Do you think it will handle the ambiguous tickets correctly, or will it stumble?

  • Run the evaluation by executing this command:
python run_eval.py

This takes about 30 seconds because it sends 48 API calls (24 tickets times 2 prompt variants) with a half-second pause between each.

What should I see?

The V1 results will show at least 2-3 misclassifications in the MISCLASSIFICATIONS section, all marked as [hard]. Overall accuracy for V1 will likely land between 70-85%.

The V2 results should show fewer misclassifications (possibly zero) and higher accuracy. The final COMPARISON section prints both scores side by side, making the improvement undeniable.

This is the designed obstacle. The V1 prompt cannot reliably handle tickets like "I can't log in and it was working fine yesterday" because it lacks rules for distinguishing account access issues from technical bugs. The eval makes this weakness visible and measurable.

Seeing an import error or API error?

  • If you see ModuleNotFoundError: No module named 'classifier', make sure you are running the command from the same folder that contains classifier.py and test_data.py.
  • If you see a rate limit error, the 0.5-second sleep between calls may not be enough. Increase time.sleep(0.5) to time.sleep(1.0) in the run_evaluation function.
  • If you see an authentication error, confirm your ANTHROPIC_API_KEY environment variable is still set in this terminal session.

My evaluation script is failing and I'm not sure why.

You now have hard numbers proving that vague prompts produce measurable failures on edge cases. In the next step, you'll dig into exactly why V2 fixes the mistakes V1 makes, and explore whether you can push accuracy even higher.

Improve the Classifier and Re-Evaluate

Your evaluation just proved what every Applied AI Architect encounters in production: a basic prompt handles obvious cases fine but crumbles on ambiguous inputs.

The fix is not a different model or more data. It is a better prompt. In this step, you will analyze exactly where V1 falls short, walk through an improved prompt with explicit category boundaries, and re-run the evaluation to prove the improvement with data.

In this step, get ready to:
  • Identify the misclassification pattern from your V1 evaluation results.
  • Examine the improved system prompt with explicit disambiguation rules.
  • Re-run the full evaluation comparing V1 and V2 side by side.
Analyze the misclassification pattern

Look back at the misclassifications your V1 evaluation printed. The hard tickets reveal a clear pattern in how the basic prompt fails.

Consider these two tickets from your test set:

  • "I can't log in and I think it might be a bug because it was working fine yesterday."
  • "I want to upgrade my plan but the upgrade button gives me an error."

Both are labeled technical because they reference something that used to work but now does not. V1 likely classified the first as account (login keyword) and the second as billing (plan upgrade keyword).

What is the root cause?

V1's category descriptions overlap. "Login issues" appears under account, but a broken login is a technical issue when it references prior working state.

Without explicit boundary rules, Claude picks the category whose keyword matches first. The fix is giving Claude decision logic, not just keyword lists.

Examine the improved system prompt

Your classifier.py already contains SYSTEM_PROMPT_V2. This is the improved version that directly addresses the patterns you just identified.

  • Open classifier.py and scroll to SYSTEM_PROMPT_V2.
  • Read through the CATEGORY DEFINITIONS WITH BOUNDARIES section and the DISAMBIGUATION RULES section.

Here is the key section that fixes the ambiguity problem:

DISAMBIGUATION RULES:
- "I can't log in" -> account (access issue, not a bug unless they say it was working before)
- "The login page is showing an error 500" -> technical (server error, not account access)
- "I want to upgrade my plan" -> billing (subscription change involves payment)
- "I want to add team members" -> account (permission/access management)
- "Can you add dark mode?" -> feature_request (new capability)
- "Dark mode is broken" -> technical (existing feature not working)

Why does this work?

V2 introduces two techniques that V1 lacked. First, each category gets a key signal (e.g., "something that USED TO WORK" for technical). This gives Claude a decision heuristic, not just a keyword match.

Second, the disambiguation rules show paired examples where the same surface language maps to different categories depending on context. This is exactly how you teach a classifier boundary cases.

Notice that the classify_ticket function already defaults to SYSTEM_PROMPT_V2. Your run_eval.py explicitly passes both prompts so you can compare them head-to-head.

Re-run the evaluation and compare results

Your run_eval.py script already runs both V1 and V2 back-to-back and prints a side-by-side comparison. Time to see whether the improved prompt actually delivers.

Before you run it, predict: do you think V2 will get all 24 tickets correct, or will some of the hard cases still trip it up?

  • Run the full evaluation comparison by running this command:
python run_eval.py

This takes about 30 seconds. The script classifies all 24 tickets twice (once with each prompt) with a half-second pause between each API call for rate limiting.

At the bottom of the output, you will see a comparison block that looks like this:

============================================================
  COMPARISON: V1 vs V2
============================================================
  V1 Accuracy: 83.3%
  V2 Accuracy: 95.8%
  Improvement: 12.5%

What should I see?

V1 typically lands between 70-85% accuracy. V2 should land above 90%. Your exact numbers may differ slightly because LLM responses are non-deterministic, but the gap between V1 and V2 should be clear.

Look at the confusion matrices. V2's matrix should have fewer off-diagonal entries, meaning fewer misclassifications between categories that share overlapping keywords.

V2 accuracy not much better than V1?

LLM outputs are non-deterministic. Run the evaluation one more time. If V2 still shows no improvement, confirm that SYSTEM_PROMPT_V2 in your classifier.py includes the full DISAMBIGUATION RULES section with all six example pairs.

Also verify that run_eval.py passes SYSTEM_PROMPT_V2 as the second argument in its run_evaluation call, not SYSTEM_PROMPT_V1 twice.

Help me debug my evaluation comparison.

✔️ Awesome, I've got everything!

Great. Confirm you have saved both classifier.py and run_eval.py and that your comparison output shows V2 outperforming V1.

ⓧ I'd like to double check the full code

Your project has three files. Scroll up and confirm each matches what you built in prior steps. The files are:

  • classifier.py with CLASSIFICATION_TOOL, SYSTEM_PROMPT_V1, SYSTEM_PROMPT_V2, and the classify_ticket function.
  • test_data.py with 24 tickets (20 easy + 4 hard).
  • run_eval.py with run_evaluation, compute_metrics, print_results, and the __main__ block running both V1 and V2 with a comparison summary.

You just completed the full evaluation-driven improvement cycle: prototype, measure, identify weakness, fix the prompt, and prove the fix with data. This is exactly what Applied AI Architects do when deploying Claude for enterprise classification workloads.

Next up, you will take this further with an automated prompt optimization loop that tests multiple strategies and picks the winner.

Secret mission

Build a Prompt Optimization Loop

You manually improved your prompt in Step 5. But what if you could test multiple prompt strategies automatically and let the data tell you which one wins? In this mission, you build an optimization loop that evaluates 3 different prompting approaches and produces a comparison report with accuracy, cost, and a declared winner.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs locally with pay-per-use API calls, so there are no ongoing costs unless you actively make new requests.

Resources you used:

  • Anthropic API key (stored as the ANTHROPIC_API_KEY environment variable).
  • Local project files (classifier.py, test_data.py, run_eval.py, optimize.py).
  • Python virtual environment inside your project folder.
  • API usage: approximately $0.30-$0.50 total across all evaluation runs.

Keep everything running

No action needed. Choose this if you plan to keep experimenting with different prompt strategies or extending the evaluation framework.

  • Your API key remains active for future projects. No charges are incurred unless you make new API calls.
  • Your local project files stay on your Desktop ready to use.
  • The virtual environment stays in place so you can rerun evaluations at any time.

Pause - I'll come back to this later

No special pause action is needed. The Anthropic API is strictly pay-per-use, so you are only charged when you make calls.

  • Close any open terminal sessions running the project.
  • Your files and virtual environment remain on disk for when you return.
  • Your API key stays active but costs nothing while idle.

Delete - I don't want to use this again

Remove all project resources and revoke your API access. Follow each step below.

  • Delete your local project folder (including the virtual environment) by running the appropriate command for your operating system.
  • macOS/Linux:
cd ~/Desktop
rm -rf ticket-classifier
  • Windows (PowerShell):
cd ~\Desktop
Remove-Item -Recurse -Force ticket-classifier
  • Revoke your Anthropic API key so it can no longer be used.
  • Open platform.claude.com/settings/keys in your browser.
  • Find the API key you created for this project.
  • Click the key's menu and select the option to disable or delete it.
  • Unset the environment variable from your current terminal session.

macOS

unset ANTHROPIC_API_KEY

Windows

Remove-Item Env:ANTHROPIC_API_KEY

Nice Work!

Nice Work!

You just built a complete support ticket classification system with a structured evaluation framework powered by Claude. This is exactly the kind of end-to-end AI integration work that Applied AI Architects deliver for enterprise customers.

You've learned how to:

  • Built a Claude-powered classifier using the tool use pattern to guarantee structured JSON output with category, confidence, and reasoning fields.
  • Designed and ran a reusable evaluation framework that measures accuracy, precision, recall, and confusion matrices across a labeled test dataset with deliberately ambiguous edge cases.
  • Identified classifier weaknesses through evaluation, improved the system prompt with explicit disambiguation rules, and proved the improvement with data. This is the full prototype-evaluate-iterate-prove cycle.
  • Secret Mission: Built an automated prompt optimization loop that tests multiple prompt strategies and compares their eval scores with cost tracking to find the best-performing configuration.

Ready to quiz yourself?