Build an Intelligent Ticket Router

Classify support tickets with an LLM and evaluate quality using LLM-as-a-judge.

Introduction

30 Second Summary

Every support team deals with tickets landing in the wrong inbox. When a billing question ends up with engineering, customers wait longer and teams waste time sorting through irrelevant requests.

In this project, you will build an intelligent ticket routing system that uses an LLM to classify support tickets into departments, then evaluates its own classification quality using a second LLM call as a judge.

What You'll Build

An LLM-powered ticket classification and evaluation pipeline that routes support tickets to the correct department and measures its own accuracy using Groq's structured outputs API.

By the end of this project, you'll have:

  • A ticket classifier that uses structured JSON outputs to reliably route tickets to IT, HR, Sales, Billing, or Engineering.
  • An LLM-as-a-judge evaluator that scores each classification on correctness, reasoning quality, and routing appropriateness.
  • A metrics summary report showing accuracy, confidence distribution, and evaluation insights across a batch of 10 test tickets.
  • Secret Mission: A prompt experiment that tests two different system prompts and compares which one produces better classifications.

Are there any prerequisites?

You need basic Python knowledge (variables, functions, loops). No prior LLM or API experience is required.

Set Up Your Python Environment and Groq API Key

Your ticket router needs to talk to a language model to classify support tickets. Groq provides the API for this, giving you access to powerful models with structured output support.

You already have Python, the Groq SDK, and your API key ready. In this step, you'll confirm everything is connected by making your first API call.

In this step, get ready to:
  • Create a test script that calls the Groq API.
  • Verify the connection succeeds and you can receive a response.
Test your Groq API connection

The best way to confirm your environment is working is to make a real API call. You'll create a short script that sends a message to the openai/gpt-oss-120b model and prints the response.

  • In VS Code, create a new file called test_connection.py inside your ticket-router folder.
  • Add the following code to make a simple chat completion call:
import os
from groq import Groq

# Create a Groq client using your API key from the environment
client = Groq(
    api_key=os.environ.get("GROQ_API_KEY"),
)

# Make a simple chat completion request to test the connection
chat_completion = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "Say hello in one sentence.",
        }
    ],
    model="openai/gpt-oss-120b",
)

# Print the model's response and a success message
print(chat_completion.choices[0].message.content)
print("\nConnection successful! Your Groq API key is working.")

What does this code do?

This script tests three things at once: that your API key is valid, that the Groq SDK can reach the API, and that the openai/gpt-oss-120b model is available.

  • os.environ.get("GROQ_API_KEY") reads your API key from the environment variable you set earlier.
  • client.chat.completions.create(...) sends a message to the model and waits for a response.
  • response.choices[0].message.content extracts the text the model generated.
  • Save test_connection.py.
  • Run the script in your terminal:
python test_connection.py

You should see a one-sentence greeting from the model, followed by the message "Connection successful! Your Groq API key is working."

Not seeing a response?

  • If you see an "AuthenticationError" or "Invalid API Key" message, double-check that your GROQ_API_KEY environment variable is set correctly in the same terminal session you're running the script from.
  • If you see a "ModuleNotFoundError: No module named 'groq'", make sure you installed the SDK with pip install groq==1.4.0 in the same Python environment you're using to run the script.
  • If you get a 429 rate-limit error, wait a moment and try again. Groq's free tier never charges you. It just rate-limits requests.

Still stuck? Help me debug my Groq API connection test.

Why openai/gpt-oss-120b?

This model supports strict structured outputs, which means it can guarantee schema-compliant JSON responses. That's essential for our ticket classifier, where we need reliable fields like category and confidence score every single time.

Older models like llama-3.3-70b-versatile are being deprecated. openai/gpt-oss-120b is Groq's recommended replacement.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file.

ⓧ I'd like to double check the full code

import os
from groq import Groq

client = Groq(
    api_key=os.environ.get("GROQ_API_KEY"),
)

chat_completion = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "Say hello in one sentence.",
        }
    ],
    model="openai/gpt-oss-120b",
)

print(chat_completion.choices[0].message.content)
print("\nConnection successful! Your Groq API key is working.")

Your environment is ready and your API connection is confirmed. Next up, you'll build the ticket classifier that uses structured outputs to reliably extract categories from support tickets.

Build the Ticket Classifier with Structured Outputs

Your environment is ready and your API connection is confirmed. Now it's time to build the core feature: a ticket classifier that reads support tickets and routes them to the right department.

The challenge with using an LLM for classification is that language models return freeform text by default. You need structured, predictable output that your code can parse reliably. Groq's structured outputs feature solves this by constraining the model to always return valid JSON matching a schema you define.

In this step, get ready to:
  • Create a dataset of 10 simulated support tickets with ground truth labels.
  • Define a JSON schema that constrains the LLM's output format.
  • Run the classifier on all tickets and measure its accuracy.
Create the ticket data and classification schema

Every classification system needs test data with known correct answers. These known answers are called ground truth labels. You'll compare the model's predictions against them to measure accuracy.

  • In VS Code, create a new file called classifier.py inside your ticket-router folder.
  • Add the imports, Groq client, and ticket dataset by pasting this code:
import os
import json
from groq import Groq

client = Groq(
    api_key=os.environ.get("GROQ_API_KEY"),
)

TICKETS = [
    {"id": 1, "text": "My laptop won't connect to the VPN. I've tried restarting but still getting a timeout error.", "ground_truth": "IT"},
    {"id": 2, "text": "I need to update my direct deposit information for my paycheck next month.", "ground_truth": "HR"},
    {"id": 3, "text": "We're interested in upgrading our enterprise plan. Can someone walk us through pricing for 500 seats?", "ground_truth": "Sales"},
    {"id": 4, "text": "I was charged twice for my subscription this month. Please refund the duplicate charge.", "ground_truth": "Billing"},
    {"id": 5, "text": "The API endpoint /v2/workflows returns a 500 error when payload exceeds 10MB.", "ground_truth": "Engineering"},
    {"id": 6, "text": "How do I reset my password? The reset email never arrives.", "ground_truth": "IT"},
    {"id": 7, "text": "I'd like to request time off for the last two weeks of December.", "ground_truth": "HR"},
    {"id": 8, "text": "Can you send me a copy of our last invoice? We need it for our quarterly audit.", "ground_truth": "Billing"},
    {"id": 9, "text": "We're evaluating automation tools for our DevOps team. Can we schedule a demo?", "ground_truth": "Sales"},
    {"id": 10, "text": "The webhook integration drops events when there are more than 100 concurrent connections.", "ground_truth": "Engineering"},
]

What does this code do?

This sets up your Groq client (just like in test_connection.py) and defines 10 simulated support tickets.

Each ticket has an id, the ticket text, and a ground_truth label. The ground truth is the correct department that ticket should be routed to. You'll use these labels later to score the model's predictions.

You won't see a change yet. The next chunk adds the schema and prompt that tell the model how to respond.

  • Below the TICKETS list, add the classification schema that defines the exact JSON structure the model must return:
CLASSIFICATION_SCHEMA = {
    "type": "object",
    "properties": {
        "category": {
            "type": "string",
            "enum": ["IT", "HR", "Sales", "Billing", "Engineering"]
        },
        "priority": {
            "type": "string",
            "enum": ["low", "medium", "high", "critical"]
        },
        "confidence_score": {
            "type": "number"
        },
        "reasoning": {
            "type": "string"
        }
    },
    "required": ["category", "priority", "confidence_score", "reasoning"],
    "additionalProperties": False
}

What is a JSON schema?

A JSON schema is a blueprint that defines what fields the response must contain and what types they must be. When you pass this schema to Groq with strict: True, the model is constrained at the token level to only produce output matching this structure.

The enum fields lock the model into specific allowed values. It can only return one of the five categories and one of the four priority levels. The additionalProperties: False prevents the model from sneaking extra fields into its response.

  • Below the schema, add the system prompt that instructs the model how to classify tickets:
SYSTEM_PROMPT = """You are an intelligent support ticket router for an enterprise automation platform.
Your job is to classify incoming support tickets into the correct department for handling.

Categories:
- IT: Hardware, software, network, access, and password issues
- HR: Payroll, time off, benefits, hiring, and employee relations
- Sales: Pricing inquiries, demos, upgrades, and new business
- Billing: Invoices, charges, refunds, and payment issues
- Engineering: API bugs, integrations, performance issues, and technical defects

Analyze the ticket carefully. Provide your confidence score as a decimal between 0 and 1.
Be precise in your reasoning."""

Why does the prompt matter so much?

The system prompt is where you teach the model what each category means. Without clear category definitions, the model would have to guess what "IT" vs "Engineering" means. Explicit boundaries ("API bugs" go to Engineering, "password issues" go to IT) reduce ambiguity and improve accuracy.

  • Save classifier.py.
  • Confirm there are no syntax errors by running this in your terminal:
python classifier.py
  • You should see no output and no errors. The script defines data but doesn't do anything with it yet.

Seeing a SyntaxError?

  • Check that the triple-quoted string for SYSTEM_PROMPT uses three double quotes (""") at both the opening and closing.
  • Make sure additionalProperties: False uses a capital F. In Python, the boolean is False, not false.
  • Verify every dictionary in the TICKETS list ends with a comma, including the last one.

Help me debug a SyntaxError in my classifier.py file.

Implement the classify_ticket function

With your data and schema in place, you can now write the function that sends each ticket to the LLM and gets back a structured classification.

  • Below SYSTEM_PROMPT, add the classify_ticket function:
def classify_ticket(ticket_text):
    response = client.chat.completions.create(
        model="openai/gpt-oss-120b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"Classify this support ticket:\n\n{ticket_text}"}
        ],
        response_format={
            "type": "json_schema",
            "json_schema": {
                "name": "ticket_classification",
                "strict": True,
                "schema": CLASSIFICATION_SCHEMA
            }
        },
        temperature=0.1
    )
    return json.loads(response.choices[0].message.content)

What does this function do?

  • response_format with "type": "json_schema" and "strict": True tells Groq to guarantee the response matches your schema exactly. The model cannot return anything else.
  • temperature=0.1 makes the model's responses nearly deterministic. Lower temperature means less randomness, which is what you want for consistent classification.
  • json.loads() converts the model's string response into a Python dictionary you can work with programmatically.

Why structured outputs instead of plain JSON mode?

Plain JSON mode only guarantees valid JSON syntax. Structured outputs with strict: True guarantees 100% schema adherence. The model is constrained at the token level, so it physically cannot produce a response missing a required field or using an invalid category value.

You won't see a change yet. The next chunk builds the loop that calls this function on every ticket.

Run classification on all 10 tickets and print results

Now you'll write the function that loops through every ticket, classifies it, and tracks how many predictions match the ground truth.

  • Below the classify_ticket function, add the run_classifier function and the main entry point:
def run_classifier():
    results = []
    correct = 0

    print("=" * 60)
    print("TICKET CLASSIFICATION RESULTS")
    print("=" * 60)

    for ticket in TICKETS:
        classification = classify_ticket(ticket["text"])
        is_correct = classification["category"] == ticket["ground_truth"]
        if is_correct:
            correct += 1

        result = {
            "ticket_id": ticket["id"],
            "ticket_text": ticket["text"],
            "ground_truth": ticket["ground_truth"],
            "predicted": classification["category"],
            "priority": classification["priority"],
            "confidence": classification["confidence_score"],
            "reasoning": classification["reasoning"],
            "is_correct": is_correct
        }
        results.append(result)

        status = "CORRECT" if is_correct else "WRONG"
        print(f"\nTicket #{ticket['id']}: [{status}]")
        print(f"  Text: {ticket['text'][:60]}...")
        print(f"  Predicted: {classification['category']} | Actual: {ticket['ground_truth']}")
        print(f"  Confidence: {classification['confidence_score']:.2f} | Priority: {classification['priority']}")

    accuracy = correct / len(TICKETS)
    print(f"\n{'=' * 60}")
    print(f"ACCURACY: {correct}/{len(TICKETS)} = {accuracy:.0%}")
    print(f"{'=' * 60}")

    return results


if __name__ == "__main__":
    results = run_classifier()

What does this code do?

  • The loop calls classify_ticket() on each ticket and compares the model's predicted category against the ground truth label.
  • Each result is stored as a dictionary with both the prediction and the ground truth. This structure makes it easy for the evaluator (in the next step) to review every decision.
  • The accuracy score at the end is your first product metric. It tells you what percentage of tickets the model routed correctly.
  • The if __name__ == "__main__" block means this code only runs when you execute the file directly. Later steps will import run_classifier without triggering the full run.
  • Save classifier.py.
  • Run the classifier by executing this command in your terminal:
python classifier.py
  • You should see each ticket's classification printed with a [CORRECT] or [WRONG] label, followed by an accuracy score at the bottom (e.g. ACCURACY: 9/10 = 90%).

What should I expect for accuracy?

Most runs score between 80% and 100%. The model is strong at clear-cut tickets like VPN issues (IT) or refund requests (Billing). Tickets with ambiguity (like a demo request that could be Sales or Engineering) are where you might see a mismatch.

Getting an API error or no output?

  • If you see a rate limit error (status 429), wait 30 seconds and try again. Groq's free tier rate-limits requests rather than charging you.
  • If you see an authentication error, confirm your GROQ_API_KEY environment variable is still set in the same terminal session.
  • If the script hangs, the model is processing. It can take 20-40 seconds per ticket because openai/gpt-oss-120b is a large model. Let it finish.

Help me troubleshoot my classifier.py script that isn't returning results.

✔️ Awesome, I've got everything!

Great. Double check you've saved classifier.py and that your terminal shows the accuracy score at the bottom of the output.

ⓧ I'd like to double check the full code

Here is the complete classifier.py file. Compare it against yours to spot any differences.

import os
import json
from groq import Groq

client = Groq(
    api_key=os.environ.get("GROQ_API_KEY"),
)

TICKETS = [
    {"id": 1, "text": "My laptop won't connect to the VPN. I've tried restarting but still getting a timeout error.", "ground_truth": "IT"},
    {"id": 2, "text": "I need to update my direct deposit information for my paycheck next month.", "ground_truth": "HR"},
    {"id": 3, "text": "We're interested in upgrading our enterprise plan. Can someone walk us through pricing for 500 seats?", "ground_truth": "Sales"},
    {"id": 4, "text": "I was charged twice for my subscription this month. Please refund the duplicate charge.", "ground_truth": "Billing"},
    {"id": 5, "text": "The API endpoint /v2/workflows returns a 500 error when payload exceeds 10MB.", "ground_truth": "Engineering"},
    {"id": 6, "text": "How do I reset my password? The reset email never arrives.", "ground_truth": "IT"},
    {"id": 7, "text": "I'd like to request time off for the last two weeks of December.", "ground_truth": "HR"},
    {"id": 8, "text": "Can you send me a copy of our last invoice? We need it for our quarterly audit.", "ground_truth": "Billing"},
    {"id": 9, "text": "We're evaluating automation tools for our DevOps team. Can we schedule a demo?", "ground_truth": "Sales"},
    {"id": 10, "text": "The webhook integration drops events when there are more than 100 concurrent connections.", "ground_truth": "Engineering"},
]

CLASSIFICATION_SCHEMA = {
    "type": "object",
    "properties": {
        "category": {
            "type": "string",
            "enum": ["IT", "HR", "Sales", "Billing", "Engineering"]
        },
        "priority": {
            "type": "string",
            "enum": ["low", "medium", "high", "critical"]
        },
        "confidence_score": {
            "type": "number"
        },
        "reasoning": {
            "type": "string"
        }
    },
    "required": ["category", "priority", "confidence_score", "reasoning"],
    "additionalProperties": False
}

SYSTEM_PROMPT = """You are an intelligent support ticket router for an enterprise automation platform.
Your job is to classify incoming support tickets into the correct department for handling.

Categories:
- IT: Hardware, software, network, access, and password issues
- HR: Payroll, time off, benefits, hiring, and employee relations
- Sales: Pricing inquiries, demos, upgrades, and new business
- Billing: Invoices, charges, refunds, and payment issues
- Engineering: API bugs, integrations, performance issues, and technical defects

Analyze the ticket carefully. Provide your confidence score as a decimal between 0 and 1.
Be precise in your reasoning."""


def classify_ticket(ticket_text):
    response = client.chat.completions.create(
        model="openai/gpt-oss-120b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"Classify this support ticket:\n\n{ticket_text}"}
        ],
        response_format={
            "type": "json_schema",
            "json_schema": {
                "name": "ticket_classification",
                "strict": True,
                "schema": CLASSIFICATION_SCHEMA
            }
        },
        temperature=0.1
    )
    return json.loads(response.choices[0].message.content)


def run_classifier():
    results = []
    correct = 0

    print("=" * 60)
    print("TICKET CLASSIFICATION RESULTS")
    print("=" * 60)

    for ticket in TICKETS:
        classification = classify_ticket(ticket["text"])
        is_correct = classification["category"] == ticket["ground_truth"]
        if is_correct:
            correct += 1

        result = {
            "ticket_id": ticket["id"],
            "ticket_text": ticket["text"],
            "ground_truth": ticket["ground_truth"],
            "predicted": classification["category"],
            "priority": classification["priority"],
            "confidence": classification["confidence_score"],
            "reasoning": classification["reasoning"],
            "is_correct": is_correct
        }
        results.append(result)

        status = "CORRECT" if is_correct else "WRONG"
        print(f"\nTicket #{ticket['id']}: [{status}]")
        print(f"  Text: {ticket['text'][:60]}...")
        print(f"  Predicted: {classification['category']} | Actual: {ticket['ground_truth']}")
        print(f"  Confidence: {classification['confidence_score']:.2f} | Priority: {classification['priority']}")

    accuracy = correct / len(TICKETS)
    print(f"\n{'=' * 60}")
    print(f"ACCURACY: {correct}/{len(TICKETS)} = {accuracy:.0%}")
    print(f"{'=' * 60}")

    return results


if __name__ == "__main__":
    results = run_classifier()

Your classifier is routing tickets and measuring its own accuracy. Next up, you'll build an LLM-as-a-judge evaluator that scores how good each classification decision actually was.

Build the LLM-as-a-Judge Evaluator

Your classifier can route tickets, but how do you know if it's doing a good job? In production ML systems, you need a way to measure quality automatically.

In this step, you'll build an LLM-as-a-judge evaluator. This is a second LLM call that reviews each classification and scores it on correctness, reasoning quality, and routing appropriateness.

In this step, get ready to:
  • Define a structured evaluation schema with scoring dimensions.
  • Build a judge function that scores each classification decision.
  • Run the evaluator across all classified tickets and review the scores.
Create the evaluator file and judge schema

The judge needs its own structured output schema, just like the classifier. Instead of categories and priorities, this schema defines scoring dimensions that rate how good each classification was.

  • Create a new file called evaluator.py in your ticket-router folder.
  • Add the imports, Groq client, and evaluation schema by pasting the following code:
import os
import json
from groq import Groq

client = Groq(
    api_key=os.environ.get("GROQ_API_KEY"),
)

# Define the structured output schema for evaluation scores
EVALUATION_SCHEMA = {
    "type": "object",
    "properties": {
        "correctness_score": {
            "type": "integer"
        },
        "reasoning_quality": {
            "type": "integer"
        },
        "routing_appropriateness": {
            "type": "integer"
        },
        "explanation": {
            "type": "string"
        },
        "suggested_category": {
            "type": "string"
        }
    },
    "required": ["correctness_score", "reasoning_quality", "routing_appropriateness", "explanation", "suggested_category"],
    "additionalProperties": False
}

What does this schema define?

This schema tells the LLM judge exactly what format to return its evaluation in.

  • correctness_score (1-5): Did the classifier pick the right category?
  • reasoning_quality (1-5): Was the reasoning logical?
  • routing_appropriateness (1-5): Even if wrong, was the routing reasonable?
  • suggested_category: What category the judge thinks is correct.
  • Save evaluator.py.

Seeing a red underline on the import?

If VS Code shows a warning on from groq import Groq, make sure you installed the SDK in Step 1 with pip install groq==1.4.0. Also check that VS Code is using the same Python interpreter where you installed the package.

Help me fix my Groq import error in VS Code.

Implement the evaluate_classification function

The judge needs instructions on how to evaluate. You'll define a system prompt that tells it to be a strict evaluator, then build the function that sends each classification for review.

  • In evaluator.py, add the judge system prompt below the EVALUATION_SCHEMA:
# Instructions for the judge LLM
JUDGE_SYSTEM_PROMPT = """You are a strict evaluator of support ticket routing decisions.
You will be given a support ticket, the classification that was made, and the correct category.

Score each dimension from 1-5:
- correctness_score: Did the router pick the right category? (5 = perfect match, 1 = completely wrong)
- reasoning_quality: Is the reasoning logical and well-explained? (5 = excellent reasoning, 1 = no logic)
- routing_appropriateness: Even if not the ground truth, is the routing reasonable? (5 = perfectly appropriate, 1 = nonsensical)

Provide your explanation of the scores and suggest what category you think is correct."""

Why a separate system prompt for the judge?

The classifier's prompt tells it to route tickets. The judge's prompt tells it to evaluate routing decisions. These are two completely different tasks, so they need different instructions.

By giving the judge explicit scoring criteria (1-5 scale with definitions), you get consistent, comparable scores across all tickets.

  • Now add the evaluate_classification function below the JUDGE_SYSTEM_PROMPT:
def evaluate_classification(ticket_text, classification, ground_truth):
    # Build the evaluation prompt with all context the judge needs
    user_message = f"""Evaluate this ticket routing decision:

TICKET: {ticket_text}

CLASSIFICATION MADE:
- Category: {classification['predicted']}
- Priority: {classification['priority']}
- Confidence: {classification['confidence']}
- Reasoning: {classification['reasoning']}

GROUND TRUTH CATEGORY: {ground_truth}

Score this classification."""

    # Call the LLM as a judge with structured output
    response = client.chat.completions.create(
        model="openai/gpt-oss-120b",
        messages=[
            {"role": "system", "content": JUDGE_SYSTEM_PROMPT},
            {"role": "user", "content": user_message}
        ],
        response_format={
            "type": "json_schema",
            "json_schema": {
                "name": "ticket_evaluation",
                "strict": True,
                "schema": EVALUATION_SCHEMA
            }
        },
        temperature=0.1
    )
    return json.loads(response.choices[0].message.content)

What does this function do?

This function sends a structured evaluation request to the same openai/gpt-oss-120b model, but with a completely different role.

  • The user_message gives the judge everything it needs: the original ticket, the classification that was made, and the ground truth answer.
  • The response_format with strict: True guarantees the judge returns valid scores in the exact schema you defined.
  • temperature=0.1 keeps the judge consistent. You want repeatable scores, not creative ones.
  • Save evaluator.py. You won't see a change yet. The next chunk runs the evaluator and prints results.
Run evaluation on all classified tickets

Now you need a function that loops through all classification results and runs the judge on each one. This function also imports the classifier so it can run the full pipeline end to end.

  • Add the run_evaluator function and the main block at the bottom of evaluator.py:
def run_evaluator(classification_results):
    evaluations = []

    print("\n" + "=" * 60)
    print("LLM-AS-A-JUDGE EVALUATION RESULTS")
    print("=" * 60)

    # Evaluate each classification result
    for result in classification_results:
        evaluation = evaluate_classification(
            result["ticket_text"],
            result,
            result["ground_truth"]
        )
        evaluation["ticket_id"] = result["ticket_id"]
        evaluations.append(evaluation)

        # Print per-ticket scores
        print(f"\nTicket #{result['ticket_id']}:")
        print(f"  Correctness: {evaluation['correctness_score']}/5")
        print(f"  Reasoning Quality: {evaluation['reasoning_quality']}/5")
        print(f"  Routing Appropriateness: {evaluation['routing_appropriateness']}/5")
        print(f"  Judge's Category: {evaluation['suggested_category']}")
        print(f"  Explanation: {evaluation['explanation'][:80]}...")

    return evaluations


if __name__ == "__main__":
    from classifier import run_classifier
    results = run_classifier()
    evaluations = run_evaluator(results)

What does this code do?

run_evaluator() takes the list of classification results from Step 2 and sends each one through the judge. It prints scores for each ticket so you can see how the judge rated the classifier's decisions.

The if __name__ == "__main__" block imports run_classifier from your own classifier.py file. This runs the full pipeline: classify all 10 tickets first, then evaluate each classification.

  • Save evaluator.py.
  • Run the evaluator by running this command in your terminal:
python evaluator.py
  • You should see the classifier results first, followed by the judge's evaluation scores for each ticket. Each ticket shows a correctness score, reasoning quality score, routing appropriateness score, and the judge's suggested category.

Why does it take a moment to run?

The script makes 20 API calls total: 10 to classify the tickets, then 10 more to evaluate those classifications. Each call goes to Groq's servers and back. This is normal for batch processing with LLM APIs.

Seeing a rate limit error?

Groq's free tier has rate limits. If you see a 429 error, wait 30-60 seconds and run the script again. The free tier is never charged. It simply pauses you when you hit the limit.

If you see a module not found error for classifier, make sure your terminal is inside the ticket-router folder where both files live.

Help me debug my evaluator.py errors.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

Here's the complete evaluator.py for comparison:

import os
import json
from groq import Groq

client = Groq(
    api_key=os.environ.get("GROQ_API_KEY"),
)

EVALUATION_SCHEMA = {
    "type": "object",
    "properties": {
        "correctness_score": {
            "type": "integer"
        },
        "reasoning_quality": {
            "type": "integer"
        },
        "routing_appropriateness": {
            "type": "integer"
        },
        "explanation": {
            "type": "string"
        },
        "suggested_category": {
            "type": "string"
        }
    },
    "required": ["correctness_score", "reasoning_quality", "routing_appropriateness", "explanation", "suggested_category"],
    "additionalProperties": False
}

JUDGE_SYSTEM_PROMPT = """You are a strict evaluator of support ticket routing decisions.
You will be given a support ticket, the classification that was made, and the correct category.

Score each dimension from 1-5:
- correctness_score: Did the router pick the right category? (5 = perfect match, 1 = completely wrong)
- reasoning_quality: Is the reasoning logical and well-explained? (5 = excellent reasoning, 1 = no logic)
- routing_appropriateness: Even if not the ground truth, is the routing reasonable? (5 = perfectly appropriate, 1 = nonsensical)

Provide your explanation of the scores and suggest what category you think is correct."""


def evaluate_classification(ticket_text, classification, ground_truth):
    user_message = f"""Evaluate this ticket routing decision:

TICKET: {ticket_text}

CLASSIFICATION MADE:
- Category: {classification['predicted']}
- Priority: {classification['priority']}
- Confidence: {classification['confidence']}
- Reasoning: {classification['reasoning']}

GROUND TRUTH CATEGORY: {ground_truth}

Score this classification."""

    response = client.chat.completions.create(
        model="openai/gpt-oss-120b",
        messages=[
            {"role": "system", "content": JUDGE_SYSTEM_PROMPT},
            {"role": "user", "content": user_message}
        ],
        response_format={
            "type": "json_schema",
            "json_schema": {
                "name": "ticket_evaluation",
                "strict": True,
                "schema": EVALUATION_SCHEMA
            }
        },
        temperature=0.1
    )
    return json.loads(response.choices[0].message.content)


def run_evaluator(classification_results):
    evaluations = []

    print("\n" + "=" * 60)
    print("LLM-AS-A-JUDGE EVALUATION RESULTS")
    print("=" * 60)

    for result in classification_results:
        evaluation = evaluate_classification(
            result["ticket_text"],
            result,
            result["ground_truth"]
        )
        evaluation["ticket_id"] = result["ticket_id"]
        evaluations.append(evaluation)

        print(f"\nTicket #{result['ticket_id']}:")
        print(f"  Correctness: {evaluation['correctness_score']}/5")
        print(f"  Reasoning Quality: {evaluation['reasoning_quality']}/5")
        print(f"  Routing Appropriateness: {evaluation['routing_appropriateness']}/5")
        print(f"  Judge's Category: {evaluation['suggested_category']}")
        print(f"  Explanation: {evaluation['explanation'][:80]}...")

    return evaluations


if __name__ == "__main__":
    from classifier import run_classifier
    results = run_classifier()
    evaluations = run_evaluator(results)

Your evaluator is now scoring every classification across three quality dimensions. Next up, you'll aggregate all these scores into a metrics summary report that reveals how well your router performs overall.

Aggregate Metrics and Generate the Summary Report

Your classifier routes tickets and your evaluator scores those decisions. Right now, those two pieces run separately with their own print statements.

In this step, you'll create an orchestration script that runs both modules in sequence, computes product-level metrics across all 10 tickets, and prints a single summary report. This is the deliverable that proves your pipeline works end-to-end.

In this step, get ready to:
  • Create metrics.py that imports and runs the full pipeline.
  • Compute aggregate metrics like accuracy, confidence, and judge agreement rate.
  • Print a formatted summary report with a per-ticket breakdown table.
Create metrics.py and import the pipeline modules

This new file ties together your classifier and evaluator into one executable pipeline. It imports the functions you already built and adds the metrics logic on top.

  • In VS Code, create a new file called metrics.py inside your ticket-router folder.
Compute aggregate metrics

The compute_metrics function takes in both sets of results and calculates the numbers that matter: how accurate the classifier is, how confident it is on average, and how often the judge agrees with the classifier's routing decisions.

  • Add the following imports and compute_metrics function to metrics.py:
from classifier import run_classifier
from evaluator import run_evaluator


def compute_metrics(classification_results, evaluations):
    # Count how many the classifier got right
    total = len(classification_results)
    correct = sum(1 for r in classification_results if r["is_correct"])
    accuracy = correct / total

    # Average confidence from the classifier's own scores
    avg_confidence = sum(r["confidence"] for r in classification_results) / total
    # Average scores from the judge's evaluations
    avg_correctness = sum(e["correctness_score"] for e in evaluations) / total
    avg_reasoning = sum(e["reasoning_quality"] for e in evaluations) / total
    avg_appropriateness = sum(e["routing_appropriateness"] for e in evaluations) / total

    # How often does the judge agree with the classifier's pick?
    judge_agrees = sum(
        1 for r, e in zip(classification_results, evaluations)
        if e["suggested_category"].upper() == r["predicted"].upper()
    )
    agreement_rate = judge_agrees / total

    return {
        "accuracy": accuracy,
        "correct": correct,
        "total": total,
        "avg_confidence": avg_confidence,
        "avg_correctness": avg_correctness,
        "avg_reasoning": avg_reasoning,
        "avg_appropriateness": avg_appropriateness,
        "agreement_rate": agreement_rate,
        "judge_agrees": judge_agrees,
    }

What does this code do?

The imports bring in both pipeline functions from your other files. Python treats each .py file as a module, so you can import specific functions directly.

The compute_metrics function calculates five key metrics. accuracy compares predicted vs ground truth. agreement_rate measures how often the judge's suggested category matches the classifier's prediction, which tells you whether an independent reviewer would route the same way.

  • Save metrics.py.

You won't see a change yet. The next chunk adds the report display that surfaces these numbers.

Print a formatted summary report

Raw numbers in a dictionary are hard to read. The print_report function formats everything into a structured report with sections for performance, judge scores, a per-ticket table, and a disagreements highlight.

  • Add the print_report function below compute_metrics in metrics.py:
def print_report(classification_results, evaluations, metrics):
    print("\n" + "=" * 60)
    print("METRICS SUMMARY REPORT")
    print("=" * 60)

    # Classification performance section
    print(f"\n--- Classification Performance ---")
    print(f"  Accuracy:           {metrics['correct']}/{metrics['total']} = {metrics['accuracy']:.0%}")
    print(f"  Avg Confidence:     {metrics['avg_confidence']:.2f}")

    # Judge evaluation scores section
    print(f"\n--- Judge Evaluation Scores (out of 5) ---")
    print(f"  Avg Correctness:         {metrics['avg_correctness']:.2f}")
    print(f"  Avg Reasoning Quality:   {metrics['avg_reasoning']:.2f}")
    print(f"  Avg Routing Appropriate: {metrics['avg_appropriateness']:.2f}")

    # Agreement between judge and classifier
    print(f"\n--- Agreement ---")
    print(f"  Judge Agreement Rate: {metrics['judge_agrees']}/{metrics['total']} = {metrics['agreement_rate']:.0%}")

    # Per-ticket breakdown table
    print(f"\n--- Per-Ticket Breakdown ---")
    print(f"  {'ID':<4} {'Predicted':<12} {'Actual':<12} {'Correct':<9} {'Confidence':<11} {'Judge Score'}")
    print(f"  {'-'*4} {'-'*12} {'-'*12} {'-'*9} {'-'*11} {'-'*11}")

    for r, e in zip(classification_results, evaluations):
        status = "Yes" if r["is_correct"] else "No"
        print(f"  {r['ticket_id']:<4} {r['predicted']:<12} {r['ground_truth']:<12} {status:<9} {r['confidence']:<11.2f} {e['correctness_score']}/5")

    # Highlight disagreements where the judge suggested a different category
    disagreements = [
        (r, e) for r, e in zip(classification_results, evaluations)
        if e["suggested_category"].upper() != r["predicted"].upper()
    ]

    if disagreements:
        print(f"\n--- Disagreements (Judge vs Classifier) ---")
        for r, e in disagreements:
            print(f"  Ticket #{r['ticket_id']}: Classifier said '{r['predicted']}', Judge suggested '{e['suggested_category']}'")
            print(f"    Reason: {e['explanation'][:100]}...")
    else:
        print(f"\n  No disagreements! Judge agreed with all classifications.")

    print(f"\n{'=' * 60}")
    print("END OF REPORT")
    print(f"{'=' * 60}")

What does this report show?

The report has four sections. The first shows raw classifier performance (accuracy and confidence). The second shows the judge's average scores across all three rubric dimensions.

The per-ticket breakdown table lets you scan for patterns. The disagreements section at the end highlights specific tickets where the judge would have routed differently. These are the most interesting cases because they show edge cases in your classification logic.

  • Save metrics.py.

You won't see output yet. The next chunk adds the main block that runs the full pipeline.

  • Add the main execution block at the bottom of metrics.py:
if __name__ == "__main__":
    print("Running full pipeline: Classify -> Evaluate -> Report\n")
    # Run the classifier on all 10 tickets
    classification_results = run_classifier()
    # Run the judge on all classification results
    evaluations = run_evaluator(classification_results)
    # Compute aggregate metrics from both sets of results
    metrics = compute_metrics(classification_results, evaluations)
    # Print the final summary report
    print_report(classification_results, evaluations, metrics)

What does this code do?

This is the orchestration layer. It calls run_classifier() to classify all 10 tickets, passes those results into run_evaluator() for the judge to score them, computes the aggregate metrics, and then prints the full report.

The if __name__ == "__main__": guard means this code only runs when you execute metrics.py directly. It won't trigger if another file imports from this module.

  • Save metrics.py.
  • Run the full pipeline by running this command in your terminal:
python metrics.py

You should see classification results print first, then judge evaluation scores, and finally the METRICS SUMMARY REPORT with a per-ticket breakdown table and any disagreements listed at the bottom.

Why does the pipeline take a moment?

The script makes 20 API calls total: 10 for classification and 10 for evaluation. Each call goes to Groq's servers, so the full run takes about 30-60 seconds depending on your connection and rate limits.

Seeing a ModuleNotFoundError?

If you see an error mentioning classifier or evaluator not found, make sure you're running the command from inside your ticket-router folder where all three files live together.

If you see a rate limit error (429), wait 30 seconds and run the command again. Groq's free tier rate-limits requests rather than charging you.

Why is my metrics.py failing to import classifier or evaluator?

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

Here is the complete metrics.py file for comparison:

from classifier import run_classifier
from evaluator import run_evaluator


def compute_metrics(classification_results, evaluations):
    total = len(classification_results)
    correct = sum(1 for r in classification_results if r["is_correct"])
    accuracy = correct / total

    avg_confidence = sum(r["confidence"] for r in classification_results) / total
    avg_correctness = sum(e["correctness_score"] for e in evaluations) / total
    avg_reasoning = sum(e["reasoning_quality"] for e in evaluations) / total
    avg_appropriateness = sum(e["routing_appropriateness"] for e in evaluations) / total

    judge_agrees = sum(
        1 for r, e in zip(classification_results, evaluations)
        if e["suggested_category"].upper() == r["predicted"].upper()
    )
    agreement_rate = judge_agrees / total

    return {
        "accuracy": accuracy,
        "correct": correct,
        "total": total,
        "avg_confidence": avg_confidence,
        "avg_correctness": avg_correctness,
        "avg_reasoning": avg_reasoning,
        "avg_appropriateness": avg_appropriateness,
        "agreement_rate": agreement_rate,
        "judge_agrees": judge_agrees,
    }


def print_report(classification_results, evaluations, metrics):
    print("\n" + "=" * 60)
    print("METRICS SUMMARY REPORT")
    print("=" * 60)

    print(f"\n--- Classification Performance ---")
    print(f"  Accuracy:           {metrics['correct']}/{metrics['total']} = {metrics['accuracy']:.0%}")
    print(f"  Avg Confidence:     {metrics['avg_confidence']:.2f}")

    print(f"\n--- Judge Evaluation Scores (out of 5) ---")
    print(f"  Avg Correctness:         {metrics['avg_correctness']:.2f}")
    print(f"  Avg Reasoning Quality:   {metrics['avg_reasoning']:.2f}")
    print(f"  Avg Routing Appropriate: {metrics['avg_appropriateness']:.2f}")

    print(f"\n--- Agreement ---")
    print(f"  Judge Agreement Rate: {metrics['judge_agrees']}/{metrics['total']} = {metrics['agreement_rate']:.0%}")

    print(f"\n--- Per-Ticket Breakdown ---")
    print(f"  {'ID':<4} {'Predicted':<12} {'Actual':<12} {'Correct':<9} {'Confidence':<11} {'Judge Score'}")
    print(f"  {'-'*4} {'-'*12} {'-'*12} {'-'*9} {'-'*11} {'-'*11}")

    for r, e in zip(classification_results, evaluations):
        status = "Yes" if r["is_correct"] else "No"
        print(f"  {r['ticket_id']:<4} {r['predicted']:<12} {r['ground_truth']:<12} {status:<9} {r['confidence']:<11.2f} {e['correctness_score']}/5")

    disagreements = [
        (r, e) for r, e in zip(classification_results, evaluations)
        if e["suggested_category"].upper() != r["predicted"].upper()
    ]

    if disagreements:
        print(f"\n--- Disagreements (Judge vs Classifier) ---")
        for r, e in disagreements:
            print(f"  Ticket #{r['ticket_id']}: Classifier said '{r['predicted']}', Judge suggested '{e['suggested_category']}'")
            print(f"    Reason: {e['explanation'][:100]}...")
    else:
        print(f"\n  No disagreements! Judge agreed with all classifications.")

    print(f"\n{'=' * 60}")
    print("END OF REPORT")
    print(f"{'=' * 60}")


if __name__ == "__main__":
    print("Running full pipeline: Classify -> Evaluate -> Report\n")
    classification_results = run_classifier()
    evaluations = run_evaluator(classification_results)
    metrics = compute_metrics(classification_results, evaluations)
    print_report(classification_results, evaluations, metrics)

Your pipeline is complete. You've built a system that classifies tickets, evaluates its own quality, and reports metrics. Next up, the Secret Mission challenges you to experiment with different prompts and measure which one performs better.

Secret mission

Run a Prompt Experiment

Your classifier works, but is your system prompt actually the best one? In this secret mission, you'll run an A/B test comparing two different prompts and measure which one produces better classifications. This is exactly how ML teams optimize LLM features in production.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally with a free-tier API key, so there are no ongoing costs. Decide whether to keep your resources for future iteration or delete them entirely.

Resources you used:

  • The ticket-router folder on your Desktop (contains test_connection.py, classifier.py, evaluator.py, metrics.py, experiment.py).
  • Groq API key (free tier, no charges).
  • The groq Python package installed via pip.
  • The GROQ_API_KEY environment variable set in your terminal session.

Keep everything running

No action needed. Choose this if you plan to keep experimenting with your ticket router or building on it for your portfolio.

  • Your code stays in the ticket-router folder, ready to run any time.
  • Your Groq API key remains active on the free tier. You will never be charged. Groq returns a 429 rate-limit error if you exceed the free plan limits.
  • The groq package stays installed in your Python environment.

Pause - I'll come back to this later

Since everything is local and the API key is free, there is nothing to pause. Your project is already in a zero-cost idle state.

  • If you set GROQ_API_KEY only for the current terminal session, it will disappear when you close the terminal. You will need to set it again next time you run the scripts.
  • Your files and installed packages remain untouched.

Delete - I don't want to use this again

Remove all project files, uninstall the package, and clean up your API key.

  • Delete the project folder. On macOS/Linux, run:
cd ~/Desktop
rm -rf ticket-router

On Windows PowerShell, run:

cd ~\Desktop
Remove-Item -Recurse -Force ticket-router
  • Uninstall the Groq Python package by running:
pip uninstall groq
  • Delete your Groq API key. Go to https://console.groq.com, navigate to API Keys, and delete the key you created for this project.
  • Clear the environment variable from your current session. On macOS/Linux, run unset GROQ_API_KEY. On Windows PowerShell, run Remove-Item Env:GROQ_API_KEY.
  • If you added the environment variable to your system environment variables (Windows) or shell profile (macOS/Linux), remove it from there as well.

Nice Work!

Nice Work!

You just built a complete intelligent ticket routing system with its own evaluation pipeline. This is the exact dual skill set that production ML teams need: building features AND measuring their quality.

You've learned how to:

  • Build an LLM-powered ticket classifier using Groq's structured outputs API to guarantee reliable JSON responses.
  • Implement an LLM-as-a-judge evaluation pipeline that scores classification quality across multiple dimensions.
  • Compute product metrics (accuracy, confidence, agreement rate) that demonstrate how to measure ML feature quality in production.
  • Secret Mission: Ran a prompt experiment comparing two system prompts and measured which performed better, demonstrating A/B testing skills for ML features.

Ready to quiz yourself?