Build a Local AI Ticket Evaluator

Build and evaluate a local AI tool that classifies support tickets as JSON.

Introduction

30 Second Summary

A support queue can fill up faster than a team can sort it. Important requests get delayed when every message needs a person to decide where it belongs.

In this project, you will build a command-line support ticket triage tool powered by a local language model. You will run a labeled evaluation suite to measure how accurately the tool classifies real support requests.

What You'll Build

Picture pasting a customer message into your terminal and receiving a clean classification that another program can safely consume.

By the end of this project, you'll have:

  • A local ticket triage workflow that turns a pasted customer message into a category, priority, and short summary while keeping inference on your machine.
  • A validated output contract using JSON Schema that gives every response dependable category, priority, and summary fields.
  • A repeatable evaluation suite that prints PASS or FAIL for each labeled ticket before reporting a final Category accuracy score.
  • Secret Mission: Add a needs_human decision and test mixed-intent tickets that should be escalated.

Are there any prerequisites?

Use Windows 10 22H2 or newer with Python plus Visual Studio Code already installed.

You also need internet access for the initial downloads plus at least 4 GB for Ollama and another 398 MB for qwen2.5:0.5b.

Before We Start

A dependable local classifier starts with a clear definition of success. This checkpoint locks in the data your classifier will produce and why you will validate its output through evaluation.

Set Up Local AI Inference

A local classifier needs a verified model endpoint before Python can request a prediction. Without that endpoint, the project has nowhere to send a support ticket.

Python 3.14.7 gives the project a known runtime. An empty local-ai-triage folder keeps the files you create later in one place.

Ollama v0.34.4 provides local model inference through a Windows service. The qwen2.5:0.5b model gives you a visible AI response before you write any project code.

In this step, get ready to:
  • Prepare the project with Python 3.14.7.
  • Confirm the Ollama local service.
  • Run the qwen2.5:0.5b model locally.
Verify Python and prepare the folder

The version check confirms which Python runtime the python command currently uses. The project standard is 3.14.7 on 64-bit Windows.

  • Press the Windows key to open Windows search.
  • Enter PowerShell in the search field.
  • Press Enter to open PowerShell.
  • Check your installed Python version by running this command:
python --version

What does this command do?

The python --version command prints the Python version used by the python command. It exits immediately after showing the result.

Your displayed version determines whether the existing runtime is ready or needs an update.

✔️ I see Python 3.14.7

Good start. Your Python runtime matches the project standard.

ⓧ I see an older Python version

The installed runtime needs an update before the project begins. Windows may ask for permission when the installer starts.

  • Open the official Python 3.14.7 release page.
  • Download the recommended 64-bit Windows installer.
  • Run the downloaded installer.
  • Complete the installer using its recommended settings.
  • Close the PowerShell window after installation.
  • Press the Windows key to open Windows search.
  • Enter PowerShell in the search field.
  • Press Enter to open a fresh PowerShell window.

Why reopen PowerShell?

A fresh terminal reads the command locations registered by the installer. This gives the updated python command a clean check.

  • Confirm the updated runtime by running this command:
python --version

What should this show?

The terminal should report Python 3.14.7. That result confirms the update reached your current PowerShell session.

Still seeing the older version?

Close every open PowerShell window before checking again. An older terminal can retain the previous command location.

Confirm that you ran the recommended 64-bit installer from the Python 3.14.7 release page. Help me diagnose why the python command still shows an older version.

ⓧ Command not found

Windows cannot currently find the Python executable. Installing the recommended 64-bit release registers the runtime for this project.

  • Open the official Python 3.14.7 release page.
  • Download the recommended 64-bit Windows installer.
  • Run the downloaded installer.
  • Complete the installer using its recommended settings.
  • Close the PowerShell window after installation.
  • Press the Windows key to open Windows search.
  • Enter PowerShell in the search field.
  • Press Enter to open a fresh PowerShell window.

Why reopen PowerShell?

A fresh terminal loads the command location registered during installation. The next check confirms that Windows can now find Python.

  • Verify the installation by running this command:
python --version

What should this show?

The terminal should report Python 3.14.7. The command is now available to the project.

Python still unavailable?

Confirm that the installer completed before reopening PowerShell. Restart Windows if the command location still has not refreshed.

Use the recommended 64-bit installer from the official release page. Help me make Python available in PowerShell.

The runtime is ready. Next, the project needs a named folder that Visual Studio Code can treat as one workspace.

  • Move to your Desktop by running this command:
cd ~/Desktop

What does this command do?

The cd command changes PowerShell's current location. The ~/Desktop path points to the Desktop inside your Windows user folder.

  • Create the local-ai-triage folder by running these commands:
mkdir local-ai-triage
cd local-ai-triage

What do these commands do?

  • The first command creates the local-ai-triage folder on your Desktop.
  • The second command moves PowerShell into that folder.

PowerShell shows the new directory details after creating it. Your current prompt now ends inside local-ai-triage.

  • Open the current folder in Visual Studio Code by running this command:
code .

What does this command do?

The code . command opens the current PowerShell folder as a Visual Studio Code workspace. The dot represents the current folder.

You should see local-ai-triage in the Explorer view. The empty workspace is ready for the Python files you create in the next step.

Folder not opening in Visual Studio Code?

Confirm that the PowerShell prompt is inside local-ai-triage before running the editor command. A different location opens the wrong folder.

If Windows cannot find the editor command, use File followed by Open Folder... in Visual Studio Code. Select local-ai-triage from your Desktop.

Help me open my local-ai-triage folder in Visual Studio Code.

Install Ollama and verify the local service

Ollama runs a local service that later accepts tickets through an HTTP API. The first check proves that both the command-line tool and the service are reachable.

This official install command downloads Ollama from its Windows installer source. The installation does not require Administrator rights.

  • Switch back to the PowerShell window from earlier.
  • Install Ollama by running this command:
irm https://ollama.com/install.ps1 | iex

What does this command do?

The command retrieves Ollama's official Windows installation script. PowerShell then runs that script to install the local application.

The installer sets up Ollama v0.34.4 as a Windows application. Reopening PowerShell gives the new ollama command a fresh session.

  • Close the current PowerShell window.
  • Press the Windows key to open Windows search.
  • Enter PowerShell in the search field.
  • Press Enter to open a fresh PowerShell window.
  • Confirm that the Ollama CLI can reach the local service by running this command:
ollama ls

What does this command prove?

The ollama ls command asks the local service for its available models. A returned model list proves that the CLI can communicate with the service.

The list can be empty at this point because the model download comes next. Match your result to one of these paths.

✔️ I see the local model list

That is the service handshake complete. Ollama is installed locally and accepting CLI requests.

ⓧ I already have an older version

Use the current official installer to standardize on Ollama v0.34.4.

  • Update Ollama by running the official install command:
irm https://ollama.com/install.ps1 | iex

What does this command do?

The command retrieves the current official Windows installer script. The script installs the current stable Ollama release.

  • Close PowerShell after the installer finishes.
  • Press the Windows key to open Windows search.
  • Enter PowerShell in the search field.
  • Press Enter to open a fresh PowerShell window.
  • Verify the updated local service by running this command:
ollama ls

What should this show?

The terminal should return the local model list without reporting a missing command. That result confirms the updated CLI can reach Ollama.

ⓧ Command not found

The fresh PowerShell session cannot find Ollama. Running the official installer again restores the missing installation.

  • Install Ollama again by running the official command:
irm https://ollama.com/install.ps1 | iex

What does this command do?

The command retrieves the official Windows installation script. PowerShell runs the script to make Ollama available locally.

  • Close PowerShell after the installation finishes.
  • Press the Windows key to open Windows search.
  • Enter PowerShell in the search field.
  • Press Enter to open a fresh PowerShell window.
  • Retry the service check by running this command:
ollama ls

What should this show?

The terminal should return the local model list. This confirms that PowerShell can now find Ollama.

Ollama still unavailable?

Confirm that the installation finished before you opened the new PowerShell window. Restart Windows if the command location still has not refreshed.

Check that Ollama is running from the Windows taskbar before retrying the list command. Help me restore the Ollama command and local service.

Download and test the local model

The model artifact contains the weights Ollama uses to generate a response. This selected artifact is 398 MB, which keeps the first local test small.

The first launch downloads the model before opening an interactive session. Set aside a few minutes while the progress display moves across the terminal.

  • Download the model and open its interactive session by running this command:
ollama run qwen2.5:0.5b

What does this command do?

The command downloads qwen2.5:0.5b when it is missing locally. Ollama then opens a terminal session where the model can receive prompts.

Before you send the verification prompt, do you expect the model to follow the one-word response instruction?

  • Test the active model session by entering this prompt:
Reply with READY only

What does this prompt check?

The prompt asks for one constrained response. A matching answer proves that the downloaded model can receive text and generate a local result.

You should see READY in the terminal response. That single word confirms your local model endpoint is ready for the classifier.

Model not responding?

Check that the 398 MB download completed before entering the prompt. An incomplete download cannot start the model session.

Confirm that Ollama is still running from the Windows taskbar. Retry the model command after the local service is available.

Help me diagnose why qwen2.5:0.5b is not responding locally.

Your local AI runtime is now taking prompts without an API account. Next, you will connect Python to that model and expose the weakness of a free-form response.

Build the Naive Ticket Triage

Your local Ollama model has already answered a direct prompt. Now you can connect it to a support-ticket workflow.

Your Python script will send one ticket through Ollama's local HTTP API.

A readable response can change shape between runs. An evaluator has no stable fields to score.

In this step, get ready to:
  • Create a Python client for the local chat endpoint.
  • Send one ticket with a one-sentence triage prompt.
  • Run the command-line tool and inspect its free-form response.
Connect the Python client to Ollama

A request to the chat endpoint carries a JSON payload. The payload identifies the local model.

It also carries the user's message. Ollama returns the model's answer in the response.

  • In the Visual Studio Code file sidebar, select the existing local-ai-triage folder.
  • Use the file creation control in the file sidebar.
  • Enter triage.py as the file name.
  • Add the imports and local model settings to triage.py by pasting this code:
import json
import urllib.request

API_URL = "http://localhost:11434/api/chat"
MODEL = "qwen2.5:0.5b"

What does this code do?

  • The json module converts Python data into the request body that Ollama accepts.
  • The urllib.request module sends the request without an extra package.
  • API_URL points to Ollama's local chat endpoint.
  • MODEL selects the downloaded qwen2.5:0.5b model.
  • Save triage.py.
  • Confirm triage.py appears directly inside local-ai-triage in the file sidebar.

Can't See the Python File?

  • Confirm that you created the file inside the existing local-ai-triage folder.
  • Check that the file name ends with .py.

Help me check why triage.py is missing from my local-ai-triage folder.

The classify_ticket(ticket) function builds the request around each ticket. A non-streaming request returns one response for the function to decode.

  • In triage.py, place your cursor on a blank line beneath MODEL = "qwen2.5:0.5b".
  • Add classify_ticket(ticket) by pasting this code:
def classify_ticket(ticket):
    payload = {
        "model": MODEL,
        "messages": [
            {
                "role": "user",
                "content": (
                    "Describe this support ticket's category and priority "
                    f"in one sentence: {ticket}"
                ),
            }
        ],
        "stream": False,
    }
    request = urllib.request.Request(
        API_URL,
        data=json.dumps(payload).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(request, timeout=120) as response:
        api_response = json.loads(response.read().decode("utf-8"))
    return api_response["message"]["content"]

How Does the Request Work?

  • The payload combines the selected model with the ticket prompt.
  • The messages list gives the ticket to the model with the user role.
  • stream is set to False so the API returns one complete response.
  • The request sends the encoded payload with POST.
  • The final line extracts the assistant text from message.content.
  • Save triage.py.
  • Confirm the function ends with return api_response["message"]["content"].

Does the Function Look Misaligned?

  • Check that every line inside classify_ticket(ticket) is indented with four spaces.
  • Confirm the closing brackets align with the structures they finish.

Help me find an indentation or bracket problem in classify_ticket.

Add the ticket prompt and inspect the response

A command-line entry point collects the customer's message. It passes that message to classify_ticket(ticket).

  • In triage.py, place your cursor on a blank line beneath the return statement.
  • Add the interactive entry point by pasting this code:
def main():
    print("Local AI Support Ticket Triage")
    ticket = input("Paste a support ticket: ")
    print(classify_ticket(ticket))


if __name__ == "__main__":
    main()

What Does the Entry Point Do?

  • main() prints the tool's title.
  • input() waits for the support ticket.
  • classify_ticket(ticket) sends the entered ticket to the local model.
  • The final guard calls main() when you run triage.py directly.
  • Save triage.py.
  • Confirm the last line contains main().

Is the Entry Point Inside the Function?

  • Check that if __name__ == "__main__": starts at the left edge of the file.
  • Confirm only the final main() line is indented beneath the guard.

Help me check the main function and entry-point indentation.

✔️ Awesome, I've got everything!

  • Save triage.py before you run it.

ⓧ I'd like to double check the full code

import json
import urllib.request

API_URL = "http://localhost:11434/api/chat"
MODEL = "qwen2.5:0.5b"


def classify_ticket(ticket):
    payload = {
        "model": MODEL,
        "messages": [
            {
                "role": "user",
                "content": (
                    "Describe this support ticket's category and priority "
                    f"in one sentence: {ticket}"
                ),
            }
        ],
        "stream": False,
    }
    request = urllib.request.Request(
        API_URL,
        data=json.dumps(payload).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(request, timeout=120) as response:
        api_response = json.loads(response.read().decode("utf-8"))
    return api_response["message"]["content"]


def main():
    print("Local AI Support Ticket Triage")
    ticket = input("Paste a support ticket: ")
    print(classify_ticket(ticket))


if __name__ == "__main__":
    main()

How to Read This Reference

The complete file contains one local API client function. It also contains one command-line entry point.

  • Open a PowerShell terminal from Visual Studio Code's terminal controls.

Before you run the script, do you think the response will use consistent fields or natural phrasing?

  • Start the ticket triage tool by running this command:
python triage.py

What Does This Run Do?

This command starts triage.py. The script prints its title before waiting at the ticket prompt.

  • Enter I was charged twice for the same subscription at the ticket prompt.
  • Press Enter to send the ticket.

You'll see a readable sentence describing the ticket as a billing issue with a priority. Your Python script has now completed a local round trip through Ollama.

The exact wording can vary between runs. An evaluator cannot reliably extract a category from this prose.

No Model Response?

  • Confirm Ollama is still running on Windows.
  • Check that the terminal is inside the existing local-ai-triage folder.
  • Confirm the command uses the saved triage.py file.

Help me diagnose why triage.py cannot get a response from my local Ollama model.

Your local classifier can now interpret a support ticket. Next, you will turn its changing prose into dependable application data.

Make Every Response Machine-Readable

Your classifier already sends tickets to Ollama. The readable answer from the previous step cannot be scored reliably because its fields are not stable.

Now you will give the model a JSON Schema contract. Python will decode each response before checking its fields against the values your application supports.

In this step, get ready to:
  • Define the response schema and classification rules.
  • Decode the model response into stable application data.
  • Print only validated ticket classifications.
Constrain the model response

A schema names the fields that every response must contain. The system prompt supplies the classification rules that determine each field's value.

  • In triage.py from earlier, select everything from import json through the line that defines MODEL.
  • Replace the selected section with the response contract by pasting this code:
import json
import urllib.error
import urllib.request

API_URL = "http://localhost:11434/api/chat"
MODEL = "qwen2.5:0.5b"
CATEGORIES = {"billing", "account", "technical", "shipping"}
PRIORITIES = {"low", "medium", "high"}

OUTPUT_SCHEMA = {
    "type": "object",
    "properties": {
        "category": {"type": "string"},
        "priority": {"type": "string"},
        "summary": {"type": "string"},
    },
    "required": ["category", "priority", "summary"],
}

SYSTEM_PROMPT = """You triage customer support tickets.
Choose exactly one category: billing, account, technical, or shipping.
Choose priority high when money is at risk, access is blocked, or a delivery is missing.
Choose priority medium for an important problem that does not fully block the customer.
Choose priority low for general questions or minor requests.
Keep the summary under 20 words."""

What does the response contract define?

  • The CATEGORIES set contains the four supported routing labels.
  • The PRIORITIES set contains the three supported urgency labels.
  • The OUTPUT_SCHEMA object requires category, priority, and summary as strings.
  • The SYSTEM_PROMPT constant gives the model one shared set of routing rules.
  • Save triage.py.
  • Confirm the editor shows all three required fields inside OUTPUT_SCHEMA.

Seeing broken syntax highlighting?

Check that SYSTEM_PROMPT starts and ends with three double quotes. Confirm that every schema brace has a matching closing brace.

Compare the commas after each schema property. Help me find the syntax problem in my response contract.

The contract only affects inference after the request sends it to the model. The next edit adds the schema plus deterministic settings to the existing payload.

  • Scroll to the payload dictionary inside classify_ticket().
  • Select the current payload dictionary.
  • Replace the selected dictionary with this schema-constrained request:
    ticket = ticket.strip()
    if not ticket:
        raise ValueError("Ticket cannot be empty.")

    payload = {
        "model": MODEL,
        "messages": [
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": ticket},
        ],
        "stream": False,
        "format": OUTPUT_SCHEMA,
        "options": {"temperature": 0},
    }

What does the constrained request do?

  • The first check removes surrounding whitespace from the ticket. It rejects an empty message before making a model request.
  • The system message provides the classification rules.
  • The format field asks Ollama to follow OUTPUT_SCHEMA.
  • The temperature value reduces variation between repeated classifications.
  • Save triage.py.

Before you run this, predict whether the output will still be prose or take the schema's three-field shape.

  • Switch back to the integrated PowerShell terminal from earlier.
  • Start the updated classifier by running this command:
python triage.py

What does this check prove?

This command starts the interactive classifier with the schema-aware payload. The current printing logic still displays the model's inner response directly.

  • Enter I was charged twice for the same subscription at the ticket prompt.

You will see JSON-shaped text containing category, priority, and summary. The model now follows a stable response shape.

Still seeing free-form prose?

Confirm that format sits inside the payload dictionary. Check that its value is OUTPUT_SCHEMA.

Make sure the saved file is the triage.py file inside local-ai-triage. Help me diagnose why Ollama is still returning prose.

Decode and validate the model response

The structured classification still arrives as text inside the outer HTTP API response. Your code must decode both layers before it can validate the resulting Python object.

  • Inside classify_ticket(), select the current request code through the line that assigns api_response.
  • Replace the selected section with this protected request and decoding logic:
    request = urllib.request.Request(
        API_URL,
        data=json.dumps(payload).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST",
    )

    try:
        with urllib.request.urlopen(request, timeout=120) as response:
            raw_response = response.read().decode("utf-8")
    except urllib.error.URLError as exc:
        raise RuntimeError(
            "Could not reach Ollama at http://localhost:11434. Make sure Ollama is running."
        ) from exc

    try:
        api_response = json.loads(raw_response)
        result = json.loads(api_response["message"]["content"])
    except (KeyError, json.JSONDecodeError) as exc:
        raise RuntimeError("Ollama returned an unexpected response.") from exc

How does the response become data?

  • The first try block sends the request. It converts a connection failure into a clear runtime error.
  • The first json.loads() call decodes the outer Ollama response.
  • The second json.loads() call decodes the structured text stored in message.content.
  • The second try block catches a missing response path or malformed JSON.
  • Save triage.py.
  • Confirm the decoding section assigns the inner model data to result.

Can't decode the response?

Check that the second decoding call reads api_response["message"]["content"]. The inner classification lives at that path.

Confirm that both exception blocks end with from exc. Help me check the two response-decoding layers.

Decoding proves the response has a JSON shape. Validation now checks that the object contains every required field plus only supported routing labels.

  • Find the old return api_response["message"]["content"] line below the decoding section.
  • Replace that line with these validation checks:
    required_fields = {"category", "priority", "summary"}
    if not required_fields.issubset(result):
        raise ValueError("The model response is missing a required field.")
    if result["category"] not in CATEGORIES:
        raise ValueError("The model returned an unsupported category.")
    if result["priority"] not in PRIORITIES:
        raise ValueError("The model returned an unsupported priority.")
    if not isinstance(result["summary"], str):
        raise ValueError("The model summary must be text.")

    return result

What does validation protect?

  • The required-field check rejects an incomplete response.
  • The category check rejects labels outside CATEGORIES.
  • The priority check rejects labels outside PRIORITIES.
  • The summary check ensures downstream code receives text.
  • Save triage.py.

Before you run the classifier, predict whether the decoded result will appear as a Python object or a free-form sentence.

  • Run the classifier again with this command:
python triage.py

What does this run exercise?

This run exercises the protected request plus both decoding layers. It also checks the three fields before returning the classification.

  • Enter I was charged twice for the same subscription at the ticket prompt.

You will see a Python dictionary with the three stable fields. The absence of a validation error confirms that the labels and summary passed your checks.

Seeing a validation error?

Compare the allowed labels in CATEGORIES and PRIORITIES with the exact lowercase values in SYSTEM_PROMPT.

Check that required_fields contains all three schema property names. Help me diagnose this ticket validation error.

Print validated JSON

The command-line entry point still prints Python's dictionary representation. The final edit catches expected failures before formatting successful classifications as indented JSON.

  • Select the current main() function in triage.py.
  • Replace the selected function with this error-handling entry point:
def main():
    print("Local AI Support Ticket Triage")
    ticket = input("Paste a support ticket: ")

    try:
        result = classify_ticket(ticket)
    except (RuntimeError, ValueError) as exc:
        print(f"Error: {exc}")
        return

    print(json.dumps(result, indent=2))

What does the entry point change?

  • The try block receives only the result returned after validation.
  • The except block turns expected model or input failures into readable terminal messages.
  • The final json.dumps() call prints the validated result with indentation.
  • Save triage.py.

Before this final run, predict which three field names will appear on separate lines.

  • Run the finished classifier with this command:
python triage.py

What does the final check prove?

This run sends the ticket through the schema plus Python validation. A successful response reaches the printing line only after every check passes.

  • Enter I was charged twice for the same subscription at the ticket prompt.

That is the reliability loop working. Your terminal now prints indented JSON containing category, priority, and summary.

Not seeing indented JSON?

Confirm the last line inside main() uses json.dumps(result, indent=2). Check that the line sits outside the exception block.

Make sure the final file still includes the if __name__ == "__main__": guard below main(). Help me diagnose why validated JSON is not printing.

✔️ Awesome, I've got everything!

Great. Double-check that you saved triage.py after the final run.

ⓧ I'd like to double check the full code

import json
import urllib.error
import urllib.request

API_URL = "http://localhost:11434/api/chat"
MODEL = "qwen2.5:0.5b"
CATEGORIES = {"billing", "account", "technical", "shipping"}
PRIORITIES = {"low", "medium", "high"}

OUTPUT_SCHEMA = {
    "type": "object",
    "properties": {
        "category": {"type": "string"},
        "priority": {"type": "string"},
        "summary": {"type": "string"},
    },
    "required": ["category", "priority", "summary"],
}

SYSTEM_PROMPT = """You triage customer support tickets.
Choose exactly one category: billing, account, technical, or shipping.
Choose priority high when money is at risk, access is blocked, or a delivery is missing.
Choose priority medium for an important problem that does not fully block the customer.
Choose priority low for general questions or minor requests.
Keep the summary under 20 words."""


def classify_ticket(ticket):
    ticket = ticket.strip()
    if not ticket:
        raise ValueError("Ticket cannot be empty.")

    payload = {
        "model": MODEL,
        "messages": [
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": ticket},
        ],
        "stream": False,
        "format": OUTPUT_SCHEMA,
        "options": {"temperature": 0},
    }

    request = urllib.request.Request(
        API_URL,
        data=json.dumps(payload).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST",
    )

    try:
        with urllib.request.urlopen(request, timeout=120) as response:
            raw_response = response.read().decode("utf-8")
    except urllib.error.URLError as exc:
        raise RuntimeError(
            "Could not reach Ollama at http://localhost:11434. Make sure Ollama is running."
        ) from exc

    try:
        api_response = json.loads(raw_response)
        result = json.loads(api_response["message"]["content"])
    except (KeyError, json.JSONDecodeError) as exc:
        raise RuntimeError("Ollama returned an unexpected response.") from exc

    required_fields = {"category", "priority", "summary"}
    if not required_fields.issubset(result):
        raise ValueError("The model response is missing a required field.")
    if result["category"] not in CATEGORIES:
        raise ValueError("The model returned an unsupported category.")
    if result["priority"] not in PRIORITIES:
        raise ValueError("The model returned an unsupported priority.")
    if not isinstance(result["summary"], str):
        raise ValueError("The model summary must be text.")

    return result


def main():
    print("Local AI Support Ticket Triage")
    ticket = input("Paste a support ticket: ")

    try:
        result = classify_ticket(ticket)
    except (RuntimeError, ValueError) as exc:
        print(f"Error: {exc}")
        return

    print(json.dumps(result, indent=2))


if __name__ == "__main__":
    main()

Your classifier now returns application data that code can inspect safely. Next, you will test it against labeled tickets to measure how often its category predictions are correct.

Measure the Classifier

Your classifier now turns one support ticket into validated application data. The project still needs evidence that its labels hold across different requests.

One convincing response cannot prove model reliability. A labeled evaluation suite compares predictions with known answers across a fixed set of tickets.

In this step, get ready to:
  • Build a labeled dataset across four support categories.
  • Compare predicted categories with expected labels.
  • Calculate the final category accuracy.
Create the labeled baseline

A labeled example pairs a ticket with its expected category. Those expected labels become the benchmark for every model prediction.

  • Select the file-shaped plus icon in the file sidebar of Visual Studio Code for the local-ai-triage folder.
  • Enter evaluate.py as the file name.

You'll see a blank evaluate.py tab inside local-ai-triage.

  • Populate evaluate.py with the labeled examples by pasting this code:
from triage import classify_ticket

TEST_CASES = [
    ("I was charged twice for one subscription renewal.", "billing"),
    ("Please send me a copy of last month's invoice.", "billing"),
    ("I reset my password but still cannot sign in.", "account"),
    ("How do I change the email address on my profile?", "account"),
    ("The Windows app closes whenever I upload a PDF.", "technical"),
    ("The export button does nothing after I click it.", "technical"),
    ("Tracking says delivered, but the package is not here.", "shipping"),
    ("When will order 4821 arrive?", "shipping"),
]

What does this test data do?

  • The import reuses the validated classify_ticket function from triage.py.
  • Each tuple stores one ticket beside its expected category.
  • The eight examples cover billing, account, technical, and shipping requests.
  • Save evaluate.py.
  • Confirm the file sidebar lists evaluate.py beside triage.py.

Can't see the test file?

Check that evaluate.py sits inside the same local-ai-triage folder as triage.py.

Confirm the filename ends with .py. Help me check why evaluate.py cannot import classify_ticket.

Turn predictions into scores

The scoring loop sends every labeled ticket through classify_ticket. It preserves each mismatch so you can investigate the exact ticket behind the score.

  • Leave two blank lines below the closing bracket of TEST_CASES.
  • Add the scoring function below those blank lines by pasting this code:
def main():
    correct = 0

    for ticket, expected in TEST_CASES:
        try:
            result = classify_ticket(ticket)
            predicted = result["category"]
            passed = predicted == expected
            correct += int(passed)
            status = "PASS" if passed else "FAIL"
            print(f"{status}: expected={expected}, predicted={predicted}")
            print(f"  Ticket: {ticket}")
        except (RuntimeError, ValueError) as exc:
            print(f"ERROR: {exc}")
            print(f"  Ticket: {ticket}")

    accuracy = correct / len(TEST_CASES)
    print(f"\nCategory accuracy: {correct}/{len(TEST_CASES)} ({accuracy:.0%})")


if __name__ == "__main__":
    main()

How does the evaluator work?

  • The for ticket, expected loop reads each labeled example.
  • The passed value records whether the predicted category matches the expected category.
  • The correct counter turns each matching Boolean value into one point.
  • The except block prints a failed request without stopping the remaining cases.
  • The accuracy calculation divides the correct predictions by the eight labeled cases.
  • Save evaluate.py.
  • Confirm evaluate.py ends with main() beneath the if __name__ == "__main__": guard.

Seeing editor warnings?

Check that every line inside for, try, and except is indented exactly as shown. Python uses indentation to determine which statements belong together.

Compare the closing brackets and parentheses with the full-file reference below. Help me find an indentation or syntax problem in evaluate.py.

✔️ Awesome, I've got everything!

Your test data and scoring loop are in place. Double-check that you saved evaluate.py.

ⓧ I'd like to double check the full code

from triage import classify_ticket

TEST_CASES = [
    ("I was charged twice for one subscription renewal.", "billing"),
    ("Please send me a copy of last month's invoice.", "billing"),
    ("I reset my password but still cannot sign in.", "account"),
    ("How do I change the email address on my profile?", "account"),
    ("The Windows app closes whenever I upload a PDF.", "technical"),
    ("The export button does nothing after I click it.", "technical"),
    ("Tracking says delivered, but the package is not here.", "shipping"),
    ("When will order 4821 arrive?", "shipping"),
]


def main():
    correct = 0

    for ticket, expected in TEST_CASES:
        try:
            result = classify_ticket(ticket)
            predicted = result["category"]
            passed = predicted == expected
            correct += int(passed)
            status = "PASS" if passed else "FAIL"
            print(f"{status}: expected={expected}, predicted={predicted}")
            print(f"  Ticket: {ticket}")
        except (RuntimeError, ValueError) as exc:
            print(f"ERROR: {exc}")
            print(f"  Ticket: {ticket}")

    accuracy = correct / len(TEST_CASES)
    print(f"\nCategory accuracy: {correct}/{len(TEST_CASES)} ({accuracy:.0%})")


if __name__ == "__main__":
    main()
Read the evaluation evidence

Category accuracy compresses eight comparisons into one baseline. The per-ticket lines keep the evidence visible when the total score is lower than expected.

Before you run the suite, predict whether all eight category labels will match.

  • Run the labeled evaluation in the PowerShell terminal from earlier by running this command:
python evaluate.py

How do I read the report?

  • Each PASS means the predicted category matches the expected label.
  • Each FAIL displays the expected label beside the model's prediction.
  • The ticket line identifies the input behind that result.
  • The final Category accuracy line reports the correct count and percentage.

You'll see one PASS or FAIL result for each of the eight tickets. The final line starts with Category accuracy.

Evaluation not completing?

If every case prints ERROR, confirm Ollama remains running on Windows. The evaluator needs the local service for each ticket.

If the script cannot load the classifier, confirm evaluate.py remains beside triage.py. Help me diagnose why the evaluation suite is not completing.

  • Scan the report for any FAIL lines.
  • Compare the expected category with the predicted category for each mismatch.
  • Read the ticket printed beneath each mismatch.
  • Review the final Category accuracy score.

Each mismatch gives you a specific example for the next prompt or data improvement. A perfect run still gives you a baseline that future changes must preserve.

That closes the reliability loop. Your classifier now has a repeatable benchmark for measuring model quality.

Secret mission

Add Human Escalation

Add a Boolean escalation decision to every classification. You will test mixed-intent tickets and measure escalation accuracy separately from category accuracy.

Clean Up Your Resources

Clean Up Your Resources

Pick the cleanup option that fits what you want to do next. Local Ollama inference is free, so keeping the service or model creates no ongoing charges.

Resources you used:

  • The local local-ai-triage project folder containing triage.py and evaluate.py.
  • An Ollama installation at version 0.34.4.
  • The local qwen2.5:0.5b model stored through Ollama.

Keep everything running

Keeping everything preserves your working classifier for more prompt experiments or evaluation cases.

  • Leave Ollama installed on Windows.
  • Keep the qwen2.5:0.5b model available for future local inference.
  • Keep the local-ai-triage folder with your classifier and evaluation files.

Pause - I'll come back to this later

Shutting down Ollama frees the memory used by the local service while preserving your model and project files.

  • Quit Ollama from the Windows taskbar.
  • Keep the local-ai-triage folder on your computer.
  • Relaunch Ollama from the Start menu when you return to the project.

Your model remains stored locally, so you can continue without downloading it again.

Delete - I don't want to use this again

Removing the project can feel final. These actions target only the local resources listed above.

  • Close the local-ai-triage folder in Visual Studio Code.
  • Use File Explorer to locate the local-ai-triage folder.
  • Delete the local-ai-triage folder.
  • Confirm the local-ai-triage folder no longer appears in File Explorer.
  • Open Add or remove programs in Windows Settings.
  • Find Ollama in the installed apps list.
  • Uninstall Ollama from that page.
  • Confirm Ollama no longer appears in Add or remove programs.
  • Delete the custom model directory separately if you changed OLLAMA_MODELS.

Your project code and local AI runtime are now removed from Windows.

Nice Work!

Nice Work!

You did it! Your Python ticket triage tool now uses Ollama for local model inference. Its evaluator measures category accuracy plus escalation accuracy.

You've learned how to:

  • Send support tickets from a Python CLI to a local model through an HTTP API.
  • Turn model responses into schema-constrained JSON. Validate required fields plus allowed labels before returning data.
  • Build a labeled evaluation suite that prints per-ticket results plus overall category accuracy.
  • Secret Mission: Add a measurable human-escalation path for mixed-intent tickets. Score escalation independently from category accuracy.

Ready to quiz yourself?