Build an AI Message Sorter with Python

Build a Python CLI that sorts support messages with a local AI model.

Introduction

30 Second Summary

Customer messages rarely arrive in neat categories. A short note can hide a billing problem, a locked account, or an urgent software issue.

In this project, you will build a Python command-line assistant that sorts support messages with a local AI model. A deliberate rule-based failure reveals why the AI replacement needs a measured evaluation.

What You'll Build

You will watch a customer message become a readable triage result in your terminal before the app scores its decisions against labeled examples.

By the end of this project, you'll have:

  • Readable triage results that turn each support message into a category, priority, summary, and recommended next action.
  • Local AI sorting powered by Ollama that runs on your Mac without an API key.
  • Measured quality from four labeled cases that report PASS or CHECK before calculating a numeric score.
  • Secret Mission: Add confidence reporting to route uncertain predictions to human review.

Are there any prerequisites?

You'll need a Mac running macOS Sonoma 14 or newer with enough free storage for the 1.2GB model.

Setup requires an internet connection. No recent coding practice or API key is required.

Before We Start

AI output needs measurement before you can judge its reliability. Before the hands-on work, commit to building a small local AI support-message sorter that turns customer messages into useful triage details.

Set Up Your Beginner Python Workspace

Clean tools keep installation problems from hiding the Python ideas you came here to practise. A version mismatch can make correct code look broken.

First, you'll prove that macOS meets the local model's requirement. You'll also confirm a supported Python version.

Visual Studio Code will hold your project workspace. Ollama will run the AI model on your Mac without an API key.

In this step, get ready to:
  • Verify the required macOS version plus Python version.
  • Prepare Visual Studio Code plus Ollama.
  • Create an isolated workspace with pinned dependencies.
Verify macOS and Python

Ollama requires macOS Sonoma 14 or newer. Pydantic 2.14.3 requires Python 3.10 or newer.

  • Click the Apple menu in the upper-left corner of your screen.
  • Select About This Mac.
  • Read the displayed macOS version number.

Choose the tab that matches the version shown on your Mac.

✔️ My Mac meets the requirement

Your Mac is ready if it shows macOS Sonoma 14 or a newer major version. That clears the operating-system requirement for local Ollama models.

ⓧ My macOS version is older

A macOS upgrade can restart your Mac. Save any open work before beginning the upgrade.

  • Open Apple's macOS update guide.
  • Select System Settings from the Apple menu.
  • Select General in the sidebar.
  • Select Software Update.
  • Install a compatible upgrade to macOS Sonoma 14 or newer.
  • Return to About This Mac after the upgrade.

You should now see macOS Sonoma 14 or a newer major version.

The next check asks your current Python executable to report its version. This reveals whether the pinned project dependencies can run.

  • Press Cmd+Space to open Spotlight.

You'll see the Spotlight search field.

  • Type Terminal into Spotlight.
  • Press Enter to open Terminal.
  • Check your installed Python version by running this command:
python3 --version

What does this command do?

  • The python3 command selects the Python 3 executable available on your Mac.
  • The --version option prints its version number before the command exits.

Choose the tab that matches the Terminal output.

✔️ I see Python 3.10 or higher

Your Python runtime supports the pinned dependencies. Keep this Terminal window available for the workspace setup.

ⓧ I see an older version

The existing Python version cannot run Pydantic 2.14.3. Install the current Python release before continuing.

  • Open the official Python macOS downloads page.
  • Download the current macOS installer for Python 3.15.2.
  • Open the downloaded installer.
  • Complete the installation prompts.
  • Close your existing Terminal window.
  • Open a new Terminal window through Spotlight.
  • Confirm the new Python version by running this command:
python3 --version

What does this check confirm?

A new Terminal session reloads the available executable paths. The command should now report Python 3.10 or newer.

ⓧ Command not found

Your Terminal cannot find a Python 3 executable. Installing the current macOS release supplies the executable needed by this project.

  • Open the official Python macOS downloads page.
  • Download the current macOS installer for Python 3.15.2.
  • Open the downloaded installer.
  • Complete the installation prompts.
  • Close your existing Terminal window.
  • Open a new Terminal window through Spotlight.
  • Confirm the installation by running this command:
python3 --version

What does this check confirm?

A version number confirms that your Mac can now locate Python 3. Continue when the reported version is 3.10 or newer.

Python version still too old?

Close every Terminal window after installation. Open a fresh window so macOS reloads the available Python paths.

Still stuck? Help me diagnose my Python version on macOS. You can also share the version output in the NextWork community.

Set up Visual Studio Code and Ollama

A workspace gives every project file one named home. Visual Studio Code uses the opened folder as that workspace.

Ollama serves models directly from your Mac. Local use requires no account sign-in or API key.

Why local Ollama?

A hosted AI service would add credentials plus metered requests. Ollama keeps the model call on your own hardware.

This keeps the project focused on Python plus model behavior. You can test every result without cloud billing.

Choose the Visual Studio Code tab that matches your Mac.

✔️ Visual Studio Code is installed

  • Press Cmd+Space to open Spotlight.

You'll see the Spotlight search field.

  • Type Visual Studio Code into Spotlight.
  • Press Enter to open the editor.

ⓧ Visual Studio Code is not installed

  • Open the official Visual Studio Code macOS setup page.
  • Download Visual Studio Code for macOS.
  • Open the downloaded .dmg file.
  • Drag Visual Studio Code.app into the Applications folder.
  • Press Cmd+Space to open Spotlight.

You'll see the Spotlight search field.

  • Type Visual Studio Code into Spotlight.
  • Press Enter to open the editor.

You should now see a Visual Studio Code window. The project folder will appear there after you create it.

Choose the Ollama tab that matches your Mac.

✔️ Ollama is installed

  • Press Cmd+Space to open Spotlight.

You'll see the Spotlight search field.

  • Type Ollama into Spotlight.
  • Press Enter to start Ollama.

ⓧ Ollama is not installed

The first launch includes an easy-to-miss permission prompt for the command-line link. Allow it so Terminal can use Ollama.

  • Open the official Ollama macOS guide.
  • Download the Ollama macOS disk image.
  • Open the downloaded ollama.dmg file.
  • Drag the Ollama application into the Applications folder.
  • Press Cmd+Space to open Spotlight.

You'll see the Spotlight search field.

  • Type Ollama into Spotlight.
  • Press Enter to start Ollama.
  • Allow the CLI link in /usr/local/bin if Ollama requests permission.

Ollama is now running locally in the background. No signed-in service is required for the model used here.

Having trouble installing the apps?

Confirm that each downloaded disk image opened before dragging its application into the Applications folder. Reopen the application through Spotlight after copying it.

Still stuck? Help me install Visual Studio Code or Ollama on my Mac. You can also share what you see in the NextWork community.

Create the isolated workspace

A virtual environment keeps this project's packages separate from other Python projects. The pinned versions make the same dependency set repeatable.

  • In Visual Studio Code, select View from the top menu.
  • Select Terminal.

You'll see an integrated Terminal panel at the bottom of Visual Studio Code.

  • Move the integrated Terminal to your Desktop by running this command:
cd ~/Desktop

What does this command do?

The cd command changes the Terminal's current location. New project files will now be created on your Desktop.

  • Create the ai-message-sorter folder by running these commands:
mkdir ai-message-sorter
ls

What do these commands do?

  • The first command creates the project folder on your Desktop.
  • The second command lists the Desktop contents so you can confirm the folder exists.

You should see ai-message-sorter in the Terminal output.

  • Select File from the Visual Studio Code menu bar.
  • Select Open Folder....

You'll see the macOS folder picker.

  • Select the ai-message-sorter folder on your Desktop.
  • Select Open.

The Explorer sidebar should now show ai-message-sorter as your workspace.

  • Select Yes, I trust the authors if the Workspace Trust dialog appears.
  • Click the New File control in the Explorer sidebar.

A filename field appears in the Explorer sidebar.

  • Type requirements.txt into the filename field.
  • Press Enter to create the file.
  • Add the pinned dependencies to requirements.txt by pasting this content:
ollama==0.6.3
pydantic==2.14.3

What does this file control?

  • The first line pins the Ollama Python package to version 0.6.3.
  • The second line pins Pydantic to version 2.14.3.
  • Pinned versions keep package behavior consistent while you build the sorter.
  • Save requirements.txt by pressing Cmd+S.
  • Confirm that requirements.txt remains listed in the Explorer sidebar.

✔️ Awesome, I've got everything!

Great. Your dependency file is saved with both required package pins.

ⓧ I'd like to double check the full code

ollama==0.6.3
pydantic==2.14.3
  • Switch back to the Visual Studio Code Terminal from earlier.
  • Move into the workspace by running this command:
cd ai-message-sorter

What does this command do?

The Terminal was left on your Desktop. This command moves it into the ai-message-sorter folder before Python creates project-specific files.

  • Create the isolated Python environment by running this command:
python3 -m venv .venv

What does this command do?

  • Python's venv module creates an isolated environment.
  • The environment files are stored in the .venv folder inside your workspace.

You should see .venv appear in the Visual Studio Code Explorer.

  • Activate the virtual environment by running this command:
source .venv/bin/activate

What does activation change?

Activation makes this Terminal use the Python executable plus packages inside .venv. Your other Python projects stay separate.

You should see (.venv) at the beginning of the Terminal prompt.

Package installation may take a minute while Python downloads the pinned releases. A short pause in the Terminal is normal.

  • Install the project dependencies by running this command:
python3 -m pip install -r requirements.txt

What does this command install?

  • Python runs pip from the active environment.
  • The -r option reads both package pins from requirements.txt.
  • The packages are installed inside .venv.

The installation should finish without an error. Your prompt should still begin with (.venv).

Dependency installation failed?

Confirm that the prompt begins with (.venv). Check that requirements.txt matches the double-check tab exactly.

Still stuck? Help me fix my virtual environment or package installation. You can also share the non-sensitive error output in the NextWork community.

The final check starts the structured-output model on your Mac. The first run downloads 1.2GB of model data, so expect several minutes on a typical connection.

Before you run this, do you expect the local model to ask for an API key before it responds?

  • Download the model plus open its local prompt by running this command:
ollama run Osmosis/Osmosis-Structure-0.6B

What does this command do?

  • Ollama downloads Osmosis/Osmosis-Structure-0.6B if the model is missing.
  • Ollama opens an interactive prompt after the download finishes.
  • The model runs locally without an API key.
  • Enter this short test sentence at the local prompt:
Reply with one short sentence about sorting support messages.

What does this test prove?

The sentence gives the local model a small request with no project code involved. A generated reply proves that Ollama can load the model plus process a prompt on your Mac.

You'll see a model-written response in the Terminal. That's the full local AI path working before you write the sorter.

No model response?

Confirm that the Ollama application is still running. Check that the model download completed before entering the test sentence.

Still stuck? Help me run the local Osmosis model with Ollama. You can also share the non-sensitive Terminal output in the NextWork community.

Your workspace now has verified tools, pinned packages, plus a responding local model. Next, you'll turn that clean setup into a working rule-based message sorter.

Build a Rule-Based Message Sorter

Your local Python workspace is ready. Now you'll turn it into a small program that sorts one customer message at a time.

A keyword rule gives you an early working result while refreshing the building blocks that the AI version reuses. This baseline gives the model something concrete to improve later.

In this step, get ready to:
  • Build a function with keyword lists and a dictionary result.
  • Use conditions to assign a category, priority, and next action.
  • Add an interactive loop and test a duplicate-charge refund message.
Create the sorter foundation

The sort_message function receives one piece of text and returns one result. Its keyword lists make each category's trigger words easy to see.

  • Create message_sorter.py inside the open ai-message-sorter workspace by selecting the new-file icon in Visual Studio Code's file sidebar.
  • Add the classification function to message_sorter.py by pasting this code:
def sort_message(message: str) -> dict:
    message_text = message.lower()
    billing_keywords = ["refund", "charge", "invoice", "payment"]
    technical_keywords = ["error", "broken", "crash", "failed"]
    account_keywords = ["login", "password", "profile", "subscription"]

    if any(keyword in message_text for keyword in billing_keywords):
        category = "billing"
        priority = "medium"
        next_action = "Review the billing details and help with the payment issue."
    elif any(keyword in message_text for keyword in technical_keywords):
        category = "technical"
        priority = "medium"
        next_action = "Collect error details and investigate the technical problem."
    elif any(keyword in message_text for keyword in account_keywords):
        category = "account"
        priority = "medium"
        next_action = "Verify the account and help restore access or update settings."
    else:
        category = "other"
        priority = "low"
        next_action = "Review the request and route it to the right team."

    result = {
        "category": category,
        "priority": priority,
        "next_action": next_action,
    }
    return result

How does the sorter decide?

  • The message_text variable stores a lowercase copy of the customer's message.
  • Each keyword list contains strings associated with one support category.
  • The condition chain uses any to stop at the first matching category.
  • The decision stores its three outputs in variables. The result dictionary bundles those values for the display code.
  • Save message_sorter.py.
  • Confirm message_sorter.py appears beside requirements.txt in the file sidebar.

Does the function look misaligned?

  • Check that each elif lines up with the opening if.
  • Check that the three lines inside each condition have the same indentation.
  • Still stuck? Help me check the indentation and structure of my sort_message function.
Add the interactive loop

The classification function needs a terminal interface so you can send it different messages. A repeating loop keeps the program ready until you choose to stop it.

  • Place your cursor two blank lines below return result in message_sorter.py.
  • Add the display function and interactive loop by pasting this code:
def show_result(result: dict) -> None:
    print("\nTriage result")
    print(f"Category: {result['category']}")
    print(f"Priority: {result['priority']}")
    print(f"Next action: {result['next_action']}\n")


def main() -> None:
    print("Message Sorter")
    print("Enter a customer message, or enter 'quit' to stop.\n")

    while True:
        message = input("Message: ").strip()

        if message.lower() == "quit":
            print("Goodbye!")
            break

        if not message:
            print("Please enter a message.\n")
            continue

        result = sort_message(message)
        show_result(result)


if __name__ == "__main__":
    main()

What does this code do?

  • The show_result function receives the returned dictionary. It prints each field on a labeled line.
  • The main function prints the opening instructions.
  • The while True loop collects one message during each pass.
  • The first guard stops the loop when the message is quit.
  • The second guard sends an empty message back to the prompt.
  • Save message_sorter.py.
  • Start the interactive sorter from the activated terminal by running this command:
python3 message_sorter.py

What does this command do?

This runs the saved message_sorter.py file with Python. The entry point calls main so the terminal can accept customer messages.

  • Enter quit at the Message: prompt.

You'll see the farewell message from your script before the activated terminal prompt returns.

Don't see the message prompt?

  • Confirm the terminal is still inside the ai-message-sorter workspace.
  • Confirm the final two lines of message_sorter.py match the code above.
  • Still stuck? Help me find why message_sorter.py does not show its input prompt.
Test the refund message

A clear test case proves that input reaches the classifier and that the returned dictionary reaches the display function.

Before you run this, predict whether the refund request reaches billing, technical, account, or other.

  • Restart the saved sorter from the activated terminal by running this command:
python3 message_sorter.py

Why run the script again?

This starts a fresh pass through the saved program. The next message tests the complete path from terminal input to the printed triage result.

  • Enter I need a refund for a duplicate charge at the prompt.

You'll see Category: billing followed by Priority: medium.

The next action reads Review the billing details and help with the payment issue.

Did the message reach another category?

  • Check that billing_keywords contains both refund and charge.
  • Check that the billing condition appears before the final else block.
  • Still stuck? Help me debug why my refund message is not classified as billing.
  • Enter quit at the next prompt to return to the activated virtual environment.

That's your first working triage flow. A clear refund request now produces a category, priority, and suggested next action.

Your message_sorter.py file should now match the full reference below.

✔️ Awesome, I've got everything!

  • Confirm message_sorter.py is saved inside the ai-message-sorter workspace.

ⓧ I'd like to double check the full code

def sort_message(message: str) -> dict:
    message_text = message.lower()
    billing_keywords = ["refund", "charge", "invoice", "payment"]
    technical_keywords = ["error", "broken", "crash", "failed"]
    account_keywords = ["login", "password", "profile", "subscription"]

    if any(keyword in message_text for keyword in billing_keywords):
        category = "billing"
        priority = "medium"
        next_action = "Review the billing details and help with the payment issue."
    elif any(keyword in message_text for keyword in technical_keywords):
        category = "technical"
        priority = "medium"
        next_action = "Collect error details and investigate the technical problem."
    elif any(keyword in message_text for keyword in account_keywords):
        category = "account"
        priority = "medium"
        next_action = "Verify the account and help restore access or update settings."
    else:
        category = "other"
        priority = "low"
        next_action = "Review the request and route it to the right team."

    result = {
        "category": category,
        "priority": priority,
        "next_action": next_action,
    }
    return result


def show_result(result: dict) -> None:
    print("\nTriage result")
    print(f"Category: {result['category']}")
    print(f"Priority: {result['priority']}")
    print(f"Next action: {result['next_action']}\n")


def main() -> None:
    print("Message Sorter")
    print("Enter a customer message, or enter 'quit' to stop.\n")

    while True:
        message = input("Message: ").strip()

        if message.lower() == "quit":
            print("Goodbye!")
            break

        if not message:
            print("Please enter a message.\n")
            continue

        result = sort_message(message)
        show_result(result)


if __name__ == "__main__":
    main()

How to use this reference

This reference combines the classifier, output formatter, and interactive loop in their final order. Matching that order keeps the script's entry point at the bottom.

Your keyword sorter handles a clear refund request. Next, you'll test a message whose meaning and urgency stretch beyond these simple word lists.

Make the Simple Sorter Fail

Your Python sorter now handles a straightforward refund request inside the Visual Studio Code workspace from the previous step. That result proves your interactive program can follow a matching rule.

A rule can look reliable after one easy success. This step tests whether literal keywords can recognize category plus urgency from context.

In this step, get ready to:
  • Trace the keyword that classifies the refund message as billing.
  • Capture the complete locked-account result.
  • Record the correct label for the locked-account message.
Trace the successful refund rule

The sorter uses keyword matching to compare a message with fixed lists of words. The refund example contains a direct match from billing_keywords.

Before you run it, which word do you expect the sorter to recognize? Hold that prediction for the result.

  • Switch back to the activated .venv terminal from the previous step.
  • Run the sorter from ai-message-sorter by running this command:
python3 message_sorter.py

What Does This Command Do?

The command asks Python to run message_sorter.py. It starts the interactive loop inside your activated .venv environment.

  • Enter I need a refund for a duplicate charge at the Message: prompt.

You'll see the category billing with medium priority. You'll also see a next action that tells the support team to review the billing details.

  • Switch back to message_sorter.py in Visual Studio Code.
  • Locate billing_keywords inside sort_message().
  • Match the message word refund with the same value in billing_keywords.

Why Did the Rule Work?

The any() condition finds refund inside message_text. That match selects the billing branch before the other conditions run.

  • Return to the activated terminal from earlier.
  • Enter quit at the next Message: prompt.

Good work. You have traced the successful classification back to one literal keyword. The terminal now prints Goodbye!.

Sorter Not Producing the Billing Result?

  • Confirm the terminal is still inside the ai-message-sorter folder.
  • Check that the message contains refund with the same spelling used in billing_keywords.
  • Still stuck? Help me check why my sorter is not producing the billing result.
Test the locked-account message

The second message combines an account problem with a deadline. This run asks the same unchanged rules to classify both clues.

Before you run it, do you expect the fixed keyword lists to capture the full meaning? Keep your prediction in mind.

  • Run the sorter again by running this command:
python3 message_sorter.py

Why Start a Fresh Run?

A fresh run uses the same unchanged message_sorter.py file. This isolates the new message as the only difference in your test.

  • Enter My account is locked and I have a client demo in 10 minutes. at the Message: prompt.

You'll see the category other with low priority. The sorter routes the message to a general review action.

Why This Failure Matters

The words locked plus 10 minutes carry important context. Neither clue appears in the fixed keyword lists.

Even the word account is missing from account_keywords. The program follows its else branch exactly as written.

You found the rule set's real limit. The program runs correctly while producing an unhelpful classification.

  • Enter quit at the next Message: prompt.

Did You See a Different Result?

  • Confirm that message_sorter.py still matches the rule-based version from the previous step.
  • Check that you entered the locked-account message with the same wording shown above.
  • Still stuck? Help me trace why my locked-account test produced a different result.
Record the expected result

An expected result acts as ground truth for a repeatable test. It gives you a clear target for judging a later version of the sorter.

A locked account is an account-access issue. A client demo in 10 minutes means the customer is urgently blocked.

  • Compare the actual category other with the expected category account.
  • Compare the actual priority low with the expected priority high.

Your expected result is category account with priority high. This difference gives you a precise case for testing the replacement sorter.

The experiment only tested your existing rules. Your source file should remain unchanged.

✔️ Awesome, I've got everything!

Your terminal showed other with low priority for the locked-account message. You also recorded account with high priority as the expected result.

ⓧ I'd like to double check the full code

  • Compare your unchanged message_sorter.py file with this reference:
def sort_message(message: str) -> dict:
    message_text = message.lower()
    billing_keywords = ["refund", "charge", "invoice", "payment"]
    technical_keywords = ["error", "broken", "crash", "failed"]
    account_keywords = ["login", "password", "profile", "subscription"]

    if any(keyword in message_text for keyword in billing_keywords):
        category = "billing"
        priority = "medium"
        next_action = "Review the billing details and help with the payment issue."
    elif any(keyword in message_text for keyword in technical_keywords):
        category = "technical"
        priority = "medium"
        next_action = "Collect error details and investigate the technical problem."
    elif any(keyword in message_text for keyword in account_keywords):
        category = "account"
        priority = "medium"
        next_action = "Verify the account and help restore access or update settings."
    else:
        category = "other"
        priority = "low"
        next_action = "Review the request and route it to the right team."

    result = {
        "category": category,
        "priority": priority,
        "next_action": next_action,
    }
    return result


def show_result(result: dict) -> None:
    print("\nTriage result")
    print(f"Category: {result['category']}")
    print(f"Priority: {result['priority']}")
    print(f"Next action: {result['next_action']}\n")


def main() -> None:
    print("Message Sorter")
    print("Enter a customer message, or enter 'quit' to stop.\n")

    while True:
        message = input("Message: ").strip()

        if message.lower() == "quit":
            print("Goodbye!")
            break

        if not message:
            print("Please enter a message.\n")
            continue

        result = sort_message(message)
        show_result(result)


if __name__ == "__main__":
    main()

Why Does the File Stay Unchanged?

This step measures the rule-based sorter's behavior without editing its logic. Keeping the file unchanged gives you a stable baseline for the next version.

Your test now proves where literal keyword rules fall short. Next, you'll replace those rules with a local model that can classify the same message from its context.

Replace Keywords with a Local AI Model

Your rule-based sorter handled the refund message. The locked-account message exposed its inability to infer meaning or urgency from context.

Now you will connect the app to a local model through Ollama. A Pydantic schema will constrain each response to the four fields your sorter needs.

In this step, get ready to:
  • Define the allowed categories and priorities in a validated result model.
  • Replace keyword matching with a local model call.
  • Display the four validated fields and handle failures.
Define the validated result shape

A structured result gives the model a fixed contract. The category and priority must match specific values. The summary and next action must be strings.

Why local AI?

The model runs on your Mac through Ollama. This keeps the project free from API keys and metered cloud requests.

Your Python app still makes a real model call. The response stays on the local path you prepared earlier.

  • In the Explorer sidebar in Visual Studio Code, select message_sorter.py.
  • Add the result schema above the existing sort_message() function by pasting this block:
from typing import Literal

from ollama import ResponseError, chat
from pydantic import BaseModel, ValidationError

MODEL_NAME = "Osmosis/Osmosis-Structure-0.6B"


class TicketResult(BaseModel):
    category: Literal["billing", "technical", "account", "other"]
    priority: Literal["low", "medium", "high"]
    summary: str
    next_action: str

What does this code do?

  • The imports bring in the local model client and the validation tools used later in this step.
  • MODEL_NAME keeps the model identifier in one place.
  • TicketResult defines the shape that every accepted model response must follow.
  • Literal limits category and priority to the listed values.
  • Save message_sorter.py.
  • Start the current sorter by running this command:
python3 message_sorter.py
  • Enter I need a refund for a duplicate charge at the message prompt.
  • Enter quit at the next message prompt.

You should still see the billing result from your keyword rules. This confirms that the new result schema loads without disrupting the existing sorter.

Seeing an import error?

Confirm that your terminal prompt still shows the activated virtual environment. The pinned packages were installed inside that environment.

If the imports still fail, help me check the Python environment and installed packages.

Call the local model

The sorter now needs context instead of keyword matches. A prompt explains each category and priority. A JSON Schema tells the model exactly which fields to return.

  • In message_sorter.py, select the existing sort_message() function from its definition through return result.
  • Replace the selection with the first part of the new function:
def sort_message(message: str) -> TicketResult:
    prompt = (
        "Classify this customer support message. "
        "Billing means payment, invoice, charge, or refund. "
        "Technical means an error, broken feature, or failed software action. "
        "Account means login, profile, password, or subscription management. "
        "Other means no category matches. "
        "High priority means the customer is blocked or money may be at risk. "
        "Medium priority means work is impaired. "
        "Low priority means a routine request.\n\n"
        f"Message: {message}"
    )

How does the prompt add context?

  • The category descriptions explain the meaning behind billing, technical, account, and other requests.
  • The priority descriptions connect urgency to blocked customers and money at risk.
  • The final line inserts the learner's message into the classification request.
  • Continue the same sort_message() function by adding this block directly below the prompt:
    response = chat(
        model=MODEL_NAME,
        messages=[
            {
                "role": "system",
                "content": (
                    "Return support triage data that matches this JSON schema: "
                    f"{TicketResult.model_json_schema()}"
                ),
            },
            {"role": "user", "content": prompt},
        ],
        format=TicketResult.model_json_schema(),
        options={"temperature": 0},
    )

    content = response.message.content
    if content is None:
        raise RuntimeError("The model returned no content.")

    return TicketResult.model_validate_json(content)

What happens to the model response?

  • chat() sends the instructions and customer message to the local model.
  • TicketResult.model_json_schema() supplies the required response structure.
  • response.message.content reads the generated text from the response.
  • TicketResult.model_validate_json() rejects content that does not match the required fields and allowed values.
  • Select the existing show_result() function.
  • Replace it with this version to display the validated fields:
def show_result(result: TicketResult) -> None:
    print("\nTriage result")
    print(f"Category: {result.category}")
    print(f"Priority: {result.priority}")
    print(f"Summary: {result.summary}")
    print(f"Next action: {result.next_action}\n")

Why use field names here?

The validated response is a TicketResult object. Each field can be read directly through names such as result.category and result.summary.

  • Save message_sorter.py.

Before you run the app, do you think the model can infer both the account problem and its urgency from the locked-account message?

  • Start the AI sorter by running this command:
python3 message_sorter.py
  • Enter My account is locked and I have a client demo in 10 minutes. at the message prompt.
  • Enter quit after reading the result.

You should see account with high priority. You should also see a generated summary and next action.

That is the key upgrade. Your sorter now uses context to understand a message that the keyword lists missed.

Model call not returning a result?

Confirm that Ollama is still running on your Mac. The Python client needs the local service to reach the downloaded model.

Check each pasted block against the schema and model-call code above. A missing parenthesis or changed field name can stop validation.

If you are still stuck, help me debug the local model call and structured response.

Handle failures and verify the upgrade

A local model call can fail before the app receives valid data. Exception handling keeps the input loop readable. It also turns connection and validation problems into one useful message.

  • In message_sorter.py, select the existing main() function.
  • Replace it with this version to catch model and validation failures:
def main() -> None:
    print("AI Message Sorter")
    print("Enter a customer message, or enter 'quit' to stop.\n")

    while True:
        message = input("Message: ").strip()

        if message.lower() == "quit":
            print("Goodbye!")
            break

        if not message:
            print("Please enter a message.\n")
            continue

        try:
            result = sort_message(message)
            show_result(result)
        except (ConnectionError, ResponseError, ValidationError, RuntimeError) as error:
            print(f"Could not classify the message: {error}\n")

How does the safety net work?

  • The try block runs the model call and displays a validated result.
  • The except block catches connection, model-response, validation, and missing-content failures.
  • The loop continues after a handled failure. This lets the learner try another message without restarting the app.
  • Save message_sorter.py.

Before the final run, do you expect the exception handling to change the successful locked-account result?

  • Run the completed sorter with this command:
python3 message_sorter.py
  • Enter My account is locked and I have a client demo in 10 minutes. at the message prompt.
  • Enter quit after checking all four fields.

You should see category, priority, summary, and next action in the terminal. The category should be account. The priority should be high instead of the rule-based low result.

Still seeing the old rule-based result?

Confirm that sort_message() contains the chat() call. The old keyword lists should no longer appear inside that function.

Confirm that you saved message_sorter.py before starting the final run.

If the old behavior remains, help me compare my sorter with the completed local AI version.

✔️ Awesome, I've got everything!

Great work. Save message_sorter.py once more before continuing.

ⓧ I'd like to double check the full code

from typing import Literal

from ollama import ResponseError, chat
from pydantic import BaseModel, ValidationError

MODEL_NAME = "Osmosis/Osmosis-Structure-0.6B"


class TicketResult(BaseModel):
    category: Literal["billing", "technical", "account", "other"]
    priority: Literal["low", "medium", "high"]
    summary: str
    next_action: str


def sort_message(message: str) -> TicketResult:
    prompt = (
        "Classify this customer support message. "
        "Billing means payment, invoice, charge, or refund. "
        "Technical means an error, broken feature, or failed software action. "
        "Account means login, profile, password, or subscription management. "
        "Other means no category matches. "
        "High priority means the customer is blocked or money may be at risk. "
        "Medium priority means work is impaired. "
        "Low priority means a routine request.\n\n"
        f"Message: {message}"
    )

    response = chat(
        model=MODEL_NAME,
        messages=[
            {
                "role": "system",
                "content": (
                    "Return support triage data that matches this JSON schema: "
                    f"{TicketResult.model_json_schema()}"
                ),
            },
            {"role": "user", "content": prompt},
        ],
        format=TicketResult.model_json_schema(),
        options={"temperature": 0},
    )

    content = response.message.content
    if content is None:
        raise RuntimeError("The model returned no content.")

    return TicketResult.model_validate_json(content)


def show_result(result: TicketResult) -> None:
    print("\nTriage result")
    print(f"Category: {result.category}")
    print(f"Priority: {result.priority}")
    print(f"Summary: {result.summary}")
    print(f"Next action: {result.next_action}\n")


def main() -> None:
    print("AI Message Sorter")
    print("Enter a customer message, or enter 'quit' to stop.\n")

    while True:
        message = input("Message: ").strip()

        if message.lower() == "quit":
            print("Goodbye!")
            break

        if not message:
            print("Please enter a message.\n")
            continue

        try:
            result = sort_message(message)
            show_result(result)
        except (ConnectionError, ResponseError, ValidationError, RuntimeError) as error:
            print(f"Could not classify the message: {error}\n")


if __name__ == "__main__":
    main()

Your sorter can now classify messages with local AI and reject malformed output. Next, you will measure its results across four labeled cases.

Evaluate the AI Instead of Trusting It

Your sorter now turns an ambiguous support message into a validated contextual result. The next goal is to find out how consistently it gets the important fields right.

One convincing result cannot show whether the sorter is reliable.

An evaluation compares several predictions with labeled answers. The resulting score turns model behavior into evidence.

In this step, get ready to:
  • Create four labeled support cases.
  • Compare the model's predicted fields with the labeled answers.
  • Run the evaluation to reveal case results plus a numeric score.
Create labeled test cases

A labeled test case pairs one message with the result you expect. Four cases give every category a place in the evaluation.

  • Create evaluate.py inside the ai-message-sorter folder with the file sidebar's new-file control.
  • Enter evaluate.py as the filename.
  • Build the labeled evaluation set by pasting this code into evaluate.py:
from message_sorter import sort_message

TEST_CASES = [
    {
        "message": "I was charged twice for the same order and need a refund.",
        "expected_category": "billing",
        "expected_priority": "high",
    },
    {
        "message": "The app crashes every time I upload a spreadsheet.",
        "expected_category": "technical",
        "expected_priority": "medium",
    },
    {
        "message": "My account is locked and I have a client demo in 10 minutes.",
        "expected_category": "account",
        "expected_priority": "high",
    },
    {
        "message": "There is a typo in the getting started guide.",
        "expected_category": "other",
        "expected_priority": "low",
    },
]

How do labeled cases work?

  • The sort_message import lets the evaluator use the same classification function as your interactive app.
  • The TEST_CASES list stores four examples that cover billing, technical, account, plus other messages.
  • Each dictionary holds one message. Its expected fields provide the answers used for comparison.
  • Save evaluate.py by pressing Cmd+S on macOS or Ctrl+S on Windows.
  • Confirm the file sidebar lists evaluate.py beneath message_sorter.py.

Can't find evaluate.py?

  • Confirm the file was created inside the ai-message-sorter folder.
  • Check that the filename ends with .py.
  • Still stuck? Help me create evaluate.py in my existing Visual Studio Code workspace.
Compare each prediction

Each model call produces a category plus a priority. The loop checks both fields before recording how many matched their labels.

  • Place your cursor on the blank line below the closing bracket for TEST_CASES in evaluate.py.
  • Add the case-by-case comparison logic by pasting this code:
def main() -> None:
    correct_fields = 0
    possible_fields = len(TEST_CASES) * 2

    for number, case in enumerate(TEST_CASES, start=1):
        result = sort_message(case["message"])
        category_ok = result.category == case["expected_category"]
        priority_ok = result.priority == case["expected_priority"]
        correct_fields += int(category_ok) + int(priority_ok)

        status = "PASS" if category_ok and priority_ok else "CHECK"
        print(f"Case {number}: {status}")
        print(f"  Message: {case['message']}")
        print(
            f"  Category: {result.category} "
            f"(expected {case['expected_category']})"
        )
        print(
            f"  Priority: {result.priority} "
            f"(expected {case['expected_priority']})\n"
        )


if __name__ == "__main__":
    main()

What does this evaluation loop do?

  • The correct_fields value starts at zero. It increases whenever a predicted field matches its label.
  • The possible_fields value counts two checked fields for every case.
  • The enumerate() call gives every case a readable number starting at one.
  • The category_ok value checks the predicted category. The priority_ok value performs the priority check.
  • The status value becomes PASS only when both fields match. Every other result becomes CHECK.
  • Save evaluate.py.

Before you run the evaluator, which cases do you think will show PASS?

  • Run the four labeled cases from your activated terminal with this command:
python3 evaluate.py

What should you see?

  • You will see one section for each of the four cases.
  • Each section displays PASS when both checked fields match.
  • A mismatched category or priority produces CHECK.
  • The predicted values appear beside the expected values for inspection.

These case results expose specific strengths plus specific misses in the model's behavior.

No case results?

  • Confirm your .venv remains activated in the terminal.
  • Confirm Ollama is still running on your Mac.
  • Check that evaluate.py sits beside message_sorter.py.
  • Still stuck? Help me troubleshoot why evaluate.py is not printing its case results.
Calculate the final score

Case labels reveal individual misses. A numeric score combines all eight checked fields into one repeatable measurement.

  • Scroll to the end of main() in evaluate.py.
  • Find this final print block inside the loop:
        print(
            f"  Priority: {result.priority} "
            f"(expected {case['expected_priority']})\n"
        )

Why find this block?

This block marks the end of the output for each case. The aggregate calculation belongs below the loop so it runs once after every case has finished.

  • Place your cursor on the blank line after the print block.
  • Add the final score calculation so this part of main() looks like this:
        print(
            f"  Priority: {result.priority} "
            f"(expected {case['expected_priority']})\n"
        )

    percentage = correct_fields / possible_fields * 100
    print(f"Score: {correct_fields}/{possible_fields} fields ({percentage:.0f}%)")

How is the score calculated?

  • The percentage value divides the matched field count by the possible field count. It converts the result into a percentage.
  • The final print statement displays the raw field count plus the rounded percentage.
  • Save evaluate.py.

Before you run it again, what numeric score do you expect from the eight checked fields?

  • Run the completed evaluator with this command:
python3 evaluate.py

How do you read the finished evaluation?

  • Confirm every labeled case ends with either PASS or CHECK.
  • Confirm the final line begins with Score:.
  • Inspect every CHECK result as evidence that the model needs human judgment for that example.
  • Keep the measured score even when it falls below perfection. The evaluation is designed to reveal reliability.

You have turned your local sorter into a measured application. Its strengths plus its mistakes are now visible in one repeatable run.

Missing the final score?

  • Confirm the percentage line is indented four spaces inside main().
  • Confirm the score lines sit above if __name__ == "__main__":.
  • Still stuck? Help me fix the final score in evaluate.py.

✔️ Awesome, I've got everything!

Your saved evaluate.py file now contains four labeled cases plus the complete scoring loop.

ⓧ I'd like to double check the full code

from message_sorter import sort_message

TEST_CASES = [
    {
        "message": "I was charged twice for the same order and need a refund.",
        "expected_category": "billing",
        "expected_priority": "high",
    },
    {
        "message": "The app crashes every time I upload a spreadsheet.",
        "expected_category": "technical",
        "expected_priority": "medium",
    },
    {
        "message": "My account is locked and I have a client demo in 10 minutes.",
        "expected_category": "account",
        "expected_priority": "high",
    },
    {
        "message": "There is a typo in the getting started guide.",
        "expected_category": "other",
        "expected_priority": "low",
    },
]


def main() -> None:
    correct_fields = 0
    possible_fields = len(TEST_CASES) * 2

    for number, case in enumerate(TEST_CASES, start=1):
        result = sort_message(case["message"])
        category_ok = result.category == case["expected_category"]
        priority_ok = result.priority == case["expected_priority"]
        correct_fields += int(category_ok) + int(priority_ok)

        status = "PASS" if category_ok and priority_ok else "CHECK"
        print(f"Case {number}: {status}")
        print(f"  Message: {case['message']}")
        print(
            f"  Category: {result.category} "
            f"(expected {case['expected_category']})"
        )
        print(
            f"  Priority: {result.priority} "
            f"(expected {case['expected_priority']})\n"
        )

    percentage = correct_fields / possible_fields * 100
    print(f"Score: {correct_fields}/{possible_fields} fields ({percentage:.0f}%)")


if __name__ == "__main__":
    main()

How to use this reference

This reference shows the complete saved evaluator. Its spacing places the percentage calculation outside the loop while keeping it inside main().

Secret mission

Add a Human Review Gate

Your sorter can return valid fields while still making an uncertain judgment. In this secret mission, you will add confidence reporting and route low-confidence results to human review.

Clean Up Your Resources

Clean Up Your Resources

Everything in this project runs locally on your Mac with no subscription charge. Decide whether to keep your setup running, pause its local processes, or remove its files and model.

Resources you used:

  • The ai-message-sorter Visual Studio Code workspace containing message_sorter.py, evaluate.py, and requirements.txt.
  • The .venv Python virtual environment inside ai-message-sorter.
  • The Ollama application running on your Mac.
  • The local Osmosis/Osmosis-Structure-0.6B model occupying about 1.2 GB before runtime overhead.

Keep everything running

No action is needed. Choose this if you are still testing messages or improving the sorter.

  • Keep the ai-message-sorter folder in its current location.
  • Keep the .venv environment inside the workspace.
  • Keep the local Osmosis/Osmosis-Structure-0.6B model available.
  • Leave Ollama running while you continue using the message sorter.

Pause - I'll come back to this later

Shut down the local processes to free memory. Your source files and downloaded model stay available for another session.

  • Enter quit if the sorter is accepting messages.
  • Leave the active .venv environment by running this command:
deactivate

What Does Deactivation Change?

This command returns the terminal to your normal Python environment. The .venv folder and its pinned packages remain unchanged.

  • Quit Ollama from the macOS menu bar.

Your local processes are now paused. The workspace and model remain on disk with no subscription charge.

Delete - I don't want to use this again

Deleting the workspace with a terminal command is permanent. Save a separate copy of any code you want to keep before you continue.

  • Enter quit if the sorter is accepting messages.
  • Leave the active .venv environment by running this command:
deactivate

Why Deactivate First?

Deactivation ends the terminal session's use of .venv. The environment can then be removed with the rest of the workspace.

  • Remove the local Osmosis/Osmosis-Structure-0.6B model through Ollama's model-removal workflow.
  • Quit Ollama from the macOS menu bar.
  • Move the Ollama application from Applications to Trash.
  • Remove the ai-message-sorter folder from its parent location by running these commands:
cd ..
rm -rf ai-message-sorter

What Do These Commands Remove?

The first command moves the terminal out of the workspace. The second command permanently removes ai-message-sorter with its source files and .venv environment.

Your project files and isolated Python environment are now removed. The model is also gone after the Ollama removal finishes.

Nice Work!

Nice Work!

You made it! Your local Python message sorter now turns support messages into validated triage results.

You've learned how to:

  • Build an interactive Python sorter that keeps accepting customer messages until you enter quit. Its functions return a category, priority, summary, and recommended next action.
  • Test deterministic keyword rules with straightforward and ambiguous messages. The visible failure shows why contextual classification matters.
  • Replace brittle rules with a local AI model through Ollama. Constrain the response with Pydantic plus JSON Schema. Measure category and priority accuracy across four labeled cases.
  • Complete the Secret Mission by adding confidence reporting. Route uncertain predictions to human review without changing the public sort_message interface.

Ready to quiz yourself?