Build an AI Ticket Triage Analyzer

Train and evaluate a local AI model that routes support tickets.

Introduction

30 Second Summary

A customer can wait longer for help simply because their message reached the wrong team. Ambiguous wording makes that mistake easy.

In this project, you will build a local AI support ticket analyzer for a fictional software company. You will compare a keyword baseline against a trained text classifier using held-out evidence.

What You'll Build

You will run your analyzer to watch it classify support tickets while producing evidence you can inspect.

By the end of this project, you'll have:

  • Visible baseline failure where two ambiguous tickets reach the wrong category because first-match rules ignore wider context.
  • A trained text classifier that uses TF-IDF features with logistic regression to route tickets across four business categories.
  • Evaluation evidence showing held-out accuracy alongside a per-class report. A saved confusion matrix plus ticket_predictions.csv lets you inspect individual errors.
  • Secret Mission: Add a confidence threshold that sends uncertain predictions to a human-review queue.

Are there any prerequisites?

This project assumes basic Python syntax plus a GitHub account on a Windows computer with Visual Studio Code. Python 3.11 or newer supports the pinned packages.

Before We Start

This is your moment to commit to building a local AI support-ticket triage system that helps support teams route customer issues faster. Human judgment remains essential because incorrect routing can delay or mishandle those issues.

Set Up the Windows Project

Your local triage system needs an environment that runs the same way every time. A compatible Python interpreter prevents the pinned packages from failing during installation.

A project-specific virtual environment keeps these dependencies separate from your other Python work. Visual Studio Code keeps every project tool in one workspace.

In this step, get ready to:
  • Verify that Python 3.11 or newer is available.
  • Create the ai-ticket-triage folder with a pinned dependency file.
  • Build an isolated environment with the pinned packages.
Verify Python in Visual Studio Code

The packages in this project require Python 3.11 or newer. The version check proves that your current installation can support them.

  • Press the Windows key to open Windows search.
  • Type Visual Studio Code into the search field.
  • Press Enter to open Visual Studio Code.
  • Press Ctrl+Shift+P to open the Command Palette.
  • Type Terminal: Select Default Profile into the Command Palette.
  • Select Command Prompt as the default profile.
  • Select Terminal from the top menu.
  • Select New Terminal.

You will see a Command Prompt terminal in the lower panel.

  • Check your installed Python version by running this command:
python --version

What does this command do?

The python command starts your installed interpreter. The --version option prints its version number before exiting.

✔️ I see version 3.11 or higher

Your Python installation meets the minimum requirement for every pinned package in this project.

ⓧ I see an older version

The installed interpreter is too old for the pinned packages. Install Python 3.15.0 before creating the environment.

  • Open the official Python Windows releases page.
  • Download the Windows installer for Python 3.15.0.
  • Run the downloaded installer.
  • Select Add Python to PATH on the first installer page.
  • Complete the installation.

Why restart the editor?

An open terminal can retain the system path from before the installation. Restarting Visual Studio Code gives the new terminal access to the updated Python command.

  • Close every Visual Studio Code window.
  • Press the Windows key to open Windows search.
  • Type Visual Studio Code into the search field.
  • Press Enter to reopen Visual Studio Code.
  • Select Terminal from the top menu.
  • Select New Terminal.
  • Verify the new installation by running this command:
python --version

What does this check prove?

This check uses the Python installation available to your new terminal. A version of 3.11 or newer can install every pinned dependency.

You should now see Python 3.15.0.

Still seeing the older version?

Run the Python installer again. Confirm that Add Python to PATH is selected before completing the installation.

Still stuck? Help me make Python 3.15.0 available in my Visual Studio Code Command Prompt terminal.

ⓧ Command not found

The terminal cannot find a Python installation. Install Python 3.15.0 from the official Windows releases page.

  • Open the official Python Windows releases page.
  • Download the Windows installer for Python 3.15.0.
  • Run the downloaded installer.
  • Select Add Python to PATH on the first installer page.
  • Complete the installation.

Why restart the editor?

Visual Studio Code reads the system path when it starts. Restarting it lets a new terminal discover the installed Python command.

  • Close every Visual Studio Code window.
  • Press the Windows key to open Windows search.
  • Type Visual Studio Code into the search field.
  • Press Enter to reopen Visual Studio Code.
  • Select Terminal from the top menu.
  • Select New Terminal.
  • Verify the installation by running this command:
python --version

What does this check prove?

The command confirms that the new terminal can locate Python. It also prints the installed version.

You should now see Python 3.15.0.

Still cannot find Python?

Run the installer again. Confirm that Add Python to PATH is selected.

Need another pair of eyes? Help me fix a missing Python command in my Visual Studio Code Command Prompt terminal.

Good work. Your verified interpreter now protects the project from incompatible package installations.

Create the project environment

The project folder gives every script a predictable home. Opening that folder as a workspace also starts new terminals in the correct location.

  • Select File from the top menu in Visual Studio Code.
  • Select Open Folder....
  • Choose your Desktop as the folder location.
  • Select New Folder.
  • Enter ai-ticket-triage as the folder name.
  • Select Select Folder.
  • Confirm that you trust the folder if the Workspace Trust dialog appears.

You will see ai-ticket-triage at the top of the Explorer view.

A dependency file records the exact package versions required by the project. This makes the environment reproducible on another Windows computer.

  • Select the New File... button in the Explorer view.
  • Enter requirements.txt as the file name.
  • Add the pinned dependencies to requirements.txt by pasting this content:
pandas==3.0.6
scikit-learn==1.9.1
matplotlib==3.11.2

What do these dependencies provide?

  • pandas at version 3.0.6 loads the ticket dataset for analysis.
  • scikit-learn at version 1.9.1 provides the text-classification pipeline.
  • Matplotlib at version 3.11.2 creates the evaluation image later in the project.
  • Press Ctrl+S to save requirements.txt.
  • Confirm that requirements.txt appears under ai-ticket-triage in the Explorer view.

Dependency file missing?

Check that the Explorer heading shows ai-ticket-triage. A different heading means another folder is open.

Need help checking the file location? Help me create requirements.txt inside my ai-ticket-triage folder in Visual Studio Code.

✔️ Awesome, I've got everything!

Your pinned dependency file is complete.

ⓧ I'd like to double check the full code

pandas==3.0.6
scikit-learn==1.9.1
matplotlib==3.11.2

The virtual environment creates an isolated place for these packages. New terminals opened from this workspace begin inside the ai-ticket-triage folder.

  • Select Terminal from the top menu.
  • Select New Terminal.
  • Create the tutorial-env virtual environment by running this command:
python -m venv tutorial-env

How does this command isolate the project?

Python runs its built-in venv module to create tutorial-env. That folder receives its own interpreter and package location.

  • Confirm that tutorial-env appears in the Explorer view.

Virtual environment folder missing?

Check that the terminal path ends with ai-ticket-triage. A different path creates the environment outside this project folder.

Still missing the folder? Help me create tutorial-env inside my ai-ticket-triage folder.

  • Activate the new environment by running this command:
tutorial-env\Scripts\activate

What does activation change?

The activation script points Python package commands at tutorial-env. Packages installed next stay inside this project environment.

You should see tutorial-env in parentheses at the start of the terminal prompt.

Environment name missing from the prompt?

Confirm that the terminal profile says Command Prompt. The activation command in this step targets that Windows shell.

Still unable to activate it? Help me activate tutorial-env in a Visual Studio Code Command Prompt terminal.

Install and verify the pinned packages

The activated environment is ready to receive the project dependencies. Expect the installation to take a few minutes while the packages download.

  • Install every version from requirements.txt by running this command:
python -m pip install -r requirements.txt

How does this install stay reproducible?

The pip installer reads each pinned entry from requirements.txt. The active environment keeps those exact versions inside tutorial-env.

The Command Prompt returns when the installation finishes.

Package installation failed?

Check that tutorial-env appears at the start of the prompt. Confirm that requirements.txt is visible in the Explorer view.

Need help with the terminal output? Help me diagnose my pinned package installation error.

Before you check, which three package names do you expect to see beside their pinned versions?

  • Verify the installed packages by running this command:
python -m pip list

What does the package list prove?

This command displays the packages installed in the active environment. The output confirms whether every required version is ready.

You should see pandas at 3.0.6. You should also see scikit-learn at 1.9.1 plus matplotlib at 3.11.2.

Pinned versions missing?

Confirm that the environment name appears in your prompt. Run the installation command again if any required package is absent.

Still seeing different versions? Help me verify the packages installed in tutorial-env.

Your reproducible Windows environment is ready. Next up, a keyword router will expose the limits of first-match rules on ambiguous tickets.

Build a Rule-Based Triage Baseline

Your pinned Python environment is ready. Visual Studio Code is still focused on the ai-ticket-triage folder.

A rule-based baseline gives you a transparent benchmark before you train a classifier. This step tests whether the first matching keyword can capture the meaning of an ambiguous ticket.

In this step, get ready to:
  • Create a balanced dataset with four support categories.
  • Build an ordered keyword router for the ticket text.
  • Run ambiguous tickets through the router to expose its limitations.
Create the balanced ticket dataset

A supervised text project starts with labeled examples. This synthetic CSV gives each category 12 tickets. The equal row counts make category coverage easy to inspect.

  • In the Visual Studio Code Explorer sidebar from earlier, select the file-creation control at the top.
  • Enter tickets.csv as the file name.
  • Add the account and billing rows by pasting this first block:
ticket_id,text,category
A001,I cannot sign in after changing my password,account
A002,My account verification code never arrives,account
A003,The login page says my email is not recognized,account
A004,I am locked out after too many password attempts,account
A005,Please help me change the email on my account,account
A006,Two factor authentication will not accept my code,account
A007,My profile was disabled without warning,account
A008,I forgot my password and need account access,account
A009,The reset link for my login has expired,account
A010,I cannot verify my identity to unlock the account,account
A011,My username is attached to the wrong email,account
A012,Signing in sends me back to the login page,account
B001,I was charged twice for the same subscription,billing
B002,My invoice shows the wrong monthly amount,billing
B003,Please refund the duplicate payment,billing
B004,My credit card was declined during renewal,billing
B005,The subscription price changed on my bill,billing
B006,I need a receipt for last month payment,billing
B007,Why is there an extra fee on my invoice,billing
B008,The refund has not reached my bank account,billing
B009,My annual billing date should be changed,billing
B010,The trial ended and I was charged unexpectedly,billing
B011,Please update the card used for payment,billing
B012,The tax amount on my receipt is incorrect,billing

How is the dataset structured?

  • The header defines a ticket identifier column. It also defines the ticket text column. The final column holds the correct category.
  • The A rows provide 12 account examples.
  • The B rows provide 12 billing examples.
  • Save tickets.csv.
  • Check the editor line count. You should see 25 lines in the file.

Do the CSV rows look misaligned?

Check that every row has two commas. Confirm the header remains on the first line.

Still seeing a different line count? Help me check the first section of my ticket CSV.

  • Place your cursor at the end of the B012 row.
  • Add the technical and feature request rows by pasting this block on the next line:
T001,The dashboard crashes when I open reports,technical
T002,Data synchronization stops with an error,technical
T003,The mobile app freezes after I sign in,technical
T004,Charts are loading very slowly today,technical
T005,The export button returns a server error,technical
T006,I reset my password but the app still crashes,technical
T007,Notifications stopped working after the update,technical
T008,Uploaded files disappear from the project,technical
T009,The search page shows a blank screen,technical
T010,The API connection times out every morning,technical
T011,Reports fail to load after I update my payment card,technical
T012,The browser shows an error when I open my profile,technical
F001,Please add a dark mode option,feature_request
F002,I want to export reports to a spreadsheet,feature_request
F003,Could you add integration with our calendar,feature_request
F004,Please support custom dashboard colors,feature_request
F005,I need a way to schedule weekly reports,feature_request
F006,Please add invoice export to CSV,feature_request
F007,Could the app support multiple languages,feature_request
F008,I would like filters saved between sessions,feature_request
F009,Please add bulk editing for user accounts,feature_request
F010,A mobile widget would improve quick access,feature_request
F011,Please let us customize email notifications,feature_request
F012,Add a download option for billing history,feature_request

What does this complete?

  • The T rows provide 12 technical examples.
  • The F rows provide 12 feature request examples.
  • The completed file contains 48 labeled tickets across four balanced categories.
  • Save tickets.csv.
  • Check the editor line count. You should see 49 lines including the header.

Is the final row count different?

Check for a missing row at the join between B012 and T001. Each ticket must occupy its own line.

Need help locating the mismatch? Help me compare my CSV category counts.

✔️ Awesome, I've got everything!

  • Keep tickets.csv saved with all 48 ticket rows.

ⓧ I'd like to double check the full code

ticket_id,text,category
A001,I cannot sign in after changing my password,account
A002,My account verification code never arrives,account
A003,The login page says my email is not recognized,account
A004,I am locked out after too many password attempts,account
A005,Please help me change the email on my account,account
A006,Two factor authentication will not accept my code,account
A007,My profile was disabled without warning,account
A008,I forgot my password and need account access,account
A009,The reset link for my login has expired,account
A010,I cannot verify my identity to unlock the account,account
A011,My username is attached to the wrong email,account
A012,Signing in sends me back to the login page,account
B001,I was charged twice for the same subscription,billing
B002,My invoice shows the wrong monthly amount,billing
B003,Please refund the duplicate payment,billing
B004,My credit card was declined during renewal,billing
B005,The subscription price changed on my bill,billing
B006,I need a receipt for last month payment,billing
B007,Why is there an extra fee on my invoice,billing
B008,The refund has not reached my bank account,billing
B009,My annual billing date should be changed,billing
B010,The trial ended and I was charged unexpectedly,billing
B011,Please update the card used for payment,billing
B012,The tax amount on my receipt is incorrect,billing
T001,The dashboard crashes when I open reports,technical
T002,Data synchronization stops with an error,technical
T003,The mobile app freezes after I sign in,technical
T004,Charts are loading very slowly today,technical
T005,The export button returns a server error,technical
T006,I reset my password but the app still crashes,technical
T007,Notifications stopped working after the update,technical
T008,Uploaded files disappear from the project,technical
T009,The search page shows a blank screen,technical
T010,The API connection times out every morning,technical
T011,Reports fail to load after I update my payment card,technical
T012,The browser shows an error when I open my profile,technical
F001,Please add a dark mode option,feature_request
F002,I want to export reports to a spreadsheet,feature_request
F003,Could you add integration with our calendar,feature_request
F004,Please support custom dashboard colors,feature_request
F005,I need a way to schedule weekly reports,feature_request
F006,Please add invoice export to CSV,feature_request
F007,Could the app support multiple languages,feature_request
F008,I would like filters saved between sessions,feature_request
F009,Please add bulk editing for user accounts,feature_request
F010,A mobile widget would improve quick access,feature_request
F011,Please let us customize email notifications,feature_request
F012,Add a download option for billing history,feature_request

How to use this reference

Compare the header first. Check each lettered group for ticket numbers ending in 012.

Build the ordered keyword router

The router stores keywords under category names. Python checks those categories in their written order. The first category containing a matching keyword wins.

  • In the Visual Studio Code Explorer sidebar from earlier, select the file-creation control at the top.
  • Enter keyword_baseline.py as the file name.
  • Start the keyword map by pasting this block:
KEYWORDS = {
    "account": (
        "login",
        "sign in",
        "password",
        "account",
        "verification",
        "profile",
    ),
    "billing": (
        "invoice",
        "charged",
        "refund",
        "payment",
        "card",
        "billing",
        "receipt",
        "fee",
    ),
}

How does the first rule block work?

  • The KEYWORDS dictionary groups related phrases under each category.
  • The account category appears first. Its keywords receive the first chance to claim a ticket.
  • The billing category appears second. It runs only when no account keyword matches.
  • Save keyword_baseline.py.
  • Confirm the file shows the account category above the billing category.

Does the dictionary show an error?

Check that every category tuple ends with ),. Confirm the dictionary ends with a closing brace.

Still seeing a syntax warning? Help me check my Python keyword dictionary.

  • Place your cursor after the billing tuple.
  • Insert the remaining keyword groups before the dictionary's closing brace by pasting this block:
    "technical": (
        "error",
        "crash",
        "freeze",
        "slow",
        "sync",
        "blank screen",
        "times out",
    ),
    "feature_request": (
        "add",
        "support",
        "custom",
        "schedule",
        "option",
        "would like",
        "want to",
    ),

Why do these groups overlap?

Real support tickets often mention more than one topic. A request for invoice export contains a billing word plus a feature request word.

The ordered router cannot weigh those meanings. It stops as soon as one keyword matches.

  • Save keyword_baseline.py.
  • Confirm the dictionary now shows four category groups.

Are the new groups outside the dictionary?

Move both new groups above the final closing brace. Keep their indentation aligned with the account category.

Need help placing the entries? Help me fix the structure of my KEYWORDS dictionary.

  • Place your cursor two lines below the closing brace.
  • Add the three ambiguous demonstration tickets by pasting this block:
DEMO_TICKETS = [
    ("I reset my password but the app still crashes", "technical"),
    ("Please add invoice export to CSV", "feature_request"),
    ("I was charged twice for the same subscription", "billing"),
]

What do the demonstration tickets test?

  • The first ticket combines an account keyword with a technical problem.
  • The second ticket combines a billing keyword with a feature request.
  • The third ticket gives the baseline a clear billing example.
  • Save keyword_baseline.py.
  • Confirm DEMO_TICKETS contains three ticket and category pairs.

Does a ticket pair look incomplete?

Each list item needs ticket text plus its correct category. Keep both values inside one pair of parentheses.

Need help checking the list? Help me validate my demonstration ticket pairs.

  • Place your cursor two lines below DEMO_TICKETS.
  • Add the keyword prediction function by pasting this block:
def predict_with_keywords(text):
    normalized_text = text.lower()
    for category, keywords in KEYWORDS.items():
        if any(keyword in normalized_text for keyword in keywords):
            return category
    return "technical"

What does this function do?

  • The normalized_text value converts the ticket to lowercase for consistent matching.
  • The loop checks each category in dictionary order.
  • The first matching keyword returns its category immediately.
  • The final return supplies a technical fallback when no keyword appears.
  • Save keyword_baseline.py.
  • Check the current script for valid Python by running this command in the activated terminal:
python keyword_baseline.py

What does this check prove?

The terminal returns to its prompt without a traceback. That confirms Python can read the keyword map plus the demonstration list plus the prediction function.

Did Python report a syntax problem?

Use the reported line number to find the incomplete structure. Common causes include a missing comma after a tuple or a missing closing bracket after DEMO_TICKETS.

Still stuck on the reported line? Help me diagnose this Python syntax error.

  • Place your cursor two lines below predict_with_keywords.
  • Add the output loop plus the script entry point by pasting this block:
def main():
    print("Rule-based baseline:")
    for text, expected in DEMO_TICKETS:
        predicted = predict_with_keywords(text)
        result = "MATCH" if predicted == expected else "WRONG"
        print(f"{result}: expected={expected} predicted={predicted} text={text}")


if __name__ == "__main__":
    main()

How does the script reveal mistakes?

  • The main() function sends each demonstration ticket through the router.
  • The result value compares the predicted category with the correct category.
  • The final condition calls main() when you run the file directly.
  • Save keyword_baseline.py.

Does the entry point look misaligned?

Keep the final condition against the left edge of the file. Indent only its main() call.

Need help checking the indentation? Help me fix the main entry point.

✔️ Awesome, I've got everything!

  • Keep keyword_baseline.py saved before running the final test.

ⓧ I'd like to double check the full code

KEYWORDS = {
    "account": (
        "login",
        "sign in",
        "password",
        "account",
        "verification",
        "profile",
    ),
    "billing": (
        "invoice",
        "charged",
        "refund",
        "payment",
        "card",
        "billing",
        "receipt",
        "fee",
    ),
    "technical": (
        "error",
        "crash",
        "freeze",
        "slow",
        "sync",
        "blank screen",
        "times out",
    ),
    "feature_request": (
        "add",
        "support",
        "custom",
        "schedule",
        "option",
        "would like",
        "want to",
    ),
}

DEMO_TICKETS = [
    ("I reset my password but the app still crashes", "technical"),
    ("Please add invoice export to CSV", "feature_request"),
    ("I was charged twice for the same subscription", "billing"),
]


def predict_with_keywords(text):
    normalized_text = text.lower()
    for category, keywords in KEYWORDS.items():
        if any(keyword in normalized_text for keyword in keywords):
            return category
    return "technical"


def main():
    print("Rule-based baseline:")
    for text, expected in DEMO_TICKETS:
        predicted = predict_with_keywords(text)
        result = "MATCH" if predicted == expected else "WRONG"
        print(f"{result}: expected={expected} predicted={predicted} text={text}")


if __name__ == "__main__":
    main()

How to use this reference

Compare each section in order. Pay close attention to the category order plus the blank lines between the top-level definitions.

Run the ambiguous ticket test

The baseline is ready to make routing decisions. Before you run it, which category do you think wins when one ticket contains both password and crashes?

  • Run the completed baseline in the activated terminal with this command:
python keyword_baseline.py

What did the baseline reveal?

  • The first line is marked WRONG. The router predicts account because it finds password before considering the crash.
  • The second line is marked WRONG. The router predicts billing because it finds invoice before considering the request to add a feature.
  • The third line is marked MATCH. Its billing language points to the correct category.
  • The first matching keyword wins every decision. That early return discards the ticket's wider context.

Do you see a different result pattern?

Confirm the category order starts with account followed by billing. Check that the demonstration ticket text matches the full-code reference.

Still missing two incorrect routes? Help me compare my baseline output with the expected routing behavior.

That is the useful failure you needed. Your baseline now proves that overlapping words can defeat first-match routing.

Next, you will train on TF-IDF features. Logistic regression will learn patterns across each complete ticket.

Train the Text Classification Pipeline

Your keyword baseline from the last step exposed two incorrect routes. Its ordered rules choose the first matching keyword.

That shortcut loses the wider sentence context. A learned model can weigh patterns across the complete ticket before choosing a category.

In this step, TF-IDF turns ticket text into numerical features. Logistic regression learns how those features relate to each category.

In this step, get ready to:
  • Load the balanced ticket dataset into a training script.
  • Train a text-classification pipeline on a stratified training split.
  • Evaluate the model on held-out tickets before comparing its routes with the keyword baseline.
Load the labeled tickets

The classifier needs labeled examples before it can learn category patterns. The first runnable version loads tickets.csv and confirms that each category has the expected number of rows.

  • Select New File in the Visual Studio Code Explorer sidebar.
  • Name the new file triage.py.

You will see triage.py appear beside the existing project files.

  • Build the first runnable data check by pasting this code into triage.py:
import matplotlib.pyplot as plt
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import ConfusionMatrixDisplay, accuracy_score, classification_report
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline

from keyword_baseline import DEMO_TICKETS, predict_with_keywords

LABELS = ["account", "billing", "feature_request", "technical"]


def build_model():
    return Pipeline(
        steps=[
            ("tfidf", TfidfVectorizer(ngram_range=(1, 2), min_df=1)),
            ("classifier", LogisticRegression(max_iter=1000, random_state=42)),
        ]
    )


def main():
    data = pd.read_csv("tickets.csv")
    print("Category counts:")
    print(data["category"].value_counts())


if __name__ == "__main__":
    main()

What does this first version do?

  • The pandas import provides the table tools used to load tickets.csv.
  • The LABELS list fixes the category order used by the evaluation report.
  • The scikit-learn Pipeline combines text features with one classifier.
  • The build_model() function uses single words plus two-word phrases as features.
  • The main() function loads the dataset before printing the category balance.

Why TF-IDF and logistic regression?

TF-IDF gives more weight to useful terms across the dataset. Logistic regression turns those weighted features into category predictions.

This local approach needs no API key or service account. It keeps the focus on the supervised-learning workflow.

  • Save triage.py.

Before you run the script, ask yourself whether every category will have the same count.

  • Run the data check in the activated terminal by running:
python triage.py

What should you see?

You should see Category counts: followed by all four categories. Each category should have a count of 12.

Equal counts confirm that the synthetic dataset is balanced before training begins.

Category counts missing?

Confirm that tickets.csv sits beside triage.py in the ai-ticket-triage folder.

Check that the terminal still shows the activated tutorial-env environment.

Need another pair of eyes? Help me troubleshoot why triage.py is not printing the category counts from tickets.csv.

Train and evaluate the model

A fair evaluation keeps some labeled tickets away from the training process. The held-out split measures how the fitted model handles examples it did not learn from.

  • In triage.py locate this line: print(data["category"].value_counts()).
  • Place your cursor on the blank line below it.
  • Add the training split plus evaluation output by pasting this code:
    X_train, X_test, y_train, y_test = train_test_split(
        data["text"], data["category"], test_size=0.25, random_state=42, stratify=data["category"]
    )
    model = build_model()
    model.fit(X_train, y_train)
    predictions = model.predict(X_test)

    print(f"\nHeld-out accuracy: {accuracy_score(y_test, predictions):.3f}")
    print("\nClassification report:")
    print(classification_report(y_test, predictions, labels=LABELS, zero_division=0))

How does the evaluation stay fair?

  • The train_test_split() call reserves one quarter of the rows for testing.
  • The stratify setting preserves the category balance across both splits.
  • The model.fit() call learns only from X_train plus y_train.
  • The accuracy value summarizes the share of correct held-out predictions.
  • The report shows precision. It also shows recall plus the F1-score for each category.
  • Save triage.py.

Before you run the evaluation, ask yourself whether one overall score can reveal which category is hardest to predict.

  • Run the trained pipeline in the activated terminal by running:
python triage.py

What should you see?

You should see the balanced category counts followed by Held-out accuracy: and a decimal value.

Below that value, you should see a classification report with one row for each category. The per-class rows reveal performance that one overall score can hide.

Evaluation not running?

Check that the split code remains indented inside main().

Confirm that model.fit(X_train, y_train) appears before model.predict(X_test).

Still stuck? Help me find the problem in my training split or evaluation code.

Compare the two routing methods

The classification report summarizes held-out performance across categories. The three ambiguous tickets provide a direct comparison with the earlier first-keyword decisions.

  • In triage.py locate this line: print(classification_report(y_test, predictions, labels=LABELS, zero_division=0)).
  • Place your cursor on the blank line below it.
  • Add the side-by-side routing comparison by pasting this code:
    print("Model comparison on ambiguous tickets:")
    for text, expected in DEMO_TICKETS:
        keyword_prediction = predict_with_keywords(text)
        model_prediction = model.predict([text])[0]
        print(f"expected={expected} keyword={keyword_prediction} model={model_prediction} text={text}")

What does this comparison show?

  • The loop reuses the exact three tickets from keyword_baseline.py.
  • The keyword_prediction value preserves the original ordered-rule result.
  • The model_prediction value comes from the fitted text-classification pipeline.
  • Each printed line places the correct category beside both routing decisions.
  • Save triage.py.

Before you run the final check, ask yourself which router can use patterns from the complete sentence.

  • Run the complete training workflow in the activated terminal by running:
python triage.py

What should you see?

You should see category counts plus a held-out accuracy value. You should also see a per-class classification report.

The final section should contain three comparison lines under Model comparison on ambiguous tickets:. Each line shows expected, keyword, model, and text values.

That is a major step forward. Your project now trains on one set of tickets before proving its behavior on separate examples.

Comparison lines missing?

Confirm that the comparison loop stays inside main() with four spaces of indentation.

Check that the import from keyword_baseline includes both DEMO_TICKETS and predict_with_keywords.

Need help comparing the outputs? Help me troubleshoot why my keyword and model prediction lines are missing.

✔️ Awesome, I've got everything!

Your saved file now contains the complete training workflow for this step.

ⓧ I'd like to double check the full code

import matplotlib.pyplot as plt
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import ConfusionMatrixDisplay, accuracy_score, classification_report
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline

from keyword_baseline import DEMO_TICKETS, predict_with_keywords

LABELS = ["account", "billing", "feature_request", "technical"]


def build_model():
    return Pipeline(
        steps=[
            ("tfidf", TfidfVectorizer(ngram_range=(1, 2), min_df=1)),
            ("classifier", LogisticRegression(max_iter=1000, random_state=42)),
        ]
    )


def main():
    data = pd.read_csv("tickets.csv")
    print("Category counts:")
    print(data["category"].value_counts())

    X_train, X_test, y_train, y_test = train_test_split(
        data["text"], data["category"], test_size=0.25, random_state=42, stratify=data["category"]
    )
    model = build_model()
    model.fit(X_train, y_train)
    predictions = model.predict(X_test)

    print(f"\nHeld-out accuracy: {accuracy_score(y_test, predictions):.3f}")
    print("\nClassification report:")
    print(classification_report(y_test, predictions, labels=LABELS, zero_division=0))

    print("Model comparison on ambiguous tickets:")
    for text, expected in DEMO_TICKETS:
        keyword_prediction = predict_with_keywords(text)
        model_prediction = model.predict([text])[0]
        print(f"expected={expected} keyword={keyword_prediction} model={model_prediction} text={text}")


if __name__ == "__main__":
    main()

Your classifier now has held-out evidence behind its predictions. Next up, you will turn that evidence into inspectable files that strengthen the project.

Evaluate and Package the Portfolio Evidence

Your classifier now trains on labeled tickets. It also measures its performance on held-out examples.

A score by itself gives an employer little to inspect. This step turns the evaluation into row-level evidence.

You will also package a chart for inspection. Finally, you will document the project so someone else can reproduce it.

In this step, get ready to:
  • Save a visual comparison of actual categories against predicted categories.
  • Export held-out predictions with confidence scores.
  • Document the project for reproducible sharing.
Save the confusion matrix

A confusion matrix compares the correct category for each held-out ticket with the model's prediction. Its grid makes category-level mistakes visible.

Your script already imports Matplotlib plus the display helper. You can now turn the held-out predictions into a saved image.

  • In triage.py, locate the end of the for loop that compares the demonstration tickets.
  • Add the chart-saving logic below the loop by pasting this code:
    display = ConfusionMatrixDisplay.from_predictions(
        y_test,
        predictions,
        labels=LABELS,
        display_labels=LABELS,
        xticks_rotation=30,
    )
    display.figure_.savefig("confusion_matrix.png", bbox_inches="tight")
    plt.close(display.figure_)

What does this code do?

  • The ConfusionMatrixDisplay.from_predictions() call compares the held-out labels with the model predictions.
  • The LABELS value keeps the four categories in a consistent order.
  • The xticks_rotation=30 setting keeps the longer category names readable.
  • The savefig() call writes the chart to confusion_matrix.png.
  • The plt.close() call releases the saved figure from memory.
  • Save triage.py.

Before you run the script, pause on this question: which four labels do you expect to appear on the chart.

  • Create the confusion-matrix image by running this command:
python triage.py

What should I see?

The script prints the category counts plus the held-out evaluation. It also creates confusion_matrix.png inside ai-ticket-triage.

  • Select confusion_matrix.png in the Visual Studio Code file list.

You should see actual categories on one axis. You should see predicted categories on the other axis.

That is a useful milestone. Your evaluation now has a visual artifact that exposes category-level behavior.

Image file missing?

  • Confirm that you saved triage.py before running the script.
  • Check that the display code sits inside main() at the same indentation level as the model-comparison loop.
  • Confirm that the activated terminal is still inside ai-ticket-triage.

Still missing the image? Help me troubleshoot why triage.py does not create confusion_matrix.png.

Export held-out predictions

A predicted category shows the model's decision. A confidence score shows how strongly the model favored that decision.

Saving both values in a CSV file lets someone inspect individual held-out tickets. This turns a summary metric into row-level evidence.

  • In triage.py, locate the line that creates display.
  • Add the held-out export logic immediately above that line by pasting this code:
    probabilities = model.predict_proba(X_test)
    confidence_scores = [max(row) for row in probabilities]

    results = data.loc[X_test.index, ["ticket_id", "text", "category"]].copy()
    results.loc[:, "predicted_category"] = predictions
    results.loc[:, "confidence"] = confidence_scores
    results.to_csv("ticket_predictions.csv", index=False)

What does this code do?

  • The predict_proba() call returns one probability for each possible category.
  • The confidence_scores list keeps the highest probability for each held-out ticket.
  • The X_test.index selection preserves the original held-out rows from tickets.csv.
  • The predicted_category column records the model's chosen category.
  • The index=False setting prevents the DataFrame index from becoming an extra CSV column.
  • Save triage.py.

Before you rerun the script, pause on this question: which five column names do you expect in the exported file.

  • Generate the held-out prediction file by running this command:
python triage.py

What should I see?

The script completes its evaluation again. It creates ticket_predictions.csv inside ai-ticket-triage.

  • Select ticket_predictions.csv in the Visual Studio Code file list.

You should see ticket_id, text, category, predicted_category, plus confidence columns. Each row comes from the held-out split.

Prediction file missing columns?

  • Confirm that the export code appears before the confusion-matrix code inside main().
  • Check that results.loc[:, "predicted_category"] uses the existing predictions value.
  • Check that results.loc[:, "confidence"] uses confidence_scores.

Still seeing the wrong columns? Help me debug the held-out prediction export in triage.py.

Package and verify the project

Reproducible evidence needs context. The project should explain the business problem plus the evaluation method.

A .gitignore file keeps local environment files out of version control. A Markdown README gives someone a clear path through the work.

  • In the file list for ai-ticket-triage, use the file-creation control to create .gitignore.
  • Add the ignore rules by pasting this code into .gitignore:
tutorial-env/
__pycache__/
*.pyc

What do these rules exclude?

  • The tutorial-env/ rule excludes the local virtual environment.
  • The __pycache__/ rule excludes Python's generated cache folders.
  • The *.pyc rule excludes compiled Python files.
  • Save .gitignore.

The .gitignore file now remains visible beside your project scripts.

Ignore file not visible?

  • Confirm that the filename begins with a full stop.
  • Confirm that the file sits directly inside ai-ticket-triage.

Still cannot find it? Help me check where I created my .gitignore file in Visual Studio Code.

✔️ Awesome, I've got everything!

Your ignore rules are saved in .gitignore.

ⓧ I'd like to double check the full code

tutorial-env/
__pycache__/
*.pyc
  • In the ai-ticket-triage file list, use the file-creation control to create README.md.
  • Add the project overview plus the method by pasting this content into README.md:
# AI Support Ticket Triage Analyzer

A local text-classification project that routes fictional software support tickets into account, billing, feature request, or technical categories.

## Business problem

Manual ticket routing delays responses. A classifier can suggest a destination, but incorrect routing still has a cost, so this project evaluates the model on held-out examples and preserves confidence for human review.

## Method

1. Build a transparent keyword baseline.
2. Split a balanced synthetic dataset into training and held-out test rows.
3. Convert ticket text into TF-IDF features.
4. Train logistic regression to predict the category.
5. Inspect accuracy, per-class metrics, a confusion matrix, and row-level predictions.

What does this section explain?

  • The opening sentence states what the finished classifier does.
  • The business problem connects routing speed to the cost of incorrect predictions.
  • The method records the full workflow from baseline design through error inspection.
  • Save README.md.

The file now shows the project title plus the numbered method.

README headings missing?

  • Confirm that the filename ends with .md.
  • Check that each heading begins with the same number of hash characters shown in the reference.

Still seeing plain text? Help me check the Markdown structure in README.md.

  • Add the run instructions plus the evidence sections beneath the method list by pasting this content:
## Run on Windows Command Prompt

```text
python -m venv tutorial-env
tutorial-env\Scripts\activate
python -m pip install -r requirements.txt
python keyword_baseline.py
python triage.py
```

## Outputs

- `confusion_matrix.png`: Actual categories compared with predicted categories.
- `ticket_predictions.csv`: Held-out tickets with the correct category, predicted category, and model confidence.

## Limitations

The dataset is small and synthetic. The metrics demonstrate a reproducible evaluation workflow, not production readiness. A real deployment would require representative historical tickets, privacy review, class-imbalance checks, drift monitoring, and stakeholder-defined error costs.

## Next improvement

Route low-confidence predictions to a human instead of automatically assigning every ticket.

Why include these sections?

  • The Windows commands give another person a reproducible path to run the project.
  • The outputs section names the two portfolio artifacts.
  • The limitations section keeps the synthetic-data results in context.
  • The final section identifies human review as the next responsible improvement.
  • Save README.md.

Your README now covers the business problem through the next improvement. Someone else can follow the same local workflow.

README code block looks broken?

  • Confirm that the Windows commands sit between matching triple-backtick lines.
  • Check that the output filenames match confusion_matrix.png plus ticket_predictions.csv.

Still seeing a formatting problem? Help me debug the Markdown formatting in my project README.

✔️ Awesome, I've got everything!

Your complete project explanation is saved in README.md.

ⓧ I'd like to double check the full code

# AI Support Ticket Triage Analyzer

A local text-classification project that routes fictional software support tickets into account, billing, feature request, or technical categories.

## Business problem

Manual ticket routing delays responses. A classifier can suggest a destination, but incorrect routing still has a cost, so this project evaluates the model on held-out examples and preserves confidence for human review.

## Method

1. Build a transparent keyword baseline.
2. Split a balanced synthetic dataset into training and held-out test rows.
3. Convert ticket text into TF-IDF features.
4. Train logistic regression to predict the category.
5. Inspect accuracy, per-class metrics, a confusion matrix, and row-level predictions.

## Run on Windows Command Prompt

```text
python -m venv tutorial-env
tutorial-env\Scripts\activate
python -m pip install -r requirements.txt
python keyword_baseline.py
python triage.py
```

## Outputs

- `confusion_matrix.png`: Actual categories compared with predicted categories.
- `ticket_predictions.csv`: Held-out tickets with the correct category, predicted category, and model confidence.

## Limitations

The dataset is small and synthetic. The metrics demonstrate a reproducible evaluation workflow, not production readiness. A real deployment would require representative historical tickets, privacy review, class-imbalance checks, drift monitoring, and stakeholder-defined error costs.

## Next improvement

Route low-confidence predictions to a human instead of automatically assigning every ticket.
  • Return to triage.py.
  • Find the current ending by using this reference:
if __name__ == "__main__":
    main()

What does this ending do?

The __main__ guard runs main() when you start the script directly. It currently finishes without naming the saved artifacts.

  • Replace that ending with this completed version:
    print("\nCreated ticket_predictions.csv")
    print("Created confusion_matrix.png")


if __name__ == "__main__":
    main()

What changed?

The two new lines confirm the artifact filenames after both files are created. The existing entry-point guard remains unchanged.

  • Save triage.py.

Before you run the final check, pause on this question: which two artifact confirmations do you expect at the end of the terminal output.

  • Run the completed analyzer by using this command:
python triage.py

What should I see?

The terminal shows category counts plus a held-out accuracy value. It also shows the per-class classification report.

The final lines say Created ticket_predictions.csv plus Created confusion_matrix.png.

  • Select confusion_matrix.png in the file list.

You should see actual categories compared with predicted categories. The axes should include account, billing, feature_request, plus technical.

  • Select ticket_predictions.csv in the file list.

You should see held-out rows with predicted_category plus confidence columns. The saved chart plus the inspectable rows now support the model score.

Final artifacts not created?

  • Confirm that the terminal still shows the activated tutorial-env environment.
  • Check that triage.py remains inside ai-ticket-triage beside tickets.csv.
  • Compare the confidence export plus confusion-matrix sections with the full file below.

Still blocked? Help me debug why the final triage analyzer does not create both evidence files.

✔️ Awesome, I've got everything!

Your analyzer now saves both portfolio evidence files. Double-check that triage.py is saved.

ⓧ I'd like to double check the full code

import matplotlib.pyplot as plt
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import ConfusionMatrixDisplay, accuracy_score, classification_report
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline

from keyword_baseline import DEMO_TICKETS, predict_with_keywords

LABELS = ["account", "billing", "feature_request", "technical"]


def build_model():
    return Pipeline(steps=[("tfidf", TfidfVectorizer(ngram_range=(1, 2), min_df=1)), ("classifier", LogisticRegression(max_iter=1000, random_state=42))])


def main():
    data = pd.read_csv("tickets.csv")
    print("Category counts:")
    print(data["category"].value_counts())
    X_train, X_test, y_train, y_test = train_test_split(data["text"], data["category"], test_size=0.25, random_state=42, stratify=data["category"])
    model = build_model()
    model.fit(X_train, y_train)
    predictions = model.predict(X_test)
    print(f"\nHeld-out accuracy: {accuracy_score(y_test, predictions):.3f}")
    print("\nClassification report:")
    print(classification_report(y_test, predictions, labels=LABELS, zero_division=0))
    print("Model comparison on ambiguous tickets:")
    for text, expected in DEMO_TICKETS:
        keyword_prediction = predict_with_keywords(text)
        model_prediction = model.predict([text])[0]
        print(f"expected={expected} keyword={keyword_prediction} model={model_prediction} text={text}")
    probabilities = model.predict_proba(X_test)
    confidence_scores = [max(row) for row in probabilities]
    results = data.loc[X_test.index, ["ticket_id", "text", "category"]].copy()
    results.loc[:, "predicted_category"] = predictions
    results.loc[:, "confidence"] = confidence_scores
    results.to_csv("ticket_predictions.csv", index=False)
    display = ConfusionMatrixDisplay.from_predictions(y_test, predictions, labels=LABELS, display_labels=LABELS, xticks_rotation=30)
    display.figure_.savefig("confusion_matrix.png", bbox_inches="tight")
    plt.close(display.figure_)
    print("\nCreated ticket_predictions.csv")
    print("Created confusion_matrix.png")


if __name__ == "__main__":
    main()

You have turned one model score into reproducible evidence that another person can inspect. Your project now documents its method plus its limitations.

Secret mission

Route Uncertain Tickets to Human Review

Your classifier already measures confidence for every held-out ticket. In this mission, you will turn that confidence into a review policy that keeps uncertain predictions away from automatic routing.

Clean Up Your Resources

Clean Up Your Resources

Everything you built runs locally, so keeping it creates no ongoing cost. Decide whether to keep the project ready, pause the current session, or delete the folder.

Resources you used:

  • Your local ai-ticket-triage folder containing the complete project.

Keep everything running

No action is needed. Choose this if you want to keep refining the classifier or retain its portfolio evidence.

  • Keep the ai-ticket-triage folder on your computer.
  • Retain tutorial-env so the virtual environment remains available for the pinned project packages.
  • Preserve confusion_matrix.png as visual evidence of held-out performance.
  • Preserve ticket_predictions.csv as row-level evidence of predictions and human-review decisions.

Pause - I'll come back to this later

Close the current development session to free up memory. Your project files remain available when you return.

  • Save any file that is still open in Visual Studio Code.
  • Close the Visual Studio Code window.
  • Reopen the existing ai-ticket-triage folder in Visual Studio Code when you return.
  • Follow the Windows run instructions in README.md when you resume.

Closing Visual Studio Code ends the activated terminal session. The complete project stays on your computer.

Delete - I don't want to use this again

Remove the complete local project if you no longer need its code or generated evidence.

Deletion Is Permanent

Deleting your project folder is a big step because it removes every local artifact in one go.

  • Confirm that you no longer need the files inside ai-ticket-triage.
  • Switch back to the Command Prompt terminal panel in Visual Studio Code.
  • Delete the ai-ticket-triage folder by running these commands:
cd ..
rmdir /s /q ai-ticket-triage

What Do These Commands Do?

  • The first command moves the terminal into the folder that contains ai-ticket-triage.
  • The second command permanently removes ai-ticket-triage and everything stored inside it.
  • Check the project file sidebar in Visual Studio Code.

You should no longer see files under ai-ticket-triage. That closes the loop on your local project.

Folder Still Visible?

A project file held open by another process can interrupt deletion.

  • Close every editor tab that uses a file from ai-ticket-triage.
  • Run only the second deletion command again from the same terminal panel.

Help me diagnose why Windows cannot delete my ai-ticket-triage folder after I moved to its parent folder.

Nice Work!

Nice Work!

You made it! Your local support-ticket analyzer now demonstrates supervised text classification with inspectable held-out evidence.

You've learned how to:

  • Built a balanced synthetic dataset across four support categories. Exposed the weaknesses of a first-keyword baseline with ambiguous tickets.
  • Trained a TF-IDF plus logistic regression pipeline with scikit-learn. Protected the evaluation with a stratified held-out split.
  • Packaged portfolio evidence into three inspectable artifacts. Saved a confusion matrix in confusion_matrix.png. Exported row-level predictions to ticket_predictions.csv. Documented the method in README.md.
  • Secret Mission: Added a confidence threshold through REVIEW_THRESHOLD to route uncertain predictions to human review. A higher threshold increased the review workload. It also reduced automated routing volume.

Ready to quiz yourself?