Build an AI Support Ticket Router

Train a local classifier that routes support tickets with human review.

Introduction

30 Second Summary

Support teams often receive requests that mention several problems at once. A rushed automatic rule can send the customer to the wrong team or hide a case that needs human judgment.

In this project, you will build a local support-ticket router from labeled data. You will evaluate its suggestions on unseen tickets before using them in a human-supervised browser app.

What You'll Build

You will paste a customer request into your browser to see the suggested team, the model score, and whether the ticket is automated or held for Human Review.

By the end of this project, you'll have:

  • A local Streamlit app where you can paste a support request and see its suggested team, model score, and workflow decision.
  • A repeatable comparison that shows a keyword baseline failing on a mixed ticket before a trained text classifier routes it correctly.
  • A holdout evaluation report showing precision, recall, and F1 score on unseen support tickets.
  • Secret Mission: Reveal the ticket terms most strongly associated with the model's suggested team.

Are there any prerequisites?

Basic Python or SQL knowledge is enough for this project. Everything runs locally with open-source tools, so you do not need a paid API or cloud account.

Before We Start

Before the hands-on work begins, this step helps you commit to the workflow you are building. Your AI router will suggest a destination team for clear tickets while uncertain tickets remain with a human reviewer.

Set Up the Local AI Workspace

Your support-ticket router depends on a compatible Python installation. A project-specific environment prevents package conflicts from stopping the router before it can classify a ticket.

You will prepare that environment in Visual Studio Code. You will install pinned versions of scikit-learn plus Streamlit inside a virtual environment.

In this step, get ready to:
  • Create the ai-ticket-router workspace in Visual Studio Code.
  • Verify that your Python version supports the pinned packages.
  • Build a virtual environment with the required dependencies.
Create the workspace and verify Python

The pinned packages support Python 3.11 through 3.15. Checking the installed version now prevents compatibility problems during dependency installation.

  • Press the Windows key to open Windows search.
  • Type PowerShell into the search field.
  • Press Enter to open PowerShell.
  • Move to your Desktop by running this command:
cd ~/Desktop

What does this command do?

This changes PowerShell's current location to your Desktop. Your project folder will be easy to find there.

  • Create the ai-ticket-router folder and open it as a Visual Studio Code workspace by running these commands:
mkdir ai-ticket-router
cd ai-ticket-router
code .

What do these commands do?

  • The first command creates the ai-ticket-router folder on your Desktop.
  • The second command moves PowerShell into that folder.
  • The final command opens the current folder in Visual Studio Code.

You will see the empty ai-ticket-router workspace in Visual Studio Code. This confirms that every file you create next will land in the correct folder.

Visual Studio Code did not open?

  • Press the Windows key to open Windows search.
  • Type Visual Studio Code into the search field.
  • Press Enter to open Visual Studio Code.

Still stuck? Help me open my ai-ticket-router workspace.

  • Press Ctrl+` in Visual Studio Code to open the integrated terminal.
  • Select PowerShell from the terminal profile menu if another shell is active.
  • Check the installed Python version by running this command:
py -3 --version

What does this command check?

The Python launcher reports the installed Python 3 version. The result shows whether this installation can run the project's pinned packages.

✔️ I see version 3.11 through 3.15

Your Python installation meets the project requirements. That compatibility check is complete.

ⓧ I see an older version

Your installed version falls outside the supported range. Install the current Windows 64-bit release before creating the virtual environment.

  • Visit the official Python Windows downloads page.
  • Download the Windows 64-bit release for Python 3.15.2.
  • Run the downloaded installer.
  • Follow the installer prompts to complete the installation.
  • Close Visual Studio Code.
  • Press the Windows key to reopen Windows search.
  • Type Visual Studio Code into the search field.
  • Press Enter to reopen Visual Studio Code.

The restarted app can now detect the new Python installation. The ai-ticket-router workspace should reopen as your most recent workspace.

  • Confirm the updated Python version by running this command in a fresh integrated PowerShell terminal:
py -3 --version

What should this confirm?

The command should report Python 3.15.2. This places your installation inside the supported range.

ⓧ Command not found

The Python launcher is unavailable. Install the current Windows 64-bit release before creating the virtual environment.

  • Visit the official Python Windows downloads page.
  • Download the Windows 64-bit release for Python 3.15.2.
  • Run the downloaded installer.
  • Follow the installer prompts to complete the installation.
  • Close Visual Studio Code.
  • Press the Windows key to reopen Windows search.
  • Type Visual Studio Code into the search field.
  • Press Enter to reopen Visual Studio Code.

The restarted app can now detect the Python launcher. The ai-ticket-router workspace should reopen as your most recent workspace.

  • Confirm that the Python launcher works by running this command in a fresh integrated PowerShell terminal:
py -3 --version

What should this confirm?

The command should report Python 3.15.2. Your installation can now support the pinned packages.

Create the isolated environment

A requirements file records the exact package releases used by the router. Exact pins make the same environment reproducible when you rebuild it later.

  • Select the Explorer icon in the Visual Studio Code Activity Bar.
  • Select the New File button in the Explorer view.

You will see a filename field inside the ai-ticket-router workspace.

  • Enter requirements.txt into the filename field.

The empty requirements.txt file opens in the editor.

  • Pin the project dependencies by entering these lines in requirements.txt:
scikit-learn==1.9.1
streamlit==1.66.1

What do these pins control?

  • The first line installs scikit-learn version 1.9.1 for the machine-learning model.
  • The second line installs Streamlit version 1.66.1 for the local browser app.
  • The double equals syntax requires each exact release.
  • Press Ctrl+S to save requirements.txt.

You will see requirements.txt in the Explorer view. The editor should contain exactly two non-empty lines.

Does the requirements file look different?

  • Check that the filename is exactly requirements.txt.
  • Remove any spaces around the double equals signs.
  • Place each package on its own line.

Need help comparing the file? Help me fix requirements.txt.

✔️ Awesome, I've got everything!

Your dependency file is complete. Make sure requirements.txt is saved before continuing.

ⓧ I'd like to double check the full code

scikit-learn==1.9.1
streamlit==1.66.1

What should match?

Your file should match these two lines exactly. Each package name uses lowercase letters.

The .venv environment keeps this router's interpreter context separate from other projects. Packages installed inside it stay scoped to ai-ticket-router.

  • Create the .venv environment from the integrated PowerShell terminal by running this command:
python -m venv .venv

What does this command do?

Python creates a .venv folder inside ai-ticket-router. That folder holds the project's isolated interpreter plus its installed packages.

You will see .venv appear in the Visual Studio Code Explorer. This confirms that the environment exists inside the project.

Do you not see the environment folder?

  • Check that the integrated terminal path ends with ai-ticket-router.
  • Confirm that the earlier version check reported Python 3.11 through 3.15.

Still stuck? Help me create the .venv environment.

  • Activate the project environment by running this command:
.venv\Scripts\Activate.ps1

What does activation change?

Activation points Python plus pip at the copies inside .venv for this terminal session. Future package installations now stay inside the project environment.

You will see (.venv) at the start of the PowerShell prompt. This prefix confirms that the project environment is active.

Did PowerShell block activation?

Changing an execution policy can feel risky. This documented remedy limits the change to your Windows user account.

  • Apply the user-scoped execution policy only when activation is blocked by running this command:
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

This policy permits local activation scripts for your Windows user. The change does not alter the system-wide policy.

  • Run the activation command from the main instructions again.

Need help with the blocked script? Help me activate my Windows virtual environment.

The active environment is ready to receive the packages listed in requirements.txt.

  • Install the pinned project dependencies by running this command:
pip install -r requirements.txt

What happens during installation?

Pip reads both exact pins from requirements.txt. It installs those releases plus their supporting packages inside .venv.

The terminal prints download progress followed by installation results. Let the command finish before continuing.

You will see installation completion output in PowerShell. The (.venv) prefix should remain visible at the prompt.

Did the dependency installation fail?

  • Confirm that the PowerShell prompt starts with (.venv).
  • Compare requirements.txt with the full-code tab.
  • Confirm that your internet connection is active.

Need another pair of eyes? Help me troubleshoot the dependency installation.

Verify the installed tools

A completed installation message shows that the installer finished. Inspecting the package confirms that the active environment contains the exact machine-learning release required by the router.

  • Inspect the installed scikit-learn package by running this command:
python -m pip show scikit-learn

What does this command check?

The command displays the installed scikit-learn version plus its location. The location should point inside the active .venv environment.

You will see version 1.9.1 in the package details. Your machine-learning dependency is pinned correctly.

Is the version missing or different?

  • Confirm that the PowerShell prompt still starts with (.venv).
  • Compare requirements.txt with the full-code tab.
  • Repeat the dependency installation after correcting the file.

Still seeing another version? Help me verify scikit-learn inside .venv.

The final check starts a real local app through the installed package. This proves that the environment can serve a browser interface for the router you build later.

Before you run the final check, predict whether the example app opens in your browser or remains inside PowerShell.

  • Launch the Streamlit Hello app by running this command:
python -m streamlit hello

What should happen?

Streamlit starts a local server from the active environment. Your default browser opens the Streamlit Hello app.

You will see the Streamlit Hello app in a browser tab. This proves that Python can load the pinned package plus serve a local app.

Did the browser app not open?

  • Confirm that the PowerShell prompt showed (.venv) before you launched Streamlit.
  • Find the local address printed in the terminal.
  • Paste the printed local address into your browser.

Need help reading the output? Help me launch the Streamlit Hello app.

  • Return to the integrated PowerShell terminal in Visual Studio Code.
  • Press Ctrl+C to stop the Streamlit server.

You will see the PowerShell prompt return. Well done. Your local AI workspace is installed without leaving a server running.

Your isolated workspace is ready for support-ticket data. Next up, you will build a keyword router that exposes its own limits on an ambiguous ticket.

Build a Router That Fails

Your local workspace is ready in Visual Studio Code. You now have a safe place to build the first version of your support router.

A rule-based baseline gives you a quick routing result. The designed multi-topic ticket reveals how fixed keyword rules struggle when several business signals compete.

In this step, get ready to:
  • Create a balanced dataset of labeled support tickets.
  • Build an ordered keyword router with four test cases.
  • Run the router to expose its designed failure.
Create the labeled ticket dataset

Labeled data pairs each support request with its correct destination team. Your CSV file uses 15 examples for each of the three teams.

  • Create a data folder inside ai-ticket-router with the new-folder control in Visual Studio Code's file sidebar.
  • Create training_data.csv inside the data folder with the new-file control.
  • Add the Billing examples to the empty training_data.csv file by pasting this data:
text,label
My card was charged twice for one order,Billing
I need a refund for the duplicate payment,Billing
The invoice amount is incorrect,Billing
My subscription renewed after I cancelled it,Billing
Why was a late fee added to my account,Billing
The payment failed but money left my bank,Billing
I need a copy of last month's invoice,Billing
The discount code was not applied to my bill,Billing
I was billed for a service I did not use,Billing
Please update the company name on my invoice,Billing
The refund has not reached my card,Billing
My monthly charge increased without notice,Billing
I cannot add a new payment method,Billing
Sales tax on the receipt looks wrong,Billing
My password reset worked but the subscription was charged twice,Billing

What does this data capture?

  • The first row names the text input column.
  • The same row names the label destination column.
  • The next 15 rows connect payment issues with the Billing team.
  • Save training_data.csv.
  • Confirm that the final visible Billing row contains the password-reset and duplicate-charge ticket.

Do the Billing rows look misaligned?

Check that every row has one comma between its ticket text and team label. Make sure the header reads text,label.

Still stuck? Help me check my Billing rows.

  • Add the Account Access examples directly below the final Billing row by pasting this data:
I forgot my password and cannot sign in,Account Access
My account is locked after too many login attempts,Account Access
The password reset email never arrived,Account Access
I need to change the email address on my account,Account Access
Two factor authentication code is not working,Account Access
Please remove a former employee from our account,Account Access
I cannot verify my identity during login,Account Access
My username is not recognized,Account Access
I need administrator access for a new colleague,Account Access
The sign in page says my session expired,Account Access
I lost the phone used for authentication,Account Access
Please merge my two user accounts,Account Access
I can sign in but do not see the correct workspace,Account Access
My invitation link has expired,Account Access
I need to update my account owner,Account Access

What do these examples add?

These 15 rows connect sign-in issues with the Account Access team. They include passwords, authentication, invitations, and account administration.

  • Save training_data.csv.
  • Confirm that the Account Access section begins with a forgotten-password ticket.
  • Confirm that the section ends with an account-owner ticket.

Missing an Account Access row?

Check that the section contains 15 rows. Make sure every row ends with Account Access.

Need a second pair of eyes? Help me find the missing Account Access row.

  • Add the Technical Support examples directly below the final Account Access row by pasting this data:
Dashboard shows a blank screen after loading,Technical Support
Report export stops with an error,Technical Support
The mobile app crashes when I open analytics,Technical Support
Data synchronization has been stuck since yesterday,Technical Support
The chart does not update after changing filters,Technical Support
API requests return a server error,Technical Support
File upload remains at zero percent,Technical Support
Notifications are not being delivered,Technical Support
The search function returns no results,Technical Support
The page is extremely slow on Chrome,Technical Support
Scheduled report did not run this morning,Technical Support
The integration disconnects every few hours,Technical Support
PDF export has broken formatting,Technical Support
The button does nothing when clicked,Technical Support
Recent records are missing from the dashboard,Technical Support

What completes the dataset?

These final 15 rows connect software problems with the Technical Support team. The examples cover errors, crashes, uploads, exports, and dashboard issues.

  • Save training_data.csv.
  • Confirm that the first Technical Support row describes a blank dashboard.
  • Confirm that the final row describes missing dashboard records.

Does the dataset have an extra blank row?

Remove any empty row between the three team sections. Keep the header only once at the top of the file.

Still unsure? Help me check the complete ticket dataset.

✔️ Awesome, I've got everything!

Solid work. Your dataset now gives every destination team 15 labeled examples.

ⓧ I'd like to double check the full code

text,label
My card was charged twice for one order,Billing
I need a refund for the duplicate payment,Billing
The invoice amount is incorrect,Billing
My subscription renewed after I cancelled it,Billing
Why was a late fee added to my account,Billing
The payment failed but money left my bank,Billing
I need a copy of last month's invoice,Billing
The discount code was not applied to my bill,Billing
I was billed for a service I did not use,Billing
Please update the company name on my invoice,Billing
The refund has not reached my card,Billing
My monthly charge increased without notice,Billing
I cannot add a new payment method,Billing
Sales tax on the receipt looks wrong,Billing
My password reset worked but the subscription was charged twice,Billing
I forgot my password and cannot sign in,Account Access
My account is locked after too many login attempts,Account Access
The password reset email never arrived,Account Access
I need to change the email address on my account,Account Access
Two factor authentication code is not working,Account Access
Please remove a former employee from our account,Account Access
I cannot verify my identity during login,Account Access
My username is not recognized,Account Access
I need administrator access for a new colleague,Account Access
The sign in page says my session expired,Account Access
I lost the phone used for authentication,Account Access
Please merge my two user accounts,Account Access
I can sign in but do not see the correct workspace,Account Access
My invitation link has expired,Account Access
I need to update my account owner,Account Access
Dashboard shows a blank screen after loading,Technical Support
Report export stops with an error,Technical Support
The mobile app crashes when I open analytics,Technical Support
Data synchronization has been stuck since yesterday,Technical Support
The chart does not update after changing filters,Technical Support
API requests return a server error,Technical Support
File upload remains at zero percent,Technical Support
Notifications are not being delivered,Technical Support
The search function returns no results,Technical Support
The page is extremely slow on Chrome,Technical Support
Scheduled report did not run this morning,Technical Support
The integration disconnects every few hours,Technical Support
PDF export has broken formatting,Technical Support
The button does nothing when clicked,Technical Support
Recent records are missing from the dashboard,Technical Support

How should the dataset be structured?

This reference contains one header followed by 45 labeled tickets. Each destination team has 15 examples.

Build the keyword router

The baseline uses a small keyword dictionary to represent each destination team. Its fixed ordering makes every routing decision repeatable.

  • Create baseline.py inside the ai-ticket-router folder with the new-file control.
  • Add the ordered keyword groups to baseline.py by pasting this code:
KEYWORDS = {
    "Account Access": [
        "password",
        "login",
        "sign in",
        "locked",
        "authentication",
        "account",
    ],
    "Billing": [
        "charged",
        "payment",
        "invoice",
        "refund",
        "subscription",
        "bill",
    ],
    "Technical Support": [
        "error",
        "crash",
        "slow",
        "dashboard",
        "upload",
        "export",
    ],
}

What does this code do?

  • The KEYWORDS dictionary connects each destination team with six trigger phrases.
  • The dictionary preserves the displayed team order when the router checks for matches.
  • Each list focuses on language commonly associated with that team's work.
  • Save baseline.py.
  • Confirm that the dictionary lists Account Access first.
  • Confirm that Billing appears before Technical Support.

Seeing a Python syntax warning?

Check that every team name has a colon after it. Make sure every keyword remains inside quotation marks.

Need help with the structure? Help me check my KEYWORDS dictionary.

  • Add the four fixed test tickets directly below the closing brace of KEYWORDS by pasting this code:
TEST_TICKETS = [
    ("The dashboard shows an error", "Technical Support"),
    ("Please send a copy of my invoice", "Billing"),
    ("I cannot sign in to my account", "Account Access"),
    (
        "My password reset worked but the subscription was charged twice",
        "Billing",
    ),
]

Why use fixed test tickets?

Each item in TEST_TICKETS pairs a support request with its expected team. The set includes three simple requests plus one multi-topic request.

  • Save baseline.py.
  • Confirm that TEST_TICKETS contains four ticket and team pairs.
  • Confirm that the final pair expects Billing.

Does the final ticket look split incorrectly?

Keep the final ticket text and Billing label inside the same pair of parentheses. Keep the comma after the label.

Still seeing a warning? Help me check my TEST_TICKETS list.

  • Add route_with_keywords() below TEST_TICKETS by pasting this code:
def route_with_keywords(ticket):
    normalized_ticket = ticket.lower()
    for team, keywords in KEYWORDS.items():
        if any(keyword in normalized_ticket for keyword in keywords):
            return team
    return "Human Review"

How does the router choose a team?

  • The normalized_ticket value converts the submitted text to lowercase.
  • The loop checks each team in the order stored by KEYWORDS.
  • The function returns the first team with a matching keyword.
  • Tickets without a match return Human Review.
  • Save baseline.py.
  • Confirm that return team remains inside the keyword match.
  • Confirm that return "Human Review" remains outside the loop.

Does the function indentation look uneven?

Use four spaces for each nested level. Align return "Human Review" with the for line.

Need help with the nesting? Help me check route_with_keywords().

  • Add the test loop below route_with_keywords() by pasting this code:
for ticket, expected_team in TEST_TICKETS:
    routed_team = route_with_keywords(ticket)
    status = "PASS" if routed_team == expected_team else "FAIL"
    print(f"Expected: {expected_team} | Routed: {routed_team} | {status}")

How does the test loop report results?

  • The loop sends each test ticket through route_with_keywords().
  • The routed_team value stores the router's answer.
  • The status value records whether the answer matches expected_team.
  • The final line prints the expected team beside the routed team.
  • Save baseline.py.
  • Confirm that the test loop sits outside route_with_keywords() with no leading spaces.

Is the test loop nested inside the function?

Remove any leading spaces from the for ticket, expected_team line. Keep the next three lines indented by four spaces.

Need help with the final structure? Help me check the test loop.

✔️ Awesome, I've got everything!

Your keyword router now has its rules, test cases, routing function, and result loop.

ⓧ I'd like to double check the full code

KEYWORDS = {
    "Account Access": [
        "password",
        "login",
        "sign in",
        "locked",
        "authentication",
        "account",
    ],
    "Billing": [
        "charged",
        "payment",
        "invoice",
        "refund",
        "subscription",
        "bill",
    ],
    "Technical Support": [
        "error",
        "crash",
        "slow",
        "dashboard",
        "upload",
        "export",
    ],
}

TEST_TICKETS = [
    ("The dashboard shows an error", "Technical Support"),
    ("Please send a copy of my invoice", "Billing"),
    ("I cannot sign in to my account", "Account Access"),
    (
        "My password reset worked but the subscription was charged twice",
        "Billing",
    ),
]


def route_with_keywords(ticket):
    normalized_ticket = ticket.lower()
    for team, keywords in KEYWORDS.items():
        if any(keyword in normalized_ticket for keyword in keywords):
            return team
    return "Human Review"


for ticket, expected_team in TEST_TICKETS:
    routed_team = route_with_keywords(ticket)
    status = "PASS" if routed_team == expected_team else "FAIL"
    print(f"Expected: {expected_team} | Routed: {routed_team} | {status}")

How should the complete router look?

This reference shows the exact cumulative contents of baseline.py. The keyword groups appear first.

The test data follows the groups. The function and test loop complete the file.

Run the designed failure

The router is ready for its first real check. This run tests simple tickets alongside the request that contains both account-access and billing language.

  • Switch back to the integrated PowerShell terminal from the previous step.
  • Reactivate .venv if the terminal prompt no longer shows the environment name.
  • Predict whether all four tickets will pass before you run the router.
  • Run the fixed test cases by entering this command:
python baseline.py

What did the failure prove?

The first three tickets contain straightforward signals. Each route matches its expected team.

The final ticket contains clues for two teams. route_with_keywords() stops at the first matching team.

Account Access appears first in KEYWORDS. The password match wins before the router weighs the Billing clues.

The first three lines end with PASS. The final line reads Expected: Billing | Routed: Account Access | FAIL.

Don't see four routing results?

  • Check that your terminal is running inside the ai-ticket-router folder.
  • Check that baseline.py is saved in that folder.
  • Compare the indentation of the final test loop with the full-code reference.

Still stuck? Help me troubleshoot the missing routing output.

You have your first useful result. The repeatable failure gives the trained model a concrete problem to solve.

Your baseline can route obvious tickets while exposing its limits on competing signals. Next up, you will train a text classifier that learns weighted patterns from the labeled examples.

Train and Evaluate the Text Classifier

Your keyword router made its limitation visible in the previous step. Its first matching word wins even when a ticket contains stronger evidence for another team.

The next router uses scikit-learn to learn weighted text patterns from your labeled tickets. A holdout split tests the model on examples excluded from fitting.

The final sanity check refits a fresh pipeline on every example for the local demo. This preserves the holdout report as your evaluation evidence.

In this step, get ready to:
  • Load the labeled tickets into Python.
  • Train a TF-IDF logistic-regression pipeline on a stratified split.
  • Evaluate holdout predictions before running a full-data sanity check.
Build the text pipeline

TF-IDF turns ticket wording into numeric features based on how informative each term is. Logistic Regression learns how strongly those features point toward each support team.

  • In the Explorer sidebar in Visual Studio Code, click the New File icon.
  • Enter model.py as the file name.
  • Press Enter to create the file.
  • Set up the classifier imports and dataset path by pasting this code into model.py:
import csv
from pathlib import Path

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline

DATA_PATH = Path(__file__).parent / "data" / "training_data.csv"

What does this foundation do?

  • The scikit-learn imports provide the feature converter, classifier, evaluation report, dataset split, and pipeline container.
  • The Pipeline keeps text conversion attached to the classifier that uses those features.
  • DATA_PATH builds a path to data/training_data.csv from the location of model.py.
  • Save model.py.
  • Print the resolved dataset path by running this command in the integrated PowerShell terminal from earlier:
python -c "import model; print(model.DATA_PATH)"

What does this check prove?

This command imports model.py before printing the path stored in DATA_PATH. A printed path confirms that Python can load the new file.

You should see a path ending in ai-ticket-router\data\training_data.csv.

Does the import fail?

  • Check that model.py is inside the ai-ticket-router folder.
  • Confirm the integrated terminal path ends with ai-ticket-router.
  • Check every scikit-learn import against the code block above.

Still stuck? Help me diagnose why Python cannot import model.py.

The dataset path gives the model a stable connection to your labeled examples. The next helper reads those examples into separate text and label lists.

  • Add the data-loading helper below DATA_PATH by pasting this code:
def load_training_data(path=DATA_PATH):
    texts = []
    labels = []

    with path.open("r", encoding="utf-8", newline="") as csv_file:
        reader = csv.DictReader(csv_file)
        for row in reader:
            texts.append(row["text"])
            labels.append(row["label"])

    return texts, labels

How does the loader organize the data?

  • csv.DictReader uses the text and label header names from the CSV file.
  • texts stores the support-ticket wording used as model input.
  • labels stores the matching destination team for each ticket.
  • The function returns both lists in matching row order.
  • Save model.py.
  • Count the loaded texts and labels by running this command:
python -c "from model import load_training_data; texts, labels = load_training_data(); print(len(texts), len(labels))"

What does this count check?

The command loads both lists before printing their lengths. Equal counts confirm that every ticket still has one destination label.

You should see 45 45 in the terminal.

Seeing a file or column error?

  • Confirm training_data.csv remains inside the data folder.
  • Check that the first CSV row is exactly text,label.
  • Check the indentation inside the with block.

Need another pair of eyes? Help me fix the training-data loader.

The model needs one repeatable recipe for feature extraction and classification. A pipeline applies that same recipe during fitting and prediction.

  • Add the model-building helper below load_training_data by pasting this code:
def build_model():
    return Pipeline(
        [
            ("tfidf", TfidfVectorizer(ngram_range=(1, 2))),
            ("classifier", LogisticRegression(C=5, max_iter=1000)),
        ]
    )

How is the pipeline structured?

  • TfidfVectorizer creates features from individual words plus two-word phrases.
  • LogisticRegression learns weighted patterns for Billing, Account Access, and Technical Support.
  • C=5 controls the regularization strength used by this project.
  • max_iter=1000 gives the classifier enough iterations to finish fitting.
  • Save model.py.
  • Inspect the unfitted pipeline by running this command:
python -c "from model import build_model; print(build_model())"

What does this inspection show?

The printed pipeline should list a tfidf step followed by a classifier step. This confirms that the two stages are connected in the intended order.

Your data loader and classification pipeline are now working as reusable building blocks.

Pipeline missing a step?

  • Check that the step names are exactly tfidf and classifier.
  • Confirm both step definitions sit inside the list passed to Pipeline.
  • Check the closing brackets in build_model.

If the structure still looks wrong, help me compare my build_model function with the intended pipeline.

Evaluate on a stratified holdout split

Training performance can hide how a classifier handles unfamiliar tickets. The holdout split keeps one fifth of the labeled examples outside the fitting process.

Stratification preserves the three team proportions across both subsets. This gives every class representation in the evaluation data.

  • Add the evaluation workflow below build_model by pasting this code:
def main():
    texts, labels = load_training_data()
    train_texts, test_texts, train_labels, test_labels = train_test_split(
        texts,
        labels,
        test_size=0.2,
        random_state=42,
        stratify=labels,
    )

    evaluation_model = build_model()
    evaluation_model.fit(train_texts, train_labels)
    test_predictions = evaluation_model.predict(test_texts)

    print("Holdout evaluation")
    print(classification_report(test_labels, test_predictions, zero_division=0))


if __name__ == "__main__":
    main()

How does this protect the evaluation?

  • test_size=0.2 reserves one fifth of the examples for testing.
  • random_state=42 makes the split reproducible.
  • stratify=labels preserves the class proportions.
  • evaluation_model.fit receives only the training subset.
  • classification_report compares the holdout labels with the model's holdout predictions.
  • Save model.py.

Before you run the evaluation, do you expect the report to describe training examples or unseen examples? Keep your prediction in mind.

  • Run the holdout evaluation with this command:
python model.py

How do you read the report?

  • Precision measures how often predictions for a team are correct.
  • Recall measures how many tickets from a team the model finds.
  • F1 score balances precision with recall in one metric.
  • Support records how many holdout examples belong to each team.

You should see Holdout evaluation followed by rows for Billing, Account Access, and Technical Support. Each row includes precision, recall, F1 score, and support.

That report is your evaluation evidence because the fitted model never received the holdout examples as training input.

Report missing or script failing?

  • Check that main sits outside build_model.
  • Confirm stratify receives labels.
  • Check that fit receives train_texts plus train_labels.

Still blocked? Help me troubleshoot my holdout evaluation.

Run the full-data sanity check

The holdout report measures performance on withheld examples. A separate full-data model prepares the classifier for the local demo by fitting every labeled ticket.

The designed ambiguous ticket provides a focused sanity check after fitting. Its result shows whether the learned weights overcome the keyword-order failure.

  • In model.py, find the line that defines main.
  • Insert the prediction helper directly above main by pasting this code:
def predict_ticket(model, ticket):
    probabilities = model.predict_proba([ticket])[0]
    suggested_team, score = max(
        zip(model.classes_, probabilities),
        key=lambda item: item[1],
    )
    return {
        "suggested_team": str(suggested_team),
        "score": float(score),
    }

How does the prediction helper work?

  • predict_proba returns one model score for each destination team.
  • model.classes_ supplies the team names in the matching score order.
  • max selects the team paired with the highest score.
  • The returned dictionary keeps the suggested team beside its model score.
  • Save model.py.
  • Confirm that Python can import the prediction helper by running this command:
python -c "from model import predict_ticket; print(callable(predict_ticket))"

What does this helper check prove?

The command imports predict_ticket before checking whether Python recognizes it as callable. You should see True.

Callable check returning an error?

  • Confirm predict_ticket starts at the left edge of model.py.
  • Check the indentation inside the returned dictionary.
  • Confirm the function name uses the exact spelling predict_ticket.

Need help finding the syntax issue? Help me debug predict_ticket.

  • Inside main, find the two print statements that display the classification report.
  • Insert the full-data sanity check below those print statements by pasting this code:
    demo_model = build_model()
    demo_model.fit(texts, labels)
    demo_ticket = (
        "My password reset worked but the subscription was charged twice"
    )
    demo_prediction = predict_ticket(demo_model, demo_ticket)

    print("Full-data sanity check")
    print(f"Suggested team: {demo_prediction['suggested_team']}")
    print(f"Model score: {demo_prediction['score']:.1%}")

Why use a fresh model here?

  • demo_model starts as a new pipeline with no fitted state from the holdout evaluation.
  • fit(texts, labels) trains the demo pipeline on all 45 labeled examples.
  • demo_ticket repeats the multi-topic ticket that defeated the keyword router.
  • The printed percentage is an uncalibrated model score for this targeted sanity check.
  • Save model.py.

Before you run the complete script, do you expect the learned pipeline to repeat the Account Access mistake or suggest Billing? Keep your prediction in mind.

  • Run the evaluation and sanity check with this command:
python model.py

What should the complete run show?

The terminal first prints the holdout classification report. It then prints Full-data sanity check followed by the learned route and its model score.

The final route should be Suggested team: Billing. This confirms that the weighted text patterns handle the competing password and charge clues more effectively.

That closes the gap your keyword router exposed. The learned pipeline now suggests Billing for the designed multi-topic ticket.

Sanity check missing or routed elsewhere?

  • Confirm demo_model.fit receives texts plus labels.
  • Check that demo_ticket matches the duplicate-charge ticket in the code block.
  • Confirm predict_ticket pairs model.classes_ with the returned scores.

If Billing still does not appear, help me trace the model output from the dataset to predict_ticket.

✔️ Awesome, I've got everything!

Great work. Save model.py once more before moving on.

ⓧ I'd like to double check the full code

import csv
from pathlib import Path

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline

DATA_PATH = Path(__file__).parent / "data" / "training_data.csv"


def load_training_data(path=DATA_PATH):
    texts = []
    labels = []

    with path.open("r", encoding="utf-8", newline="") as csv_file:
        reader = csv.DictReader(csv_file)
        for row in reader:
            texts.append(row["text"])
            labels.append(row["label"])

    return texts, labels


def build_model():
    return Pipeline(
        [
            ("tfidf", TfidfVectorizer(ngram_range=(1, 2))),
            ("classifier", LogisticRegression(C=5, max_iter=1000)),
        ]
    )


def predict_ticket(model, ticket):
    probabilities = model.predict_proba([ticket])[0]
    suggested_team, score = max(
        zip(model.classes_, probabilities),
        key=lambda item: item[1],
    )
    return {
        "suggested_team": str(suggested_team),
        "score": float(score),
    }


def main():
    texts, labels = load_training_data()
    train_texts, test_texts, train_labels, test_labels = train_test_split(
        texts,
        labels,
        test_size=0.2,
        random_state=42,
        stratify=labels,
    )

    evaluation_model = build_model()
    evaluation_model.fit(train_texts, train_labels)
    test_predictions = evaluation_model.predict(test_texts)

    print("Holdout evaluation")
    print(classification_report(test_labels, test_predictions, zero_division=0))

    demo_model = build_model()
    demo_model.fit(texts, labels)
    demo_ticket = (
        "My password reset worked but the subscription was charged twice"
    )
    demo_prediction = predict_ticket(demo_model, demo_ticket)

    print("Full-data sanity check")
    print(f"Suggested team: {demo_prediction['suggested_team']}")
    print(f"Model score: {demo_prediction['score']:.1%}")


if __name__ == "__main__":
    main()

Your classifier can now learn from labeled tickets, report holdout metrics, and recover the intended Billing route. Next, you'll add a business decision layer that sends uncertain cases to a human reviewer.

Turn the Model Into a Human-Supervised App

Your classifier now weighs learned text patterns to suggest a destination team. However, even a vague ticket receives a highest-scoring label.

This step wraps the model in a local Streamlit app. A decision threshold adds human oversight by sending low-score cases to a reviewer.

In this step, get ready to:
  • Add a workflow rule that can request human review.
  • Build a local interface for submitting support tickets.
  • Test the workflow with clear and vague requests.
Add a human-review decision rule

A model score describes how strongly the fitted classifier favors its top label. The business workflow still needs to decide whether that score is high enough for automation.

  • Return to model.py in Visual Studio Code.
  • Find the end of predict_ticket() above main().
  • Add the workflow decision function by pasting this code between those functions:
def route_ticket(model, ticket, threshold=0.55):
    prediction = predict_ticket(model, ticket)
    workflow_decision = (
        prediction["suggested_team"]
        if prediction["score"] >= threshold
        else "Human Review"
    )
    return {
        **prediction,
        "workflow_decision": workflow_decision,
    }

What does this code do?

  • The threshold=0.55 parameter defines the demonstration cutoff for automated routing.
  • The call to predict_ticket() preserves the suggested team and model score.
  • The score comparison chooses the suggested team when the cutoff is met.
  • The fallback chooses Human Review when the score is below the cutoff.
  • Save model.py.
  • Confirm the existing evaluation still runs by returning to the integrated PowerShell terminal from earlier and running:
python model.py

What does this check prove?

This reruns the holdout evaluation and the full-data sanity check. A successful run confirms that the new workflow function has not broken the existing model pipeline.

You should see the classification report followed by the full-data sanity check. The final suggestion should still be Billing.

Seeing a Python error?

  • Check that route_ticket() sits after the closing brace of predict_ticket().
  • Check that the new function appears before main().
  • Check that the terminal path ends with ai-ticket-router.

Still stuck? Help me debug the route_ticket function.

✔️ Awesome, I've got everything!

Your model still evaluates successfully. The workflow decision rule is ready for the app.

ⓧ I'd like to double check the full code

import csv
from pathlib import Path

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline

DATA_PATH = Path(__file__).parent / "data" / "training_data.csv"


def load_training_data(path=DATA_PATH):
    texts = []
    labels = []

    with path.open("r", encoding="utf-8", newline="") as csv_file:
        reader = csv.DictReader(csv_file)
        for row in reader:
            texts.append(row["text"])
            labels.append(row["label"])

    return texts, labels


def build_model():
    return Pipeline(
        [
            ("tfidf", TfidfVectorizer(ngram_range=(1, 2))),
            ("classifier", LogisticRegression(C=5, max_iter=1000)),
        ]
    )


def predict_ticket(model, ticket):
    probabilities = model.predict_proba([ticket])[0]
    suggested_team, score = max(
        zip(model.classes_, probabilities),
        key=lambda item: item[1],
    )
    return {
        "suggested_team": str(suggested_team),
        "score": float(score),
    }


def route_ticket(model, ticket, threshold=0.55):
    prediction = predict_ticket(model, ticket)
    workflow_decision = (
        prediction["suggested_team"]
        if prediction["score"] >= threshold
        else "Human Review"
    )
    return {
        **prediction,
        "workflow_decision": workflow_decision,
    }


def main():
    texts, labels = load_training_data()
    train_texts, test_texts, train_labels, test_labels = train_test_split(
        texts,
        labels,
        test_size=0.2,
        random_state=42,
        stratify=labels,
    )

    evaluation_model = build_model()
    evaluation_model.fit(train_texts, train_labels)
    test_predictions = evaluation_model.predict(test_texts)

    print("Holdout evaluation")
    print(classification_report(test_labels, test_predictions, zero_division=0))

    demo_model = build_model()
    demo_model.fit(texts, labels)
    demo_ticket = (
        "My password reset worked but the subscription was charged twice"
    )
    demo_prediction = predict_ticket(demo_model, demo_ticket)

    print("Full-data sanity check")
    print(f"Suggested team: {demo_prediction['suggested_team']}")
    print(f"Model score: {demo_prediction['score']:.1%}")


if __name__ == "__main__":
    main()

How to compare this file

Compare the function order and indentation with your file. The new route_ticket() function should sit between predict_ticket() and main().

Build the Streamlit interface

The app needs a fitted model that can be reused across Streamlit reruns. Resource caching keeps that model available when someone submits a ticket.

  • Create app.py inside the open ai-ticket-router folder using the new-file control in the Visual Studio Code sidebar.
  • Build the cached model and page introduction by pasting this code into app.py:
import streamlit as st

from model import build_model, load_training_data, route_ticket


@st.cache_resource
def get_model():
    texts, labels = load_training_data()
    model = build_model()
    model.fit(texts, labels)
    return model


model = get_model()

st.title("AI Support Ticket Router")
st.write(
    "Route clear support requests automatically and send uncertain cases "
    "to a human reviewer."
)

What does this code do?

  • The imports connect the interface to the model functions you already built.
  • The get_model() function loads every local training example and fits a fresh model for the demo.
  • The @st.cache_resource decorator reuses the fitted model across app reruns.
  • The title and description tell the user what the router does.
  • Save app.py.
  • Start the local Streamlit server by running this command in the integrated PowerShell terminal:
python -m streamlit run app.py

What happens when the app starts?

Streamlit starts a local server and opens the app in your default browser. The terminal remains busy while that server is running.

You should see the AI Support Ticket Router title and its short description in the browser. That visible page confirms the cached model loaded successfully.

App did not open?

  • Check that the integrated terminal still shows the active .venv environment.
  • Check that app.py is saved inside ai-ticket-router.
  • Check the terminal output for a local address if the browser did not open automatically.

Need a hand? Help me troubleshoot the Streamlit app launch.

A Streamlit form groups the ticket field with one submit action. The app processes the ticket only after the user requests a route.

  • Return to app.py in Visual Studio Code.
  • Add the ticket form below the introductory st.write() block by pasting:
with st.form("ticket_form"):
    ticket = st.text_area(
        "Support ticket",
        placeholder="Describe the customer's issue",
    )
    submitted = st.form_submit_button("Route ticket", type="primary")

How does the form work?

  • The ticket_form container groups the input with its submit button.
  • The multiline field stores the submitted support request in ticket.
  • The submitted value becomes true when the user selects the routing button.
  • Save app.py.
  • Return to the browser tab running the app.

You should now see a multiline Support ticket field and a Route ticket button beneath the introduction.

Form missing from the page?

  • Check that the entire form block starts at the left edge of app.py.
  • Check that the ticket and submitted lines are indented inside the form.
  • Check the running terminal for a Python syntax error after saving.

Still missing? Help me fix the Streamlit form.

The final interface section connects the submitted text to the business workflow. It keeps the model suggestion visible even when automation is refused.

  • Add the result-handling block below the form in app.py by pasting:
if submitted:
    if not ticket.strip():
        st.write("Enter a support ticket before routing it.")
    else:
        result = route_ticket(model, ticket)
        st.write("### Routing result")
        st.write("Suggested team:", result["suggested_team"])
        st.write("Workflow decision:", result["workflow_decision"])
        st.write("Model score:", f"{result['score']:.1%}")
        st.write(
            "This score is an uncalibrated model output used with an "
            "illustrative 55% review threshold."
        )

How are results handled?

  • The empty-input check asks for ticket text before calling the model.
  • The route_ticket() call returns the suggestion, score, and workflow decision.
  • The result section shows the model output separately from the automation decision.
  • The final note labels the score as uncalibrated and identifies the threshold as illustrative.
  • Save app.py.
  • Return to the browser tab running the app.

You should still see the title, ticket field, and routing button. The successful rerun confirms that the result block has valid Python syntax.

Page replaced by an error?

  • Check that if submitted: starts at the left edge of the file.
  • Check that the result lines are indented beneath the else: line.
  • Check that route_ticket appears in the import from model.

Need help? Help me debug the result display.

✔️ Awesome, I've got everything!

Your interface contains the cached model, ticket form, workflow output, and score caveat.

ⓧ I'd like to double check the full code

import streamlit as st

from model import build_model, load_training_data, route_ticket


@st.cache_resource
def get_model():
    texts, labels = load_training_data()
    model = build_model()
    model.fit(texts, labels)
    return model


model = get_model()

st.title("AI Support Ticket Router")
st.write(
    "Route clear support requests automatically and send uncertain cases "
    "to a human reviewer."
)

with st.form("ticket_form"):
    ticket = st.text_area(
        "Support ticket",
        placeholder="Describe the customer's issue",
    )
    submitted = st.form_submit_button("Route ticket", type="primary")

if submitted:
    if not ticket.strip():
        st.write("Enter a support ticket before routing it.")
    else:
        result = route_ticket(model, ticket)
        st.write("### Routing result")
        st.write("Suggested team:", result["suggested_team"])
        st.write("Workflow decision:", result["workflow_decision"])
        st.write("Model score:", f"{result['score']:.1%}")
        st.write(
            "This score is an uncalibrated model output used with an "
            "illustrative 55% review threshold."
        )

How to compare this file

Compare the imports, indentation, and block order with your file. The form should appear after the page introduction, while the submitted result block should appear after the form.

Test clear and vague tickets

The final check compares a clear business signal with a vague request. This reveals whether the decision layer can automate one case while reserving the other for review.

  • Stop the current Streamlit server by pressing Ctrl+C in the integrated terminal.

Before you restart the app, which request do you expect the workflow to automate?

  • Restart the completed app by running:
python -m streamlit run app.py

What does this command run?

This starts the completed local interface through the installed Streamlit module. It also loads the full local dataset and fits the cached demo model.

  • Enter My password reset worked but the subscription was charged twice in the Support ticket field.
  • Click Route ticket.

You should see Billing as the suggested team. You should also see Billing as the workflow decision.

  • Replace the ticket text with I need help with my request.
  • Click Route ticket again.

You should see Human Review as the workflow decision. The model suggestion and score should remain visible above the threshold note.

Why is the threshold separate?

The classifier always ranks its available labels. The workflow rule translates that ranking into an operational action.

The 55% threshold is an illustrative project value. A production threshold requires validation against real routing costs, calibration data, and review capacity.

Getting the wrong workflow decision?

  • Check that route_ticket() uses threshold=0.55.
  • Check that workflow_decision falls back to Human Review below the threshold.
  • Check that you copied each test ticket exactly before submitting it.

Still seeing a different result? Help me inspect the routing decision.

That is the governance layer working. Your app automates the clear case while protecting the vague request with human review.

Secret mission

Show the Words Behind the Prediction

Your router shows a suggested team plus a model score. Extend it with an inspection aid that reveals up to three ticket terms carrying positive learned weight for the suggested team.

Clean Up Your Resources

Clean Up Your Resources

This local project has no ongoing costs. Decide whether to keep the folder for more practice, pause the local app, or delete the project entirely.

Resources you used:

  • The local Streamlit server process that runs app.py.
  • Your local ai-ticket-router folder, including .venv, requirements.txt, data/training_data.csv, baseline.py, model.py, and app.py.

Keep everything running

Keeping the folder preserves your trained router for more experiments. You can restart the local app whenever you want to test another support ticket.

  • Keep the ai-ticket-router folder on your computer.
  • Switch back to the integrated PowerShell terminal when you return to the project.
  • Reactivate the project environment by running this command:
.venv\Scripts\Activate.ps1

Why Reactivate the Environment?

This switches PowerShell back to the isolated environment containing the project's pinned dependencies. Your prompt shows that the environment is active again.

  • Start the ticket router by running this command:
python -m streamlit run app.py

What Starts Here?

This starts the local app server. Your browser opens the support ticket router with its routing result and model-associated terms.

Pause - I'll come back to this later

Pausing stops the local process. Your project files stay unchanged for the next time you return.

  • Switch back to the integrated PowerShell terminal.
  • Press Ctrl+C if the terminal still shows server activity.

The server stops after PowerShell returns to its normal prompt.

  • Exit the active virtual environment by running this command:
deactivate

What Does This Command Do?

This removes the virtual environment from the current PowerShell session. The environment marker disappears from the prompt while every project file remains available.

Delete - I don't want to use this again

Deleting the folder is permanent. The removal stays confined to your local ai-ticket-router project.

  • Switch back to the integrated PowerShell terminal.
  • Press Ctrl+C if the app server is still running.
  • Exit the active virtual environment by running this command:
deactivate

Why Deactivate Before Deleting?

This releases the virtual environment from the current PowerShell session. The project folder is ready to remove once the environment marker disappears.

  • Close Visual Studio Code.
  • Press the Windows key to open the search bar.
  • Type File Explorer into the search bar.
  • Press Enter to open it.
  • Browse to the location containing the ai-ticket-router folder.
  • Right-click the ai-ticket-router folder.
  • Select Delete.

The ai-ticket-router folder disappears from its previous location. This removes the virtual environment, synthetic dataset, Python files, and pinned requirements together.

Nice Work!

Nice Work!

You did it! You've built a local Streamlit support-ticket router that combines a trained classifier with human oversight.

You've learned how to:

  • Compare a brittle keyword baseline with a multiclass classifier trained on labeled data using TF-IDF features and logistic regression.
  • Run a stratified holdout evaluation that reports precision, recall, and F1 score on unseen support tickets.
  • Apply a decision threshold that sends low-score requests to Human Review while keeping the model's suggested team visible.
  • Secret Mission: Reveal model-associated ticket terms as a compact inspection aid for reviewers.

Ready to quiz yourself?