Build a Support Ticket Triage System

Build and evaluate a Python model that routes banking tickets by intent.

Introduction

30 Second Summary

A support message can look obvious to a person. A routing system may still send it to the wrong queue after latching onto one familiar word.

In this project, you will build a command-line support ticket triage system that routes banking messages into six categories. You will expose the limits of a keyword router before training a classifier whose mistakes you can measure.

What You'll Build

You enter a banking support message in your terminal. The system returns one of six predicted intents beside a probability estimate.

By the end of this project, you'll have:

  • An interactive ticket router that accepts banking support messages and predicts where each ticket belongs.
  • A repeatable model comparison that shows whether the trained classifier beats a most-frequent baseline. Its macro F1, per-class metrics, confusion matrix, and real mistakes make the result explainable.
  • A public GitHub demo with runnable code, setup instructions, dataset attribution, measured results, and honest model limitations.
  • Secret Mission: Add a human review route for low-probability predictions before the system routes them automatically.

Are there any prerequisites?

No prior Python or machine learning experience is required. You only need a Windows laptop with internet access because every setup step is included.

Before We Start

Before any hands-on work, lock in the system you are building and why its evaluation matters. This commitment keeps every later model choice tied to how real banking support tickets are routed.

Set Up Your Windows Python Workspace

Your future classifier depends on one consistent Python interpreter. A mismatch can send package installs to a different Python than the editor runs.

In this step, you will install Python 3 plus Visual Studio Code. The Python extension connects the editor to a project-specific virtual environment.

In this step, get ready to:
  • Verify that Python 3 meets the project requirement.
  • Prepare Visual Studio Code for Python development.
  • Run hello.py through the selected .venv interpreter.
Install Python and Visual Studio Code

The later classifier uses scikit-learn 1.9.1. This release requires Python 3.11 or newer.

  • Press the Windows key to open the Windows search bar.
  • Type Command Prompt and press Enter.
  • Check your installed Python 3 version by running this command:
py -3 --version

What Does This Check Prove?

The Windows launcher looks for an installed Python 3 interpreter. Its output tells you whether the interpreter meets the later dependency requirement.

✔️ I see version 3.11 or higher

Your installed Python meets the 3.11 minimum. That gives later dependency installs a compatible interpreter.

ⓧ I see an older version

Your current interpreter is too old for scikit-learn 1.9.1.

  • Open python.org in your browser.
  • Select the first Download Python option.
  • Open the downloaded Python installer from your Downloads folder.
  • Approve the Windows permission request if it appears.
  • Complete the installer using its default options.
  • Return to the Command Prompt window from earlier.
  • Repeat the version check shown above.

The output now reports Python 3.11 or newer.

ⓧ Command not found

Windows cannot find an installed Python interpreter yet.

  • Open python.org in your browser.
  • Select the first Download Python option.
  • Open the downloaded Python installer from your Downloads folder.
  • Approve the Windows permission request if it appears.
  • Complete the installer using its default options.
  • Return to the Command Prompt window from earlier.
  • Repeat the version check shown above.

The output now reports Python 3.11 or newer.

Your compatible interpreter is in place. Visual Studio Code gives you one workspace for files plus interpreter controls.

  • Open the official Getting Started with Python in VS Code guide.
  • Download the User setup for Windows.
  • Open the downloaded Visual Studio Code installer.
  • Complete the installer using its default options.
  • Press the Windows key to open the Windows search bar.
  • Type Visual Studio Code and press Enter.

You will see the Visual Studio Code welcome window. Your interpreter plus editor are ready for the project workspace.

Having Trouble With an Installer?

Confirm that the installer finished before repeating the version check or opening Visual Studio Code. Reopen the installer from your Downloads folder if Windows closed it before completion.

Ask for targeted help here: help me troubleshoot my Windows Python or Visual Studio Code installation. Share the installer behavior in the NextWork community if the issue continues.

Create the isolated workspace

A virtual environment stores this project's future packages inside .venv. This keeps the ticket router isolated from packages used by other Python projects.

  • Click the Extensions icon in the left Activity Bar.
  • Type Python into the Extensions search box.
  • Select the Python extension published by Microsoft.
  • Click Install on the extension page.

The extension page confirms that the Python extension is installed.

What Does the Python Extension Add?

The extension adds Python-aware editing plus controls for running files.

It also exposes the commands that create virtual environments and select interpreters.

A named folder keeps every project file in one visible location. You will place this folder on your Desktop so it is easy to find.

  • Click File in the top menu bar.
  • Click Open Folder.
  • Select Desktop in the folder picker.
  • Click New folder.
  • Type support-ticket-triage and press Enter.
  • Select the support-ticket-triage folder.
  • Click Select Folder.
  • Confirm that you trust the folder if a workspace trust prompt appears.

You will see support-ticket-triage at the top of the Explorer sidebar. This confirms that VS Code is working inside the correct folder.

Creating .venv can take a moment while VS Code prepares an isolated Python installation. A short pause here is expected.

  • Click View in the top menu bar.
  • Click Command Palette.
  • Type Python: Create Environment into the Command Palette.
  • Select Python: Create Environment.
  • Select Venv as the environment type.
  • Select the installed Python 3.11 or newer interpreter.

You will see a .venv folder in the Explorer sidebar. That folder confirms that the isolated environment exists.

Cannot Create .venv?

Return to the Extensions view if the environment command is missing. Confirm that the Python extension finished installing.

Choose an interpreter that reports Python 3.11 or newer if environment creation stops.

Ask for help here: help me create .venv in my support-ticket-triage workspace.

Select the interpreter and run Python

The selected interpreter controls which Python runs from the editor. Selecting .venv keeps editor runs aligned with the isolated workspace.

  • Click View in the top menu bar.
  • Click Command Palette.
  • Type Python: Select Interpreter into the Command Palette.
  • Select Python: Select Interpreter.
  • Select the interpreter inside .venv.

You will see .venv in the Visual Studio Code status bar. This confirms that the editor is using the project environment.

  • Click the New File icon in the Explorer sidebar.
  • Type hello.py and press Enter.

You will see hello.py in the Explorer sidebar. The file is ready for the workspace check.

  • Add the workspace check to hello.py by pasting this line:
print("Workspace ready")

What Does This Code Do?

The print() function sends the text inside its parentheses to the terminal. This one-line program gives you a visible signal that the selected interpreter can run your file.

  • Save hello.py by pressing Ctrl+S.

Before you run it, consider this: what exact line do you expect to see in the terminal.

  • Click the Run Python File button in the top-right corner of the editor.

The integrated terminal prints Workspace ready. That result confirms that your folder, selected interpreter, extension, and Python file work together.

Do You See a Different Result?

Confirm that hello.py is saved inside support-ticket-triage. Compare the file against the complete version below if the terminal mentions a syntax problem.

Check the status bar for .venv if the wrong interpreter runs.

Ask for help here: help me run hello.py with the selected .venv interpreter.

✔️ Awesome, I've got everything!

Your saved hello.py file now provides a working Python workspace check.

ⓧ I'd like to double check the full code

Here is the complete hello.py file for comparison.

print("Workspace ready")

Your Windows Python workspace now runs files through its isolated interpreter. Next, you will build the first keyword router and expose where its rules fall short.

Build a Keyword Router

Your Python workspace can now run project code. Your support tickets still need a system that chooses a route from each message.

This first version uses a rule-based router to match keywords. Its single-word matches create the blind spot that this step exposes.

In this step, get ready to:
  • Create keyword rules for four support ticket topics.
  • Run four straightforward messages through the router.
  • Expose the router with an ambiguous withdrawal message.
Create the routing rules

Each rule looks for a word linked to one support topic. The function returns a route as soon as it finds a match.

  • Create ticket_triage.py inside the support-ticket-triage folder using the Visual Studio Code file sidebar.
  • Paste the routing function into ticket_triage.py using the code below:
def rule_based_route(ticket):
    text = ticket.lower()
    if "cash" in text or "withdraw" in text or "atm" in text:
        return "cash_withdrawal_not_recognised"
    if "card" in text:
        return "card_payment_not_recognised"
    if "transfer" in text:
        return "failed_transfer"
    if "passcode" in text or "password" in text or "code" in text:
        return "passcode_forgotten"
    return "unknown"

What Does This Code Do?

  • The ticket.lower() call makes matching work with uppercase or lowercase input.
  • The first matching condition returns the route immediately.
  • The final unknown route catches messages without any listed keyword.
  • Save ticket_triage.py.
  • Confirm the editor shows no red syntax markers inside rule_based_route().

Seeing a Syntax Problem?

Check that every return line sits inside its matching condition. Confirm that the final return "unknown" lines up with the first if statement.

Ask for help with the function's indentation:

Demonstrate the simple routes

Straightforward examples show what keyword rules can handle. Each message contains a word that points directly to one route.

  • Add the demonstration below rule_based_route() using this code:
def show_rule_based_obstacle():
    examples = [
        "I do not recognize this card payment",
        "There is a cash withdrawal I did not make",
        "My bank transfer failed",
        "I forgot the code for my banking app",
    ]

    print("Keyword router demonstration")
    for example in examples:
        print(f"- {example}")
        print(f"  predicted: {rule_based_route(example)}")


if __name__ == "__main__":
    show_rule_based_obstacle()

How Does the Demonstration Work?

  • The examples list stores four sample support messages.
  • The loop passes each message to rule_based_route().
  • The direct-run check starts the demonstration when you run the file.
  • Save ticket_triage.py.
  • Run the four straightforward examples in the VS Code terminal using this command:
python ticket_triage.py

What Should You See?

  • The card message routes to card_payment_not_recognised.
  • The cash message routes to cash_withdrawal_not_recognised.
  • The transfer message routes to failed_transfer.
  • The banking app code message routes to passcode_forgotten.

That is your first working ticket router. Four direct messages now reach routes that match their topics.

Router Output Missing?

Confirm that the terminal is running inside the support-ticket-triage folder. Check that the direct-run block sits below both functions.

Ask for help with missing router output:

Expose the ambiguous withdrawal

A topic word cannot describe the full meaning of a ticket. A withdrawal can be unrecognized or declined while still containing the same cash-related words.

  • Replace the existing show_rule_based_obstacle() function with this expanded demonstration:
def show_rule_based_obstacle():
    examples = [
        "I do not recognize this card payment",
        "There is a cash withdrawal I did not make",
        "My bank transfer failed",
        "I forgot the code for my banking app",
        "The cash withdrawal was declined",
    ]

    print("Keyword router demonstration")
    for example in examples:
        print(f"- {example}")
        print(f"  predicted: {rule_based_route(example)}")

    print(
        "\nDesigned obstacle: the final message is about a declined withdrawal, "
        "but the keyword rule predicts an unrecognized withdrawal.\n"
    )

What Changed?

The fifth message tests whether the router can distinguish two intents that share withdrawal vocabulary. The final print statement makes the resulting limitation visible.

  • Save ticket_triage.py.

Before you run the router, which route do you think the final withdrawal message receives?

  • Run the expanded demonstration in the VS Code terminal using this command:
python ticket_triage.py

The Designed Obstacle

You'll see The cash withdrawal was declined paired with cash_withdrawal_not_recognised. The cash keyword identifies the topic while missing the declined outcome.

This shortfall is intentional. It gives you a measurable reason to replace keyword matching with learned word combinations.

Final Message Not Printed?

Confirm that the declined-withdrawal message sits inside the examples list. Check that the closing square bracket comes after all five strings.

Ask for help with the designed obstacle:

You have now proved exactly where the keyword router breaks. Its visible mistake becomes the baseline for the trained classifier.

✔️ Awesome, I've got everything!

Great. Double check that you saved ticket_triage.py with all five examples.

ⓧ I'd like to double check the full code

Compare your complete ticket_triage.py file with this reference.

def rule_based_route(ticket):
    text = ticket.lower()
    if "cash" in text or "withdraw" in text or "atm" in text:
        return "cash_withdrawal_not_recognised"
    if "card" in text:
        return "card_payment_not_recognised"
    if "transfer" in text:
        return "failed_transfer"
    if "passcode" in text or "password" in text or "code" in text:
        return "passcode_forgotten"
    return "unknown"


def show_rule_based_obstacle():
    examples = [
        "I do not recognize this card payment",
        "There is a cash withdrawal I did not make",
        "My bank transfer failed",
        "I forgot the code for my banking app",
        "The cash withdrawal was declined",
    ]

    print("Keyword router demonstration")
    for example in examples:
        print(f"- {example}")
        print(f"  predicted: {rule_based_route(example)}")

    print(
        "\nDesigned obstacle: the final message is about a declined withdrawal, "
        "but the keyword rule predicts an unrecognized withdrawal.\n"
    )


if __name__ == "__main__":
    show_rule_based_obstacle()

What to Compare

Check the order of the keyword conditions. Confirm that the demonstration contains five messages followed by the direct-run block.

Your weak baseline is working as designed. Next, you'll train a classifier that learns from labeled support messages.

Train the Intent Classifier

Your keyword router can spot a topic word. The declined-withdrawal example proved that one word cannot capture the full intent of a ticket.

A supervised learning model can learn combinations of words from labeled examples. The BANKING77 dataset supplies those examples across six selected banking intents.

In this step, get ready to:
  • Install the pinned machine learning dependency.
  • Download the official BANKING77 training and test data.
  • Compare a most-frequent baseline with a TF-IDF classifier.
Pin and install scikit-learn

The scikit-learn library provides the baseline and text-classification tools used in this project. Pinning version 1.9.1 makes the dependency reproducible.

  • Select the support-ticket-triage folder in the VS Code Explorer sidebar.
  • Click the new-file icon at the top of the Explorer sidebar.
  • Name the new file requirements.txt.
  • Pin the project dependency by adding this line:
scikit-learn==1.9.1

Why pin the dependency?

The exact pin tells the installer to use scikit-learn 1.9.1. Future installs can recreate the same dependency version.

  • Save requirements.txt.
  • Switch back to the VS Code terminal that uses .venv.
  • Install the pinned dependency by running this command:
py -m pip install -r requirements.txt

What does this command do?

The command asks Python's package installer to read requirements.txt. The selected virtual environment keeps the installed package isolated inside this project.

Good progress. Your .venv environment now contains the machine learning library used by the classifier.

Dependency installation failed?

Confirm that the VS Code terminal shows .venv in its prompt. Select the project interpreter again if the environment is missing.

Check that requirements.txt contains the dependency line exactly as shown above.

help me diagnose my scikit-learn installation failure

Download and filter BANKING77

The official BANKING77 CSV files contain ticket text in the text field. Each matching intent appears in the category field.

  • Switch back to ticket_triage.py in the editor.
  • Place the data imports and project constants above rule_based_route(ticket) by adding this code:
import csv
from pathlib import Path
from urllib.request import urlretrieve

TRAIN_URL = (
    "https://raw.githubusercontent.com/PolyAI-LDN/task-specific-datasets/"
    "master/banking_data/train.csv"
)
TEST_URL = (
    "https://raw.githubusercontent.com/PolyAI-LDN/task-specific-datasets/"
    "master/banking_data/test.csv"
)

DATA_DIR = Path("data")
TRAIN_FILE = DATA_DIR / "train.csv"
TEST_FILE = DATA_DIR / "test.csv"

SELECTED_LABELS = {
    "card_payment_not_recognised",
    "cash_withdrawal_not_recognised",
    "declined_card_payment",
    "declined_cash_withdrawal",
    "failed_transfer",
    "passcode_forgotten",
}

What does this code set up?

  • The URL constants identify the official training and test files.
  • The path constants place both files inside the local data folder.
  • The label set limits the project to six fine-grained support intents.
  • Scroll to the bottom of ticket_triage.py.
  • Replace the current direct-call entry point with this download function and runnable entry point:
def download_data():
    DATA_DIR.mkdir(exist_ok=True)

    if not TRAIN_FILE.exists():
        print("Downloading BANKING77 training data...")
        urlretrieve(TRAIN_URL, TRAIN_FILE)

    if not TEST_FILE.exists():
        print("Downloading BANKING77 test data...")
        urlretrieve(TEST_URL, TEST_FILE)


def main():
    show_rule_based_obstacle()
    download_data()


if __name__ == "__main__":
    main()

How does the download work?

The function creates the data folder when needed. It downloads each file only when that file is missing.

The new main() function keeps the existing keyword demonstration. It downloads the dataset immediately afterward.

  • Save ticket_triage.py.
  • Download the training and test files by running this command:
python ticket_triage.py

What should this run create?

The script retains the keyword-router demonstration at the top of its output. It also creates data/train.csv and data/test.csv.

You should see both CSV files inside the data folder in the VS Code Explorer sidebar.

Dataset files missing?

Confirm that your laptop has an active internet connection. The first run needs access to the official raw files.

Check the terminal for a download-related error. Remove any empty CSV file before trying the command again.

help me troubleshoot the BANKING77 download

The downloaded rows still cover all 77 dataset intents. The next function keeps only the six labels used by this classifier.

  • Place the split-loading function between download_data() and main() by adding this code:
def load_split(path):
    texts = []
    labels = []

    with path.open(encoding="utf-8", newline="") as csv_file:
        reader = csv.DictReader(csv_file)
        for row in reader:
            label = row["category"].strip()
            if label in SELECTED_LABELS:
                texts.append(row["text"].strip())
                labels.append(label)

    return texts, labels

How are the rows filtered?

  • The two lists keep ticket messages aligned with their intent labels.
  • The dictionary reader uses the CSV header to access text and category values.
  • The membership check discards every category outside SELECTED_LABELS.
  • Replace the current main() function with this expanded version:
def main():
    show_rule_based_obstacle()
    download_data()

    train_texts, train_labels = load_split(TRAIN_FILE)
    test_texts, test_labels = load_split(TEST_FILE)

    print(f"Selected intents: {len(SELECTED_LABELS)}")
    print(f"Training examples: {len(train_texts)}")
    print(f"Test examples: {len(test_texts)}\n")

What does the expanded main function prove?

The function loads the filtered training and test splits independently. The three printed counts confirm that the six selected intents have usable examples in both files.

  • Save ticket_triage.py.
  • Verify the filtered data counts by running the script again:
python ticket_triage.py

What should the counts show?

You should see Selected intents: 6. You should also see nonzero training and test example counts.

Train and compare both models

A most-frequent baseline always chooses the most common training label. It reveals how little a classifier achieves if it ignores every ticket message.

The trained Pipeline converts text into numerical features before passing them to Complement Naive Bayes. Macro F1 gives each selected intent equal weight in the final average.

  • Place the machine learning imports below the existing standard-library imports:
from sklearn.dummy import DummyClassifier
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics import f1_score
from sklearn.naive_bayes import ComplementNB
from sklearn.pipeline import Pipeline

What do these imports provide?

The imports provide the baseline estimator and the text feature extractor. They also provide the trained classifier and the macro F1 calculation.

  • Replace the current main() function and entry point with this baseline training block:
def train_baseline(train_labels, test_size):
    baseline = DummyClassifier(strategy="most_frequent")
    baseline.fit([[0]] * len(train_labels), train_labels)
    return baseline.predict([[0]] * test_size)


def main():
    show_rule_based_obstacle()
    download_data()

    train_texts, train_labels = load_split(TRAIN_FILE)
    test_texts, test_labels = load_split(TEST_FILE)

    print(f"Selected intents: {len(SELECTED_LABELS)}")
    print(f"Training examples: {len(train_texts)}")
    print(f"Test examples: {len(test_texts)}\n")

    baseline_predictions = train_baseline(train_labels, len(test_labels))

    baseline_f1 = f1_score(
        test_labels,
        baseline_predictions,
        average="macro",
        zero_division=0,
    )

    print(f"Most-frequent baseline macro F1: {baseline_f1:.3f}")


if __name__ == "__main__":
    main()

How does the baseline work?

The baseline learns only which label appears most often. It then predicts that label for every test ticket.

Its macro F1 becomes the benchmark for the text-aware model. A useful trained model needs to handle the six intents more evenly.

  • Save ticket_triage.py.
  • Run the script to calculate the baseline benchmark:
python ticket_triage.py

What should the baseline run show?

The terminal prints a line beginning with Most-frequent baseline macro F1:. This score records the performance of a classifier that ignores the ticket text.

  • Place the trained-model function between train_baseline() and main() by adding this code:
def train_model(train_texts, train_labels):
    model = Pipeline(
        [
            ("tfidf", TfidfVectorizer(ngram_range=(1, 2), min_df=2)),
            ("classifier", ComplementNB()),
        ]
    )
    model.fit(train_texts, train_labels)
    return model

How does the trained pipeline learn?

  • The vectorizer learns single words plus two-word combinations from the training messages.
  • The minimum document frequency removes features that appear in fewer than two training messages.
  • The classifier learns how those text features relate to the six intent labels.
  • Find the baseline print line near the bottom of main().
  • Replace that single print line with the trained-model calculation and both score lines:
    model = train_model(train_texts, train_labels)
    predictions = model.predict(test_texts)
    model_f1 = f1_score(
        test_labels,
        predictions,
        average="macro",
        zero_division=0,
    )

    print(f"Most-frequent baseline macro F1: {baseline_f1:.3f}")
    print(f"Trained model macro F1: {model_f1:.3f}")

What does this comparison measure?

Both scores use the same filtered test labels. The comparison shows how much the text features improve on the most-frequent strategy.

Before you run the completed training script, which model do you expect to earn the stronger macro F1 score?

  • Save ticket_triage.py.
  • Train both models and reveal the comparison by running this command:
python ticket_triage.py

What should the completed run show?

You should see the existing keyword-router demonstration followed by the selected-intent count. The terminal also prints the training and test counts.

The final two score lines label the most-frequent baseline and the trained model separately. You now have a measurable machine learning comparison.

That is the turning point. Your project now trains a data-driven classifier and measures it against a deliberately weak baseline.

✔️ Awesome, I've got everything!

Great. Double check that you saved requirements.txt and ticket_triage.py.

ⓧ I'd like to double check the full code

Compare your files with these complete training versions.

scikit-learn==1.9.1
import csv
from pathlib import Path
from urllib.request import urlretrieve

from sklearn.dummy import DummyClassifier
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics import f1_score
from sklearn.naive_bayes import ComplementNB
from sklearn.pipeline import Pipeline

TRAIN_URL = (
    "https://raw.githubusercontent.com/PolyAI-LDN/task-specific-datasets/"
    "master/banking_data/train.csv"
)
TEST_URL = (
    "https://raw.githubusercontent.com/PolyAI-LDN/task-specific-datasets/"
    "master/banking_data/test.csv"
)

DATA_DIR = Path("data")
TRAIN_FILE = DATA_DIR / "train.csv"
TEST_FILE = DATA_DIR / "test.csv"

SELECTED_LABELS = {
    "card_payment_not_recognised",
    "cash_withdrawal_not_recognised",
    "declined_card_payment",
    "declined_cash_withdrawal",
    "failed_transfer",
    "passcode_forgotten",
}


def rule_based_route(ticket):
    text = ticket.lower()
    if "cash" in text or "withdraw" in text or "atm" in text:
        return "cash_withdrawal_not_recognised"
    if "card" in text:
        return "card_payment_not_recognised"
    if "transfer" in text:
        return "failed_transfer"
    if "passcode" in text or "password" in text or "code" in text:
        return "passcode_forgotten"
    return "unknown"


def show_rule_based_obstacle():
    examples = [
        "I do not recognize this card payment",
        "There is a cash withdrawal I did not make",
        "My bank transfer failed",
        "I forgot the code for my banking app",
        "The cash withdrawal was declined",
    ]

    print("Keyword router demonstration")
    for example in examples:
        print(f"- {example}")
        print(f"  predicted: {rule_based_route(example)}")

    print(
        "\nDesigned obstacle: the final message is about a declined withdrawal, "
        "but the keyword rule predicts an unrecognized withdrawal.\n"
    )


def download_data():
    DATA_DIR.mkdir(exist_ok=True)

    if not TRAIN_FILE.exists():
        print("Downloading BANKING77 training data...")
        urlretrieve(TRAIN_URL, TRAIN_FILE)

    if not TEST_FILE.exists():
        print("Downloading BANKING77 test data...")
        urlretrieve(TEST_URL, TEST_FILE)


def load_split(path):
    texts = []
    labels = []

    with path.open(encoding="utf-8", newline="") as csv_file:
        reader = csv.DictReader(csv_file)
        for row in reader:
            label = row["category"].strip()
            if label in SELECTED_LABELS:
                texts.append(row["text"].strip())
                labels.append(label)

    return texts, labels


def train_baseline(train_labels, test_size):
    baseline = DummyClassifier(strategy="most_frequent")
    baseline.fit([[0]] * len(train_labels), train_labels)
    return baseline.predict([[0]] * test_size)


def train_model(train_texts, train_labels):
    model = Pipeline(
        [
            ("tfidf", TfidfVectorizer(ngram_range=(1, 2), min_df=2)),
            ("classifier", ComplementNB()),
        ]
    )
    model.fit(train_texts, train_labels)
    return model


def main():
    show_rule_based_obstacle()
    download_data()

    train_texts, train_labels = load_split(TRAIN_FILE)
    test_texts, test_labels = load_split(TEST_FILE)

    print(f"Selected intents: {len(SELECTED_LABELS)}")
    print(f"Training examples: {len(train_texts)}")
    print(f"Test examples: {len(test_texts)}\n")

    baseline_predictions = train_baseline(train_labels, len(test_labels))

    baseline_f1 = f1_score(
        test_labels,
        baseline_predictions,
        average="macro",
        zero_division=0,
    )

    model = train_model(train_texts, train_labels)
    predictions = model.predict(test_texts)
    model_f1 = f1_score(
        test_labels,
        predictions,
        average="macro",
        zero_division=0,
    )

    print(f"Most-frequent baseline macro F1: {baseline_f1:.3f}")
    print(f"Trained model macro F1: {model_f1:.3f}")


if __name__ == "__main__":
    main()

Your classifier is trained and benchmarked. Next, you will break its performance down by intent and inspect the mistakes behind the score.

Evaluate Model Mistakes

Your trained TF-IDF and Complement Naive Bayes pipeline can outperform the most-frequent baseline. However, one score cannot reveal which support tickets still receive the wrong route.

This step adds a classification report, a confusion matrix, and direct error analysis. These checks turn model performance into evidence you can investigate.

In this step, get ready to:
  • Measure performance for every selected intent.
  • Connect confusion matrix positions to their class labels.
  • Inspect original messages that the model classified incorrectly.
Print per-class performance

Macro F1 gives every selected intent equal weight. A per-class report reveals whether the combined score hides a weak ticket category.

  • Return to ticket_triage.py in Visual Studio Code.
  • Find the current metrics import near the top of the file:
from sklearn.metrics import f1_score

What does this import provide?

The existing f1_score import calculates the macro F1 values. The expanded evaluation needs two additional metrics helpers.

  • Replace the current metrics import with this expanded version:
from sklearn.metrics import classification_report, confusion_matrix, f1_score

What do the new imports add?

  • The classification_report helper calculates per-class precision, recall, F1, and support.
  • The confusion_matrix helper counts every true-label and predicted-label pairing.
  • The existing f1_score helper preserves the baseline comparison.
  • Add the initial evaluate function below train_model by copying this code:
def evaluate(model, test_texts, test_labels, baseline_predictions):
    predictions = model.predict(test_texts)
    class_labels = model.named_steps["classifier"].classes_

    baseline_f1 = f1_score(
        test_labels,
        baseline_predictions,
        average="macro",
        zero_division=0,
    )
    model_f1 = f1_score(
        test_labels,
        predictions,
        average="macro",
        zero_division=0,
    )

    print(f"Most-frequent baseline macro F1: {baseline_f1:.3f}")
    print(f"Trained model macro F1: {model_f1:.3f}")

    print("\nClassification report")
    print(
        classification_report(
            test_labels,
            predictions,
            labels=class_labels,
            zero_division=0,
        )
    )

What does this function do?

  • The model predicts every message in test_texts before the metrics are calculated.
  • The classifier's classes_ value provides one consistent class order.
  • The two f1_score calls preserve the comparison between the baseline and trained model.
  • The final call prints one performance row for every selected intent.
  • In main, delete the existing lines that calculate and print the two macro F1 values.
  • Add the new evaluation call below the model assignment by copying this line:
    evaluate(model, test_texts, test_labels, baseline_predictions)

Why call evaluate here?

The main function already holds the model plus the filtered test data. Passing those values into evaluate keeps all reporting logic together.

  • Save ticket_triage.py.
  • Run the updated evaluation in the terminal from earlier by entering this command:
python ticket_triage.py

What should you see?

You should see the two macro F1 values followed by Classification report. The report contains one row for each selected banking intent.

Your headline score now has class-level detail. You can see which intents contribute strong results or weak results.

Report missing or incomplete?

Confirm that classification_report appears in the metrics import. Check that the evaluate call sits below the model assignment inside main.

Make sure you removed the previous score-calculation block from main. Keeping both versions causes duplicate score output.

Help me fix my evaluation report.

Map prediction errors

A confusion matrix counts how true labels line up with predicted labels. The off-diagonal values expose the exact intent pairs the model mixes up.

  • Add this code at the bottom of evaluate after the classification report:
    print("Class order")
    for index, label in enumerate(class_labels):
        print(f"{index}: {label}")

    print("\nConfusion matrix")
    print(confusion_matrix(test_labels, predictions, labels=class_labels))

How does the class order help?

The numbered list maps every matrix row to an intent. It also maps every matrix column to the same intent order.

Values on the main diagonal represent correct predictions. Off-diagonal values represent tickets sent to a different intent.

  • Save ticket_triage.py.
  • Generate the labeled matrix by running the script again:
python ticket_triage.py

What should you see?

You should see six numbered class labels followed by a six-by-six matrix. The numbers in Class order identify the intent represented by each matrix position.

The matrix gives your aggregate score a map. Its largest off-diagonal values identify the most frequent confusion pairs.

Matrix labels do not line up?

Check that the class-order loop uses class_labels. Confirm that the matrix call also passes labels=class_labels.

Help me align the confusion matrix.

Inspect real prediction mistakes

Matrix counts reveal the size of each confusion pattern. Original ticket messages reveal the language behind those mistakes.

  • Insert the show_mistakes function above evaluate by copying this code:
def show_mistakes(test_texts, test_labels, predictions, limit=5):
    print("\nExample mistakes")
    mistakes_shown = 0

    for text, true_label, predicted_label in zip(
        test_texts, test_labels, predictions
    ):
        if true_label != predicted_label:
            print(f"- text: {text}")
            print(f"  true: {true_label}")
            print(f"  predicted: {predicted_label}")
            mistakes_shown += 1

            if mistakes_shown == limit:
                break

    if mistakes_shown == 0:
        print("No mistakes found in this filtered test split.")

What does this function do?

  • The zip loop keeps each original message beside its true label plus its predicted label.
  • The comparison skips every correct prediction.
  • The limit value stops the output after five mistakes.
  • The final check handles a filtered test split with no mistakes.
  • Add the mistake review at the bottom of evaluate by copying this line:
    show_mistakes(test_texts, test_labels, predictions)

Why reuse these predictions?

The report plus the matrix already evaluate predictions. Reusing that array ensures every displayed mistake comes from the same model run.

  • Save ticket_triage.py.

Before you run the final check, which pair of intents do you expect to appear among the displayed mistakes?

  • Run the completed evaluation from the terminal:
python ticket_triage.py

What should you see?

You should see both macro F1 values plus the per-class report. You should also see a six-by-six confusion matrix followed by up to five mistaken predictions.

Each mistake includes the original text. The next two lines show its true label plus the model's predicted label.

That is the evaluation loop complete. Your model now exposes measurable performance plus concrete failure modes.

Example mistakes do not appear?

Confirm that show_mistakes is defined above evaluate. Check that its call remains indented inside evaluate.

Verify that the call passes test_texts, test_labels, and predictions in that order.

Help me display the model's mistakes.

✔️ Awesome, I've got everything!

Great. Double check that you saved ticket_triage.py after the successful evaluation run.

ⓧ I'd like to double check the full code

Compare your cumulative ticket_triage.py file against this evaluation version.

import csv
from pathlib import Path
from urllib.request import urlretrieve

from sklearn.dummy import DummyClassifier
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics import classification_report, confusion_matrix, f1_score
from sklearn.naive_bayes import ComplementNB
from sklearn.pipeline import Pipeline

TRAIN_URL = (
    "https://raw.githubusercontent.com/PolyAI-LDN/task-specific-datasets/"
    "master/banking_data/train.csv"
)
TEST_URL = (
    "https://raw.githubusercontent.com/PolyAI-LDN/task-specific-datasets/"
    "master/banking_data/test.csv"
)

DATA_DIR = Path("data")
TRAIN_FILE = DATA_DIR / "train.csv"
TEST_FILE = DATA_DIR / "test.csv"

SELECTED_LABELS = {
    "card_payment_not_recognised",
    "cash_withdrawal_not_recognised",
    "declined_card_payment",
    "declined_cash_withdrawal",
    "failed_transfer",
    "passcode_forgotten",
}


def rule_based_route(ticket):
    text = ticket.lower()
    if "cash" in text or "withdraw" in text or "atm" in text:
        return "cash_withdrawal_not_recognised"
    if "card" in text:
        return "card_payment_not_recognised"
    if "transfer" in text:
        return "failed_transfer"
    if "passcode" in text or "password" in text or "code" in text:
        return "passcode_forgotten"
    return "unknown"


def show_rule_based_obstacle():
    examples = [
        "I do not recognize this card payment",
        "There is a cash withdrawal I did not make",
        "My bank transfer failed",
        "I forgot the code for my banking app",
        "The cash withdrawal was declined",
    ]

    print("Keyword router demonstration")
    for example in examples:
        print(f"- {example}")
        print(f"  predicted: {rule_based_route(example)}")

    print(
        "\nDesigned obstacle: the final message is about a declined withdrawal, "
        "but the keyword rule predicts an unrecognized withdrawal.\n"
    )


def download_data():
    DATA_DIR.mkdir(exist_ok=True)

    if not TRAIN_FILE.exists():
        print("Downloading BANKING77 training data...")
        urlretrieve(TRAIN_URL, TRAIN_FILE)

    if not TEST_FILE.exists():
        print("Downloading BANKING77 test data...")
        urlretrieve(TEST_URL, TEST_FILE)


def load_split(path):
    texts = []
    labels = []

    with path.open(encoding="utf-8", newline="") as csv_file:
        reader = csv.DictReader(csv_file)
        for row in reader:
            label = row["category"].strip()
            if label in SELECTED_LABELS:
                texts.append(row["text"].strip())
                labels.append(label)

    return texts, labels


def train_baseline(train_labels, test_size):
    baseline = DummyClassifier(strategy="most_frequent")
    baseline.fit([[0]] * len(train_labels), train_labels)
    return baseline.predict([[0]] * test_size)


def train_model(train_texts, train_labels):
    model = Pipeline(
        [
            ("tfidf", TfidfVectorizer(ngram_range=(1, 2), min_df=2)),
            ("classifier", ComplementNB()),
        ]
    )
    model.fit(train_texts, train_labels)
    return model


def show_mistakes(test_texts, test_labels, predictions, limit=5):
    print("\nExample mistakes")
    mistakes_shown = 0

    for text, true_label, predicted_label in zip(
        test_texts, test_labels, predictions
    ):
        if true_label != predicted_label:
            print(f"- text: {text}")
            print(f"  true: {true_label}")
            print(f"  predicted: {predicted_label}")
            mistakes_shown += 1

            if mistakes_shown == limit:
                break

    if mistakes_shown == 0:
        print("No mistakes found in this filtered test split.")


def evaluate(model, test_texts, test_labels, baseline_predictions):
    predictions = model.predict(test_texts)
    class_labels = model.named_steps["classifier"].classes_

    baseline_f1 = f1_score(
        test_labels,
        baseline_predictions,
        average="macro",
        zero_division=0,
    )
    model_f1 = f1_score(
        test_labels,
        predictions,
        average="macro",
        zero_division=0,
    )

    print(f"Most-frequent baseline macro F1: {baseline_f1:.3f}")
    print(f"Trained model macro F1: {model_f1:.3f}")

    print("\nClassification report")
    print(
        classification_report(
            test_labels,
            predictions,
            labels=class_labels,
            zero_division=0,
        )
    )

    print("Class order")
    for index, label in enumerate(class_labels):
        print(f"{index}: {label}")

    print("\nConfusion matrix")
    print(confusion_matrix(test_labels, predictions, labels=class_labels))

    show_mistakes(test_texts, test_labels, predictions)


def main():
    show_rule_based_obstacle()
    download_data()

    train_texts, train_labels = load_split(TRAIN_FILE)
    test_texts, test_labels = load_split(TEST_FILE)

    print(f"Selected intents: {len(SELECTED_LABELS)}")
    print(f"Training examples: {len(train_texts)}")
    print(f"Test examples: {len(test_texts)}\n")

    baseline_predictions = train_baseline(train_labels, len(test_labels))
    model = train_model(train_texts, train_labels)

    evaluate(model, test_texts, test_labels, baseline_predictions)


if __name__ == "__main__":
    main()

Your classifier now shows where it performs well plus where its decisions break down. Next up, you will turn it into an interactive terminal demo that returns an intent with a probability estimate.

Publish Your GitHub Demo

Your classifier can already show where it performs well. It also exposes the ticket types that it confuses.

This step turns the evaluated model into an interactive terminal demo. You will document the evidence in README.md before publishing three reproducible files to GitHub.

In this step, get ready to:
  • Add an interactive loop that predicts an intent with a probability estimate.
  • Document the project architecture, evaluation evidence, example usage, and limitations.
  • Publish the three project files in a public repository.
Add the interactive ticket demo

The trained classifier can return a probability estimate for each intent. This number measures the model's own output for the message.

How should you describe the number?

Call the displayed value a model probability estimate. The project has not proved that the value is a calibrated confidence score.

  • In ticket_triage.py, find the end of evaluate().
  • Add the prediction helper directly below evaluate() by pasting this code:
def predict_ticket(model, ticket):
    probabilities = model.predict_proba([ticket])[0]
    best_index = max(range(len(probabilities)), key=probabilities.__getitem__)
    class_labels = model.named_steps["classifier"].classes_
    return class_labels[best_index], float(probabilities[best_index])

What does this helper do?

  • The predict_proba() call produces one probability estimate for each of the six intents.
  • The best_index value identifies the highest estimate.
  • The return value pairs the matching intent label with its probability estimate.
  • Save ticket_triage.py.
  • Return to the terminal from earlier.
  • Confirm the existing evaluation still completes by running this command:
python ticket_triage.py

What does this check prove?

The script still downloads or reuses the dataset before training both models. A complete evaluation confirms that the new helper has not broken the existing workflow.

You should still see both macro F1 values, the classification report, the class order, the confusion matrix, and up to five mistakes.

Does the evaluation stop with an error?

Check that predict_ticket() starts at the left edge of the file. An extra indent can place the helper inside evaluate().

Compare each bracket in predict_proba([ticket])[0] with the snippet above.

Help me fix the new prediction helper.

  • Add the interactive input loop directly below predict_ticket() by pasting this code:
def run_demo(model):
    print("\nInteractive support ticket triage")
    print("Type a ticket, or type quit to stop.")

    while True:
        ticket = input("\nTicket: ").strip()
        if ticket.lower() == "quit":
            break
        if not ticket:
            print("Please enter a ticket message.")
            continue

        label, probability = predict_ticket(model, ticket)
        print(f"Predicted intent: {label}")
        print(f"Model probability estimate: {probability:.3f}")

How does the loop work?

  • The loop keeps accepting support messages until the learner enters quit.
  • An empty message triggers a clear prompt without asking the model to classify it.
  • Every valid message displays its predicted intent and model probability estimate.
  • Save ticket_triage.py.
  • Scroll to the bottom of main().
  • Find this final function call:
    evaluate(model, test_texts, test_labels, baseline_predictions)

Where does the demo belong?

The evaluation call finishes the repeatable model checks. The interactive demo belongs immediately after it so every session trains and evaluates the model before accepting a live ticket.

  • Replace that line with these two lines:
    evaluate(model, test_texts, test_labels, baseline_predictions)
    run_demo(model)

What changes in the program flow?

The first line preserves the full evaluation. The second line starts the interactive prompt with the trained model from the same run.

Before you run the program, which intent do you expect for a message about forgetting an app code?

  • Save ticket_triage.py.
  • Start the interactive demo by running this command:
python ticket_triage.py

What happens during this run?

The command runs the keyword demonstration before training and evaluation. It then keeps the terminal open at the ticket prompt.

  • Enter I forgot the code for my banking app at the Ticket: prompt.

You should see Predicted intent: passcode_forgotten followed by a Model probability estimate: value.

  • Enter quit at the next prompt.

Does the prompt fail to appear?

Confirm that run_demo(model) is indented inside main() directly below the evaluation call.

Check that the new function is named run_demo everywhere it appears.

Help me debug the interactive ticket prompt.

✔️ Awesome, I've got everything!

Your interactive demo is working. Double check that you saved ticket_triage.py.

ⓧ I'd like to double check the full code

Compare your complete ticket_triage.py file with this reference.

import csv
from pathlib import Path
from urllib.request import urlretrieve

from sklearn.dummy import DummyClassifier
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics import classification_report, confusion_matrix, f1_score
from sklearn.naive_bayes import ComplementNB
from sklearn.pipeline import Pipeline

TRAIN_URL = (
    "https://raw.githubusercontent.com/PolyAI-LDN/task-specific-datasets/"
    "master/banking_data/train.csv"
)
TEST_URL = (
    "https://raw.githubusercontent.com/PolyAI-LDN/task-specific-datasets/"
    "master/banking_data/test.csv"
)

DATA_DIR = Path("data")
TRAIN_FILE = DATA_DIR / "train.csv"
TEST_FILE = DATA_DIR / "test.csv"

SELECTED_LABELS = {
    "card_payment_not_recognised",
    "cash_withdrawal_not_recognised",
    "declined_card_payment",
    "declined_cash_withdrawal",
    "failed_transfer",
    "passcode_forgotten",
}


def rule_based_route(ticket):
    text = ticket.lower()
    if "cash" in text or "withdraw" in text or "atm" in text:
        return "cash_withdrawal_not_recognised"
    if "card" in text:
        return "card_payment_not_recognised"
    if "transfer" in text:
        return "failed_transfer"
    if "passcode" in text or "password" in text or "code" in text:
        return "passcode_forgotten"
    return "unknown"


def show_rule_based_obstacle():
    examples = [
        "I do not recognize this card payment",
        "There is a cash withdrawal I did not make",
        "My bank transfer failed",
        "I forgot the code for my banking app",
        "The cash withdrawal was declined",
    ]

    print("Keyword router demonstration")
    for example in examples:
        print(f"- {example}")
        print(f"  predicted: {rule_based_route(example)}")

    print(
        "\nDesigned obstacle: the final message is about a declined withdrawal, "
        "but the keyword rule predicts an unrecognized withdrawal.\n"
    )


def download_data():
    DATA_DIR.mkdir(exist_ok=True)

    if not TRAIN_FILE.exists():
        print("Downloading BANKING77 training data...")
        urlretrieve(TRAIN_URL, TRAIN_FILE)

    if not TEST_FILE.exists():
        print("Downloading BANKING77 test data...")
        urlretrieve(TEST_URL, TEST_FILE)


def load_split(path):
    texts = []
    labels = []

    with path.open(encoding="utf-8", newline="") as csv_file:
        reader = csv.DictReader(csv_file)
        for row in reader:
            label = row["category"].strip()
            if label in SELECTED_LABELS:
                texts.append(row["text"].strip())
                labels.append(label)

    return texts, labels


def train_baseline(train_labels, test_size):
    baseline = DummyClassifier(strategy="most_frequent")
    baseline.fit([[0]] * len(train_labels), train_labels)
    return baseline.predict([[0]] * test_size)


def train_model(train_texts, train_labels):
    model = Pipeline(
        [
            ("tfidf", TfidfVectorizer(ngram_range=(1, 2), min_df=2)),
            ("classifier", ComplementNB()),
        ]
    )
    model.fit(train_texts, train_labels)
    return model


def show_mistakes(test_texts, test_labels, predictions, limit=5):
    print("\nExample mistakes")
    mistakes_shown = 0

    for text, true_label, predicted_label in zip(
        test_texts, test_labels, predictions
    ):
        if true_label != predicted_label:
            print(f"- text: {text}")
            print(f"  true: {true_label}")
            print(f"  predicted: {predicted_label}")
            mistakes_shown += 1

            if mistakes_shown == limit:
                break

    if mistakes_shown == 0:
        print("No mistakes found in this filtered test split.")


def evaluate(model, test_texts, test_labels, baseline_predictions):
    predictions = model.predict(test_texts)
    class_labels = model.named_steps["classifier"].classes_

    baseline_f1 = f1_score(
        test_labels,
        baseline_predictions,
        average="macro",
        zero_division=0,
    )
    model_f1 = f1_score(
        test_labels,
        predictions,
        average="macro",
        zero_division=0,
    )

    print(f"Most-frequent baseline macro F1: {baseline_f1:.3f}")
    print(f"Trained model macro F1: {model_f1:.3f}")

    print("\nClassification report")
    print(
        classification_report(
            test_labels,
            predictions,
            labels=class_labels,
            zero_division=0,
        )
    )

    print("Class order")
    for index, label in enumerate(class_labels):
        print(f"{index}: {label}")

    print("\nConfusion matrix")
    print(confusion_matrix(test_labels, predictions, labels=class_labels))

    show_mistakes(test_texts, test_labels, predictions)


def predict_ticket(model, ticket):
    probabilities = model.predict_proba([ticket])[0]
    best_index = max(range(len(probabilities)), key=probabilities.__getitem__)
    class_labels = model.named_steps["classifier"].classes_
    return class_labels[best_index], float(probabilities[best_index])


def run_demo(model):
    print("\nInteractive support ticket triage")
    print("Type a ticket, or type quit to stop.")

    while True:
        ticket = input("\nTicket: ").strip()
        if ticket.lower() == "quit":
            break
        if not ticket:
            print("Please enter a ticket message.")
            continue

        label, probability = predict_ticket(model, ticket)
        print(f"Predicted intent: {label}")
        print(f"Model probability estimate: {probability:.3f}")


def main():
    show_rule_based_obstacle()
    download_data()

    train_texts, train_labels = load_split(TRAIN_FILE)
    test_texts, test_labels = load_split(TEST_FILE)

    print(f"Selected intents: {len(SELECTED_LABELS)}")
    print(f"Training examples: {len(train_texts)}")
    print(f"Test examples: {len(test_texts)}\n")

    baseline_predictions = train_baseline(train_labels, len(test_labels))
    model = train_model(train_texts, train_labels)

    evaluate(model, test_texts, test_labels, baseline_predictions)
    run_demo(model)


if __name__ == "__main__":
    main()
Document the project

A recruiter needs enough context to reproduce the demo and interpret its results. Your README also records the model's limits so the probability estimate is presented honestly.

  • Use the Explorer sidebar in Visual Studio Code to create README.md inside the support-ticket-triage folder.
  • Add the project overview, architecture, and dataset attribution by pasting this first section:
# Intelligent Support Ticket Triage

A beginner AI engineering project that classifies banking support messages into six fine-grained intents. The project compares a deliberately weak baseline with a data-driven text classification pipeline and exposes real prediction mistakes.

## What it demonstrates

- Python project setup and dependency pinning
- A rule-based baseline and its limitations
- TF-IDF text features
- Complement Naive Bayes classification
- Macro F1 and per-class evaluation
- Confusion matrix analysis
- Probability-based interactive predictions
- Honest documentation of model limitations

## Architecture

`Ticket text -> TF-IDF vectorizer -> ComplementNB classifier -> Intent + probability estimate`

## Dataset

This project uses a six-intent subset of BANKING77 from PolyAI. BANKING77 contains online banking queries labeled by intent and is licensed under Creative Commons Attribution 4.0 International.

Source: https://github.com/PolyAI-LDN/task-specific-datasets/tree/master/banking_data

Selected intents:

- `card_payment_not_recognised`
- `cash_withdrawal_not_recognised`
- `declined_card_payment`
- `declined_cash_withdrawal`
- `failed_transfer`
- `passcode_forgotten`

What does this section document?

  • The opening explains the problem and the project evidence.
  • The architecture follows one ticket from text features to the final prediction.
  • The dataset section attributes BANKING77 and names the six supported intents.
  • Save README.md.
  • Confirm the editor shows the project title, the architecture, and all six selected intents.

Does the Markdown look compressed?

Check that each heading starts on its own line. Keep one blank line between headings, paragraphs, and lists.

Help me fix the first part of README.md.

  • Add the Windows run instructions and evaluation section at the end of README.md by pasting this code:

## Run locally on Windows

1. Create and select a Python virtual environment in Visual Studio Code.
2. Install the dependency:

   ```powershell
   py -m pip install -r requirements.txt
   ```

3. Run the project:

   ```powershell
   python ticket_triage.py
   ```

The dataset downloads automatically into the `data` folder.

## Evaluation

The script prints:

- Most-frequent baseline macro F1
- Trained model macro F1
- Precision, recall, F1, and support for each intent
- Confusion matrix
- Up to five misclassified test examples

After running the project, record your measured scores and one recurring error pattern here. Use only values produced by your run.

Why include repeatable evaluation?

The setup commands give another person a clear path from the dependency file to the running program. The evaluation checklist shows that your project compares models and investigates class-specific errors.

  • Save README.md.
  • Confirm the file now shows the Windows setup commands and five evaluation outputs.

Are the commands outside their code blocks?

Keep the opening and closing backtick lines around each PowerShell command. The backticks must sit on separate lines.

Help me repair the command blocks in README.md.

  • Finish README.md by adding the example, limitations, and interview prompts:

## Example

Input:

```text
I forgot the code for my banking app
```

Output includes the predicted intent and the model probability estimate.

## Limitations

- The model recognizes only six BANKING77 intents.
- The displayed probability is not proven to be calibrated confidence.
- The model was evaluated on the supplied BANKING77 test split, not live support traffic.
- Production routing would require broader data, privacy controls, monitoring, and human escalation.

## Interview talking points

- Why the keyword router failed
- Why a baseline is necessary
- Why macro F1 is useful across multiple classes
- What the confusion matrix revealed
- Which messages were misclassified and why
- How a human review fallback could reduce risk

Why end with limitations?

The example gives readers a concrete message to test. The limitations define where the evidence ends and create useful interview talking points.

  • Save README.md.
  • Confirm the final sections show one example, four limitations, and six interview talking points.

Is a section missing?

Compare the heading order against the full reference below. A heading that appears inside a code block usually means a closing backtick line is missing.

Help me find the missing README section.

✔️ Awesome, I've got everything!

Your project story is documented. Double check that you saved README.md.

ⓧ I'd like to double check the full code

Compare your complete README.md file with this reference.

# Intelligent Support Ticket Triage

A beginner AI engineering project that classifies banking support messages into six fine-grained intents. The project compares a deliberately weak baseline with a data-driven text classification pipeline and exposes real prediction mistakes.

## What it demonstrates

- Python project setup and dependency pinning
- A rule-based baseline and its limitations
- TF-IDF text features
- Complement Naive Bayes classification
- Macro F1 and per-class evaluation
- Confusion matrix analysis
- Probability-based interactive predictions
- Honest documentation of model limitations

## Architecture

`Ticket text -> TF-IDF vectorizer -> ComplementNB classifier -> Intent + probability estimate`

## Dataset

This project uses a six-intent subset of BANKING77 from PolyAI. BANKING77 contains online banking queries labeled by intent and is licensed under Creative Commons Attribution 4.0 International.

Source: https://github.com/PolyAI-LDN/task-specific-datasets/tree/master/banking_data

Selected intents:

- `card_payment_not_recognised`
- `cash_withdrawal_not_recognised`
- `declined_card_payment`
- `declined_cash_withdrawal`
- `failed_transfer`
- `passcode_forgotten`

## Run locally on Windows

1. Create and select a Python virtual environment in Visual Studio Code.
2. Install the dependency:

   ```powershell
   py -m pip install -r requirements.txt
   ```

3. Run the project:

   ```powershell
   python ticket_triage.py
   ```

The dataset downloads automatically into the `data` folder.

## Evaluation

The script prints:

- Most-frequent baseline macro F1
- Trained model macro F1
- Precision, recall, F1, and support for each intent
- Confusion matrix
- Up to five misclassified test examples

After running the project, record your measured scores and one recurring error pattern here. Use only values produced by your run.

## Example

Input:

```text
I forgot the code for my banking app
```

Output includes the predicted intent and the model probability estimate.

## Limitations

- The model recognizes only six BANKING77 intents.
- The displayed probability is not proven to be calibrated confidence.
- The model was evaluated on the supplied BANKING77 test split, not live support traffic.
- Production routing would require broader data, privacy controls, monitoring, and human escalation.

## Interview talking points

- Why the keyword router failed
- Why a baseline is necessary
- Why macro F1 is useful across multiple classes
- What the confusion matrix revealed
- Which messages were misclassified and why
- How a human review fallback could reduce risk
Publish the repository

A public repository gives someone else direct access to your runnable code and project explanation. The dataset stays reproducible because the program downloads it when needed.

  • Visit GitHub in your web browser.
  • Sign in to your GitHub account.
  • Select New repository.
  • Enter support-ticket-triage as the repository name.
  • Choose public repository visibility.
  • Click Create repository.

You should see an empty repository named support-ticket-triage with public visibility.

  • Select Add file on the repository page.
  • Click Upload files.
  • Choose ticket_triage.py, requirements.txt, and README.md from the support-ticket-triage folder.
  • Enter Publish support ticket triage demo as the commit message.
  • Click Propose changes.

You should see ticket_triage.py, requirements.txt, and README.md listed on the repository page.

Did the upload fail?

Confirm that you selected the three files from the support-ticket-triage folder. Leave the downloaded data folder on your laptop.

Return to the repository page before selecting Add file again.

Help me troubleshoot my GitHub upload.

Before the final run, do you expect the forgotten banking-app code message to produce an intent without a probability estimate?

  • Return to the terminal from earlier.
  • Run the published program locally with this command:
python ticket_triage.py

What does this final run verify?

This run proves that the code represented in the repository still completes its training and evaluation workflow. The interactive prompt then proves that the final model accepts a live message.

  • Enter I forgot the code for my banking app at the Ticket: prompt.

You should see Predicted intent: passcode_forgotten plus a Model probability estimate: value. The estimate accompanies the intent without claiming guaranteed confidence.

  • Enter quit at the next prompt.
  • Return to the repository page in your browser.
  • Confirm that the repository visibility is public.
  • Confirm that ticket_triage.py, requirements.txt, and README.md are visible.

Does the repository show an older file?

Open the local file in Visual Studio Code first. Confirm that ticket_triage.py contains run_demo(model) and that README.md contains the limitations section.

Upload the corrected file again with Add file and Upload files.

Help me update the published repository files.

Strong finish. Your public repository now presents a runnable classifier with evaluation evidence and clear limitations.

Secret mission

Add a Human Review Route

Your classifier currently routes every prediction automatically, including predictions with weak probability estimates. Add a testable policy that sends low-probability predictions to human review while allowing stronger predictions to continue automatically.

Clean Up Your Resources

Clean Up Your Resources

Your classifier has no running service or ongoing charges. Decide whether to keep the local project and public GitHub repository, pause your work, or delete both copies.

Resources you used:

  • The local support-ticket-triage folder containing ticket_triage.py, requirements.txt, README.md, and hello.py.
  • The isolated .venv environment inside the local project folder.
  • The downloaded BANKING77 training and test files under data/.
  • The public support-ticket-triage GitHub repository.

Keep everything running

No action needed. Choose this if you are still improving the classifier or testing the safety-routing policy.

  • Retain the local support-ticket-triage folder with its code, virtual environment, and downloaded data.
  • Keep the public support-ticket-triage repository available with the documented human-review route.
  • Return to the project in Visual Studio Code whenever you want to test another support message.

Pause - I'll come back to this later

Pausing closes your current editor session while preserving every local file and the public repository.

  • Type quit at the ticket prompt if the interactive demo is still running.
  • Close Visual Studio Code.
  • Leave the local support-ticket-triage folder unchanged.
  • Leave the public support-ticket-triage repository unchanged.

Your classifier uses no background service, so nothing keeps running after you close the editor.

Delete - I don't want to use this again

Deleting both copies permanently removes your classifier, downloaded data, virtual environment, and public repository. Save anything you want to keep before continuing.

  • Type quit at the ticket prompt if the interactive demo is still running.
  • Close Visual Studio Code.
  • Open File Explorer from Windows search.
  • Enter support-ticket-triage in the File Explorer search box.
  • Select the local support-ticket-triage folder in the search results.
  • Press Shift+Delete to remove the folder permanently.
  • Confirm the permanent deletion.

You should no longer see the local support-ticket-triage folder in File Explorer.

The public repository is now the remaining copy of your project.

  • Return to the public support-ticket-triage repository page on GitHub.
  • Open the settings for the repository.
  • Scroll to the controls for permanently deleting the repository.
  • Choose the permanent repository deletion option.
  • Complete the on-screen confirmation.
  • Reload the former repository URL.

You should see that the public repository is no longer available. Both copies of your project are now removed.

Nice Work!

Nice Work!

You made it! Your evaluated six-intent support ticket classifier now powers an interactive demo in a public GitHub repository.

You learned how to:

  • Built a rule-based baseline that routed straightforward tickets. Exposed its limitation with a declined withdrawal that it misclassified as an unrecognized withdrawal.
  • Trained a six-intent classifier on the BANKING77 dataset using TF-IDF features with Complement Naive Bayes. Compared its macro F1 against a most-frequent baseline.
  • Performed error analysis with per-class metrics. Used a confusion matrix to locate intent mix-ups. Inspected real misclassified messages. Published the interactive GitHub demo with reproducible setup notes plus honest limitations.
  • Secret Mission: Added a human review fallback through route_prediction(probability). Set REVIEW_THRESHOLD to 0.55 as a documented project policy. Verified human_review for a low estimate. Verified automatic_routing for a higher estimate.

Ready to quiz yourself?