Build Your First ML Classifier

Train and compare two ML models to classify iris species with scikit-learn.

Introduction

30 Second Summary

Every time your email app filters spam, your phone recognizes a face, or a streaming service suggests what to watch next, a classification model is making predictions behind the scenes.

In this project, you will build a Python script that trains two different machine learning models to predict flower species from physical measurements. This teaches you the universal workflow that applies to every supervised learning problem you will encounter.

What You'll Build

A Python classification script that loads the Iris dataset, visualizes feature patterns, trains a Decision Tree and a K-Nearest Neighbors classifier, and compares their performance on unseen data.

By the end of this project, you'll have:

  • A working Python script (iris_classifier.py) that trains and evaluates two ML classifiers on real data.
  • Saved visualization images showing how features separate different species.
  • A concrete understanding of the end-to-end ML workflow: load, explore, split, train, evaluate, compare.
  • Secret Mission: Add cross-validation and hyperparameter tuning to find the optimal model configuration.

Are there any prerequisites?

You should be comfortable with Python basics (variables, functions, loops) and have a general sense of basic statistics (mean, standard deviation). No prior machine learning experience is needed.

Set Up Your ML Environment

Every machine learning project starts with a clean workspace and the right libraries installed. Before you write any ML code, you need a dedicated folder and an isolated Python environment where your packages live without interfering with other projects.

In this step, you'll create your project folder, set up a virtual environment, and install three core libraries for ML in Python: scikit-learn for building models, pandas for data handling, and matplotlib for visualization.

In this step, get ready to:
  • Create a project folder and virtual environment.
  • Install scikit-learn, pandas, and matplotlib.
  • Create your classifier script with all necessary imports.
Create a project folder and virtual environment

You need a dedicated folder for your ML project, plus an isolated Python environment so your installed packages don't conflict with anything else on your system.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or Command Prompt (Windows) and press Enter to open it.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir ml-classifier
cd ml-classifier

Your terminal is now inside the ml-classifier folder on your Desktop.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type VS Code and press Enter to open it.
  • In VS Code, click File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop, select the ml-classifier folder, then click Open.
  • Open the integrated terminal by pressing Ctrl+` (the backtick key above Tab).

What is a virtual environment?

A virtual environment is an isolated copy of Python that lives inside your project folder. Packages you install there stay contained instead of mixing with your system Python.

This prevents version conflicts between projects. Each project gets exactly the packages it needs.

  • Create the virtual environment by running this command in your VS Code terminal:
python3 -m venv sklearn-env
  • Activate the virtual environment by running the command for your operating system:

macOS

source sklearn-env/bin/activate

Windows

sklearn-env\Scripts\Activate.ps1

You should see (sklearn-env) appear at the start of your terminal prompt. This confirms the virtual environment is active.

Don't see (sklearn-env) in your prompt?

On Windows, if you see an error about running scripts being disabled, run this command first:

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

Then try the activation command again.

On macOS, make sure you typed source before the path. Without it, the activation script won't run.

help me activate my Python virtual environment

  • Install the required libraries by running this command:
pip install numpy pandas matplotlib scikit-learn

What does this install?

  • numpy provides fast math operations on arrays and matrices.
  • pandas gives you DataFrames for organizing and exploring tabular data.
  • matplotlib handles plotting charts and visualizations.
  • scikit-learn provides the ML algorithms, dataset loaders, and evaluation tools.

The installation may take a minute or two as pip downloads each package and its dependencies.

  • Verify the installation by running this command:
python -c "import sklearn; print(sklearn.__version__)"

You should see a version number like 1.9.0 printed in the terminal.

Seeing a ModuleNotFoundError?

Make sure (sklearn-env) appears in your terminal prompt. If it doesn't, re-run the activation command for your OS.

If the activation is showing but the import still fails, try running pip install scikit-learn again and check for error messages in the output.

help me fix my scikit-learn installation error

Create the project file

Now that your environment is ready, you need a Python file to hold your classifier code. You'll add all the imports upfront so they're ready for the steps ahead.

  • In the VS Code Explorer sidebar, click the new file icon.
  • Name the file iris_classifier.py and press Enter.
  • Add the following imports to iris_classifier.py:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix

What do these imports do?

  • numpy and pandas handle data loading and numeric operations.
  • matplotlib.pyplot provides the plotting functions you'll use to visualize your data.
  • load_iris loads the built-in Iris flower dataset directly from scikit-learn.
  • train_test_split divides your data into training and test portions.
  • DecisionTreeClassifier and KNeighborsClassifier are the two ML models you'll train and compare.
  • accuracy_score, classification_report, and confusion_matrix measure how well your models perform.
  • Save the file by pressing Cmd+S (macOS) or Ctrl+S (Windows).
  • In the integrated terminal, run the file to confirm there are no import errors:
python iris_classifier.py

The command should complete with no output and no errors. Silence means success here.

Seeing an import error?

Check that (sklearn-env) still appears in your terminal prompt. If VS Code opened a new terminal, it may not be using your virtual environment.

Re-run the activation command from earlier, then try python iris_classifier.py again.

help me fix import errors in my Python ML project

✔️ Awesome, I've got everything!

Great. Double check you've saved iris_classifier.py and confirmed it runs without errors.

ⓧ I'd like to double check the full code

Here's the complete iris_classifier.py file so far:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix

Your ML environment is ready to go. Next up, you'll load the famous Iris dataset and explore what's inside it.

Load and Explore the Iris Dataset

Your ML environment is set up and ready to go. Now it's time to load some real data and understand what you're working with before training any models.

Every ML project starts with data exploration. You need to know how many samples you have, what features are available, and what you're trying to predict. The Iris dataset is a classic starting point for classification because it's small, clean, and built directly into scikit-learn.

In this step, get ready to:
  • Load the Iris dataset into a pandas DataFrame.
  • Print the dataset's shape, first rows, and statistical summary.
  • Identify the features and target classes you'll use for prediction.
Load the dataset and inspect its structure

The Iris dataset contains measurements of 150 iris flowers across three species. Each row is one flower, and each column is a physical measurement. Your job will be to train a model that predicts the species from those measurements.

  • In iris_classifier.py, add this code below your import lines to load the dataset and print its basic structure:
# --- Step 2: Load and Explore ---
iris = load_iris(as_frame=True)
df = iris.frame

print("Dataset shape:", df.shape)
print("\nFirst 5 rows:")
print(df.head())
print("\nDataset statistics:")
print(df.describe())

What does this code do?

  • load_iris(as_frame=True) loads scikit-learn's built-in Iris dataset and returns it as a pandas DataFrame instead of raw numpy arrays.
  • iris.frame gives you the full DataFrame with both the feature columns and the target column in one table.
  • .shape, .head(), and .describe() are standard pandas methods for quickly understanding the size, content, and distribution of your data.
  • Save iris_classifier.py.
  • Run the script in your terminal:
python iris_classifier.py

You should see Dataset shape: (150, 5) at the top, followed by a table of the first 5 rows with columns for sepal length, sepal width, petal length, petal width, and target.

Seeing a NameError or ImportError?

  • Check that your virtual environment is still active. You should see (sklearn-env) at the start of your terminal prompt.
  • Make sure the new code is below all of the import lines, not between them.

Help me debug why my iris_classifier.py won't run.

Now you know the dataset has 150 rows and 5 columns. Next, explore what you're actually predicting.

  • Add this code below the print(df.describe()) line to explore the target classes:
print("\nTarget classes:", iris.target_names)
print("\nSamples per class:")
print(df["target"].value_counts())

What does this code do?

  • iris.target_names shows the three species the model will learn to predict: setosa, versicolor, and virginica.
  • value_counts() shows how many samples exist for each class. A balanced dataset (equal samples per class) means no species is over-represented during training.
  • Save iris_classifier.py and run it again:
python iris_classifier.py

At the bottom of the output, you should see the three target classes listed, followed by a count of 50 samples for each class.

Seeing a KeyError for 'target'?

Make sure you used iris.frame (not iris.data). The .frame attribute includes the target column, while .data only contains the feature columns.

Help me fix a KeyError in my iris dataset code.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix

# --- Step 2: Load and Explore ---
iris = load_iris(as_frame=True)
df = iris.frame

print("Dataset shape:", df.shape)
print("\nFirst 5 rows:")
print(df.head())
print("\nDataset statistics:")
print(df.describe())
print("\nTarget classes:", iris.target_names)
print("\nSamples per class:")
print(df["target"].value_counts())
Understand the features

Now that you've seen the data printed out, let's make sure you understand what each part means before moving on to visualization.

What are the features and target?

The dataset has 4 features (input measurements) and 1 target (what you're predicting):

  • Features: sepal length, sepal width, petal length, petal width (all measured in centimeters).
  • Target: the species of iris flower. There are three classes: setosa, versicolor, and virginica.
  • Balance: each class has exactly 50 samples, so no species is over-represented. There are no missing values.

What is classification?

Classification is a type of supervised learning where the model learns to assign each input to one of several predefined categories. Here, you're training the model on labeled examples (flowers with known species) so it can predict the species of new, unseen flowers.

You now know exactly what your data looks like. Next, you'll visualize it to build intuition about which features separate the species best.

Visualize the Data

You now have a full picture of the Iris dataset: 150 flowers, 4 measurements each, 3 species to predict. Numbers and tables are useful, but the real question is: which measurements actually help tell the species apart?

Visualizing the data answers that question. In this step, you will plot histograms and a scatter plot to see how well each feature separates the three species. This is a critical step in any ML project because it builds your intuition about what the model will learn.

In this step, get ready to:
  • Create a 2x2 histogram grid showing feature distributions colored by species.
  • Create a scatter plot of the two most informative features.
  • Identify which features best separate the flower species.
Create feature distribution histograms

A histogram shows how values are distributed for a single feature. By coloring each histogram by species, you can see whether that feature's values cluster differently for each flower type.

  • In iris_classifier.py, add the following code at the bottom of the file, below the existing Step 2 code:
# --- Step 3: Visualize ---
features = iris.feature_names
colors = ["blue", "orange", "green"]
target_names = iris.target_names

fig, axes = plt.subplots(2, 2, figsize=(12, 10))

for i, ax in enumerate(axes.flat):
    for target_idx, target_name in enumerate(target_names):
        subset = df[df["target"] == target_idx]
        ax.hist(
            subset[features[i]], alpha=0.6, label=target_name, color=colors[target_idx]
        )
    ax.set_xlabel(features[i])
    ax.set_ylabel("Count")
    ax.legend()

plt.suptitle("Feature Distributions by Species")
plt.tight_layout()
plt.savefig("feature_distributions.png", dpi=150, bbox_inches="tight")
plt.show()

What does this code do?

The first three lines store references you will reuse across both plots: the feature column names, one color per species, and the species labels.

  • plt.subplots(2, 2) creates a 2x2 grid of subplot axes in a single figure.
  • The outer loop iterates over the 4 features. The inner loop draws one histogram per species on the same axis so you can compare their distributions.
  • alpha=0.6 makes the bars semi-transparent so overlapping distributions stay visible.
  • plt.savefig() writes the figure to a PNG file before plt.show() displays it in a popup window.
  • Save iris_classifier.py.
  • Run the script by entering this command in your terminal:
python iris_classifier.py

A plot window opens showing four histograms arranged in a grid. Close the window to let the script continue. You should also see feature_distributions.png saved in your ml-classifier folder.

What should I see?

Look at the petal length and petal width histograms. Setosa (blue) clusters far to the left, clearly separated from the other two species. Versicolor (orange) and virginica (green) overlap a bit, but are still mostly distinct.

Compare that to sepal width, where all three species overlap heavily. This tells you petal measurements will be more useful for classification than sepal width.

Plot window not appearing?

If no window opens but the script does not error, your matplotlib backend may not support GUI display. The PNG file is still saved regardless. Open feature_distributions.png directly from your file explorer to view it.

If you see a ModuleNotFoundError, confirm your virtual environment is activated (you should see (sklearn-env) in your terminal prompt).

Help me debug my matplotlib histogram code

Create a scatter plot of petal measurements

The histograms revealed that petal measurements separate species best. A scatter plot lets you see how two features work together to form natural clusters.

  • Add the following code at the bottom of iris_classifier.py, below the histogram code you just wrote:
fig, ax = plt.subplots(figsize=(8, 6))
for target_idx, target_name in enumerate(target_names):
    subset = df[df["target"] == target_idx]
    ax.scatter(
        subset["petal length (cm)"],
        subset["petal width (cm)"],
        label=target_name,
        alpha=0.7,
        color=colors[target_idx],
    )

ax.set_xlabel("Petal Length (cm)")
ax.set_ylabel("Petal Width (cm)")
ax.set_title("Iris Species by Petal Measurements")
ax.legend()
plt.savefig("petal_scatter.png", dpi=150, bbox_inches="tight")
plt.show()

What does this code do?

  • This creates a single scatter plot with petal length on the x-axis and petal width on the y-axis.
  • Each species gets its own color, so you can visually identify clusters.
  • The plot is saved as petal_scatter.png before the display window opens.

This is exactly what a classification model does: it learns the boundaries between these clusters so it can assign new data points to the correct species.

  • Save iris_classifier.py and run the script again:
python iris_classifier.py

Close the first histogram window when it appears, then a scatter plot window opens. You should see three clear clusters of dots, with setosa completely separated on the lower left.

  • Confirm both PNG files exist by running:
ls *.png

You should see feature_distributions.png and petal_scatter.png in the output.

Why does this matter for machine learning?

The scatter plot shows that setosa is perfectly separable from the other two species using just petal measurements. Versicolor and virginica overlap slightly, which means any model will occasionally confuse those two.

ML models find these boundaries automatically. Your job is to choose the right algorithm and give it good data. The visualization confirms this dataset has real structure for a model to learn.

Only seeing one PNG file?

Make sure you closed the first plot window before expecting the second one. plt.show() blocks script execution until you close the window.

If petal_scatter.png is missing, check that the scatter plot code is below the histogram code, not inside it. Both plt.savefig() calls should be at the top indentation level (no extra indentation).

Help me debug my scatter plot code

✔️ Awesome, I've got everything!

Great. Double check that you have saved iris_classifier.py and both PNG files are in your ml-classifier folder.

ⓧ I'd like to double check the full code

Here is the complete iris_classifier.py file up to this point:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix

# --- Step 2: Load and Explore ---
iris = load_iris(as_frame=True)
df = iris.frame

print("Dataset shape:", df.shape)
print("\nFirst 5 rows:")
print(df.head())
print("\nDataset statistics:")
print(df.describe())
print("\nTarget classes:", iris.target_names)
print("\nSamples per class:")
print(df["target"].value_counts())

# --- Step 3: Visualize ---
features = iris.feature_names
colors = ["blue", "orange", "green"]
target_names = iris.target_names

fig, axes = plt.subplots(2, 2, figsize=(12, 10))

for i, ax in enumerate(axes.flat):
    for target_idx, target_name in enumerate(target_names):
        subset = df[df["target"] == target_idx]
        ax.hist(
            subset[features[i]], alpha=0.6, label=target_name, color=colors[target_idx]
        )
    ax.set_xlabel(features[i])
    ax.set_ylabel("Count")
    ax.legend()

plt.suptitle("Feature Distributions by Species")
plt.tight_layout()
plt.savefig("feature_distributions.png", dpi=150, bbox_inches="tight")
plt.show()

fig, ax = plt.subplots(figsize=(8, 6))
for target_idx, target_name in enumerate(target_names):
    subset = df[df["target"] == target_idx]
    ax.scatter(
        subset["petal length (cm)"],
        subset["petal width (cm)"],
        label=target_name,
        alpha=0.7,
        color=colors[target_idx],
    )

ax.set_xlabel("Petal Length (cm)")
ax.set_ylabel("Petal Width (cm)")
ax.set_title("Iris Species by Petal Measurements")
ax.legend()
plt.savefig("petal_scatter.png", dpi=150, bbox_inches="tight")
plt.show()

Your visualizations confirm that the Iris dataset has real, learnable structure. Next up, you will train your first machine learning model to find those species boundaries automatically.

Train Your First Model (Decision Tree)

Your visualizations revealed that petal measurements create clear boundaries between species. Now it's time to teach a computer to find those boundaries automatically.

In this step, you'll train a Decision Tree classifier on the Iris data and measure how well it predicts species on data it has never seen before.

In this step, get ready to:
  • Split the dataset into training and test sets.
  • Train a Decision Tree classifier on the training data.
  • Evaluate the model's accuracy on unseen test data.
Prepare the data with a train/test split

Before training a model, you need to hold back some data for testing. If you evaluate a model on the same data it learned from, you have no idea whether it actually learned patterns or just memorized the answers.

Why split the data?

Imagine studying for an exam by memorizing the answer key. You'd score 100% on that exact test, but fail any new questions. The same problem applies to ML models.

By holding back 20% of the data as a "test set," you can measure how well the model generalizes to flowers it has never seen during training.

  • In iris_classifier.py, scroll to the bottom of the file (after the scatter plot code) and add the train/test split by copying this code:
# --- Step 4: Train a Decision Tree ---
X = df[features]
y = df["target"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

print(f"\nTraining set: {X_train.shape[0]} samples")
print(f"Test set: {X_test.shape[0]} samples")

What does this code do?

  • X = df[features] separates the input measurements (the 4 feature columns) from the target.
  • y = df["target"] isolates the species labels (what you want to predict).
  • train_test_split randomly shuffles the data and splits it into 80% training (120 samples) and 20% test (30 samples).
  • stratify=y ensures each species is equally represented in both sets.
  • random_state=42 makes the split reproducible. You will get the same results every time you run the script.
  • Save iris_classifier.py and run it in your terminal:
python iris_classifier.py
  • At the bottom of the terminal output (after the visualization windows close), you should see:
Training set: 120 samples
Test set: 30 samples

Not seeing the training/test output?

  • Close any matplotlib plot windows that are still open. The script pauses while plots are displayed.
  • Make sure you added the new code after the scatter plot section, not inside it.
  • Check that your virtual environment is active. You should see (sklearn-env) at the start of your terminal prompt.

Why isn't my train/test split printing output?

Train and evaluate a Decision Tree classifier

A Decision Tree learns a series of if-then rules from the training data. For example: "if petal length is greater than 2.5 cm and less than 4.8 cm, the species is versicolor."

  • Add the model training and evaluation code below the print statements you just added:
dt_model = DecisionTreeClassifier(random_state=42)
dt_model.fit(X_train, y_train)

dt_predictions = dt_model.predict(X_test)

print("\n--- Decision Tree Results ---")
print(f"Accuracy: {accuracy_score(y_test, dt_predictions):.4f}")
print(f"\nConfusion Matrix:\n{confusion_matrix(y_test, dt_predictions)}")
print(
    f"\nClassification Report:\n{classification_report(y_test, dt_predictions, target_names=target_names)}"
)

What does this code do?

  • DecisionTreeClassifier(random_state=42) creates a new Decision Tree model.
  • .fit(X_train, y_train) trains the model by finding rules that map measurements to species.
  • .predict(X_test) asks the trained model to predict species for the 30 test flowers it has never seen.
  • accuracy_score, confusion_matrix, and classification_report compare the predictions to the real answers and show how well the model did.
  • Save iris_classifier.py and run it again:
python iris_classifier.py
  • After the plot windows close, you should see a --- Decision Tree Results --- section with an accuracy score (typically 0.9667 or 1.0000), a confusion matrix, and a classification report.

Seeing a NameError?

  • If you see NameError: name 'X_train' is not defined, make sure the training code comes after the train_test_split block, not before it.
  • If you see NameError: name 'target_names' is not defined, confirm that the variable was defined in the Step 3 visualization section above.

Help me fix my Decision Tree training error.

Understand the output

Your terminal now shows three different ways to measure model performance. Each tells you something different about how well the model predicts.

What do these metrics mean?

  • Accuracy is the percentage of correct predictions out of all 30 test flowers. An accuracy of 0.9667 means the model got 29 out of 30 correct.
  • Confusion matrix is a 3x3 grid showing which species get confused with each other. Each row is the true species and each column is the predicted species. Numbers on the diagonal are correct predictions.
  • Precision tells you how reliable each prediction is. If the model says "versicolor," how often is it actually versicolor?
  • Recall tells you how many of each species the model actually finds. Out of all the real versicolor flowers, how many did the model correctly identify?

Why is accuracy so high?

The Iris dataset is relatively small (150 samples) and the species are well-separated, especially setosa. This makes it a great learning dataset because models perform well quickly.

Real-world datasets are messier. You would typically see lower accuracy and more confusion between similar classes.

✔️ Awesome, I've got everything!

Great. Double check you've saved iris_classifier.py and that your terminal shows the Decision Tree results.

ⓧ I'd like to double check the full code

Here is the complete iris_classifier.py file as it should look at the end of this step:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix

# --- Step 2: Load and Explore ---
iris = load_iris(as_frame=True)
df = iris.frame

print("Dataset shape:", df.shape)
print("\nFirst 5 rows:")
print(df.head())
print("\nDataset statistics:")
print(df.describe())
print("\nTarget classes:", iris.target_names)
print("\nSamples per class:")
print(df["target"].value_counts())

# --- Step 3: Visualize ---
features = iris.feature_names
colors = ["blue", "orange", "green"]
target_names = iris.target_names

fig, axes = plt.subplots(2, 2, figsize=(12, 10))

for i, ax in enumerate(axes.flat):
    for target_idx, target_name in enumerate(target_names):
        subset = df[df["target"] == target_idx]
        ax.hist(
            subset[features[i]], alpha=0.6, label=target_name, color=colors[target_idx]
        )
    ax.set_xlabel(features[i])
    ax.set_ylabel("Count")
    ax.legend()

plt.suptitle("Feature Distributions by Species")
plt.tight_layout()
plt.savefig("feature_distributions.png", dpi=150, bbox_inches="tight")
plt.show()

fig, ax = plt.subplots(figsize=(8, 6))
for target_idx, target_name in enumerate(target_names):
    subset = df[df["target"] == target_idx]
    ax.scatter(
        subset["petal length (cm)"],
        subset["petal width (cm)"],
        label=target_name,
        alpha=0.7,
        color=colors[target_idx],
    )

ax.set_xlabel("Petal Length (cm)")
ax.set_ylabel("Petal Width (cm)")
ax.set_title("Iris Species by Petal Measurements")
ax.legend()
plt.savefig("petal_scatter.png", dpi=150, bbox_inches="tight")
plt.show()

# --- Step 4: Train a Decision Tree ---
X = df[features]
y = df["target"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

print(f"\nTraining set: {X_train.shape[0]} samples")
print(f"Test set: {X_test.shape[0]} samples")

dt_model = DecisionTreeClassifier(random_state=42)
dt_model.fit(X_train, y_train)

dt_predictions = dt_model.predict(X_test)

print("\n--- Decision Tree Results ---")
print(f"Accuracy: {accuracy_score(y_test, dt_predictions):.4f}")
print(f"\nConfusion Matrix:\n{confusion_matrix(y_test, dt_predictions)}")
print(
    f"\nClassification Report:\n{classification_report(y_test, dt_predictions, target_names=target_names)}"
)

Your Decision Tree is trained and evaluated on unseen data. Next up, you'll train a second model using a completely different algorithm and compare their performance.

Compare with K-Nearest Neighbors

Your Decision Tree is trained and evaluated. You have concrete numbers showing how well it predicts iris species on data it has never seen.

One model is good. Two models are better. In this step, you will train a completely different algorithm on the exact same data and compare its performance. This is how data scientists decide which approach to deploy in the real world.

In this step, get ready to:
  • Train a K-Nearest Neighbors classifier on the same training data.
  • Print a side-by-side accuracy comparison of both models.
Train a KNN classifier

The Decision Tree you trained in the previous step learns if-then rules from the data. K-Nearest Neighbors takes a completely different approach. When it sees a new flower, it looks at the 5 most similar flowers in the training data and lets them vote on the species.

  • In iris_classifier.py, scroll to the bottom of your file (after the Decision Tree results section).
  • Add the KNN training and evaluation code below the existing content:
# --- Step 5: Compare with KNN ---
knn_model = KNeighborsClassifier(n_neighbors=5)
knn_model.fit(X_train, y_train)

knn_predictions = knn_model.predict(X_test)

print("\n--- K-Nearest Neighbors Results ---")
print(f"Accuracy: {accuracy_score(y_test, knn_predictions):.4f}")
print(f"\nConfusion Matrix:\n{confusion_matrix(y_test, knn_predictions)}")
print(
    f"\nClassification Report:\n{classification_report(y_test, knn_predictions, target_names=target_names)}"
)

What does this code do?

  • KNeighborsClassifier(n_neighbors=5) creates a KNN model that looks at the 5 closest training samples to make each prediction.
  • .fit(X_train, y_train) stores the training data. KNN does not learn rules during training. It memorizes the data and does the real work at prediction time.
  • .predict(X_test) measures the distance from each test flower to all training flowers, picks the 5 nearest, and returns the majority-vote species.
  • Save iris_classifier.py.
  • Run the script in your terminal by running:
python iris_classifier.py
  • Scroll to the bottom of the terminal output. You should see --- K-Nearest Neighbors Results --- followed by an accuracy score, confusion matrix, and classification report.

Not seeing KNN results?

  • Make sure you added the KNN code below the existing Decision Tree results section, not inside it.
  • Check that KNeighborsClassifier is spelled exactly as shown (capital K, capital N, capital C). Python is case-sensitive.
  • Confirm you are using X_train and y_train (uppercase X, lowercase y) in the .fit() call.

Why isn't my KNN model producing results?

How does KNN measure similarity?

KNN uses Euclidean distance to find the nearest neighbors. This is the straight-line distance formula you may recognize from linear algebra. For two flowers with measurements (a1, a2, a3, a4) and (b1, b2, b3, b4), the distance is the square root of the sum of squared differences across all four features.

The model calculates this distance from the test flower to every single training flower, sorts by closest, and picks the top 5 to vote.

Compare both models

You now have two trained models producing predictions on the same test data. A direct comparison makes it easy to see which algorithm performs better on this specific problem.

  • Add the model comparison code at the very bottom of iris_classifier.py:
# --- Model Comparison ---
print("\n--- Model Comparison ---")
print(f"Decision Tree Accuracy: {accuracy_score(y_test, dt_predictions):.4f}")
print(f"KNN (k=5) Accuracy:     {accuracy_score(y_test, knn_predictions):.4f}")

What does this code do?

This prints both accuracy scores side by side so you can immediately see which model performed better on the held-out test set. The :.4f format shows the accuracy to 4 decimal places for a precise comparison.

  • Save iris_classifier.py and run the script again:
python iris_classifier.py
  • Scroll to the very bottom of the output. You should see --- Model Comparison --- followed by two accuracy scores printed on separate lines.

Seeing a NameError?

  • If you see NameError: name 'dt_predictions' is not defined, make sure the Decision Tree code from Step 4 is still intact above your new code.
  • If you see NameError: name 'knn_predictions' is not defined, confirm the KNN training code from the previous substep appears above the model comparison section.

Help me fix my model comparison code.

Why do the two models get different results?

The Decision Tree learns explicit rules from the training data. Think of it as a flowchart: if petal length is greater than 2.5 cm and petal width is less than 1.7 cm, predict versicolor. It splits the feature space into regions using thresholds.

KNN takes a completely different approach. It does not learn rules at all. Instead, it stores all training flowers and classifies a new flower by finding the 5 most similar ones (by Euclidean distance) and letting them vote. Same data, different strategy, different mistakes.

This is why comparing multiple algorithms matters. No single algorithm is best for every dataset.

✔️ Awesome, I've got everything!

Great. Double check you have saved iris_classifier.py.

ⓧ I'd like to double check the full code

Here is the complete iris_classifier.py file as it should look at the end of this step:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix

# --- Step 2: Load and Explore ---
iris = load_iris(as_frame=True)
df = iris.frame

print("Dataset shape:", df.shape)
print("\nFirst 5 rows:")
print(df.head())
print("\nDataset statistics:")
print(df.describe())
print("\nTarget classes:", iris.target_names)
print("\nSamples per class:")
print(df["target"].value_counts())

# --- Step 3: Visualize ---
features = iris.feature_names
colors = ["blue", "orange", "green"]
target_names = iris.target_names

fig, axes = plt.subplots(2, 2, figsize=(12, 10))

for i, ax in enumerate(axes.flat):
    for target_idx, target_name in enumerate(target_names):
        subset = df[df["target"] == target_idx]
        ax.hist(
            subset[features[i]], alpha=0.6, label=target_name, color=colors[target_idx]
        )
    ax.set_xlabel(features[i])
    ax.set_ylabel("Count")
    ax.legend()

plt.suptitle("Feature Distributions by Species")
plt.tight_layout()
plt.savefig("feature_distributions.png", dpi=150, bbox_inches="tight")
plt.show()

fig, ax = plt.subplots(figsize=(8, 6))
for target_idx, target_name in enumerate(target_names):
    subset = df[df["target"] == target_idx]
    ax.scatter(
        subset["petal length (cm)"],
        subset["petal width (cm)"],
        label=target_name,
        alpha=0.7,
        color=colors[target_idx],
    )

ax.set_xlabel("Petal Length (cm)")
ax.set_ylabel("Petal Width (cm)")
ax.set_title("Iris Species by Petal Measurements")
ax.legend()
plt.savefig("petal_scatter.png", dpi=150, bbox_inches="tight")
plt.show()

# --- Step 4: Train a Decision Tree ---
X = df[features]
y = df["target"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

print(f"\nTraining set: {X_train.shape[0]} samples")
print(f"Test set: {X_test.shape[0]} samples")

dt_model = DecisionTreeClassifier(random_state=42)
dt_model.fit(X_train, y_train)

dt_predictions = dt_model.predict(X_test)

print("\n--- Decision Tree Results ---")
print(f"Accuracy: {accuracy_score(y_test, dt_predictions):.4f}")
print(f"\nConfusion Matrix:\n{confusion_matrix(y_test, dt_predictions)}")
print(
    f"\nClassification Report:\n{classification_report(y_test, dt_predictions, target_names=target_names)}"
)

# --- Step 5: Compare with KNN ---
knn_model = KNeighborsClassifier(n_neighbors=5)
knn_model.fit(X_train, y_train)

knn_predictions = knn_model.predict(X_test)

print("\n--- K-Nearest Neighbors Results ---")
print(f"Accuracy: {accuracy_score(y_test, knn_predictions):.4f}")
print(f"\nConfusion Matrix:\n{confusion_matrix(y_test, knn_predictions)}")
print(
    f"\nClassification Report:\n{classification_report(y_test, knn_predictions, target_names=target_names)}"
)

# --- Model Comparison ---
print("\n--- Model Comparison ---")
print(f"Decision Tree Accuracy: {accuracy_score(y_test, dt_predictions):.4f}")
print(f"KNN (k=5) Accuracy:     {accuracy_score(y_test, knn_predictions):.4f}")

You have now completed the full ML workflow: load, explore, visualize, split, train, evaluate, and compare. You have two models with concrete accuracy numbers and you understand why they approach the same problem differently. Next up, the Secret Mission takes this further with cross-validation and hyperparameter tuning.

Secret mission

Cross-Validation and Hyperparameter Tuning

Your model accuracy came from a single train/test split. What if that split was lucky? Use cross-validation to get a more reliable estimate, then systematically find the best value of k for KNN.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

This project runs entirely locally, so there are no ongoing costs.

Resources you used:

  • The ml-classifier/ project folder on your Desktop (contains iris_classifier.py and all generated PNG images).
  • The sklearn-env/ virtual environment folder inside your project folder.

Keep everything running

No action needed. Choose this if you are still actively building or want to experiment with different models.

  • Your virtual environment stays activated and ready to use.
  • Your PNG visualizations and script remain on your Desktop for reference.

Pause - I'll come back to this later

Deactivate the virtual environment to free up your terminal session. Your files and folders stay on disk so you can return anytime.

  • Deactivate the virtual environment by running:
deactivate
  • When you come back, navigate to your project folder and reactivate with source sklearn-env/bin/activate (macOS/Linux) or sklearn-env\Scripts\activate (Windows).

Delete - I don't want to use this again

Remove all project resources and start fresh. This deletes your script, visualizations, and virtual environment permanently.

  • Deactivate the virtual environment first by running:
deactivate
  • Navigate to your Desktop by running:
cd ~/Desktop
  • Delete the entire project folder (macOS/Linux) by running:
rm -rf ml-classifier
  • On Windows, run this instead:
rmdir /s /q ml-classifier
  • This removes iris_classifier.py, all PNG image files, and the sklearn-env/ virtual environment in one step.

That's a Wrap!

That's a Wrap!

Nice work! You just built a complete machine learning classification pipeline from scratch using scikit-learn.

You've learned how to:

  • Load, explore, and visualize a real dataset to build intuition about which features separate classes, then use that understanding to guide model selection.
  • Train a Decision Tree and a K-Nearest Neighbors classifier, then evaluate both using accuracy scores, confusion matrices, and classification reports.
  • Apply the train/test split pattern to measure how a model generalizes to unseen data, preventing you from confusing memorization with real learning.
  • Secret Mission: Used cross-validation for robust evaluation and ran hyperparameter tuning to find the optimal k value for KNN.

Ready to quiz yourself?