Real-Time AI Network Intrusion Detection

Build an ensemble ML system that detects and explains network threats in real time.

Introduction

30 Second Summary

Security teams drown in thousands of alerts every day. Most are false alarms, and the real attacks hide in the noise.

In this project, you will build an AI-powered network intrusion detection system that uses an ensemble of machine learning models to flag threats with dramatically fewer false positives. You will then add SHAP explainability to every alert and display it all on a live Streamlit dashboard.

What You'll Build

You will generate simulated network attacks, watch your AI ensemble flag them in real-time, and click any alert to see a visual breakdown of exactly why it was flagged.

By the end of this project, you'll have:

  • A running ensemble detector where you feed network traffic data through the system and instantly see which packets are flagged as threats, with a confidence score for each detection.
  • A SHAP explanation panel where you click any flagged alert and see exactly which network features (packet size, port entropy, connection frequency) pushed the AI toward its decision.
  • A model comparison dashboard showing single-model vs ensemble detection rates side-by-side, proving how the ensemble crushes false positives in real numbers.
  • Secret Mission: Add an autoencoder layer for zero-day threat detection that catches novel attacks your ensemble has never seen.

Are there any prerequisites?

You should have Python 3.12 or higher installed, along with VS Code. Familiarity with Python and basic ML concepts (training a model, evaluation metrics) will help you move through the project smoothly.

Before We Start

Before you can build an AI-powered intrusion detection system, you need a clean Python environment with the right libraries installed.

In this step, you will create a dedicated project folder, set up a virtual environment, and install the ML, explainability, and visualization libraries your system needs.

In this step, get ready to:
  • Create a project directory and Python virtual environment.
  • Install all ML, explainability, and visualization dependencies.
  • Verify Streamlit launches successfully in your browser.
Create your project directory and virtual environment
  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or PowerShell (Windows) and press Enter to open it.
  • Move to your Desktop and create your project folder by running these commands:
cd ~/Desktop
mkdir network-ids
cd network-ids
  • Create a Python virtual environment inside the project folder by running:
python3 -m venv .venv
  • Confirm the virtual environment was created by running:
ls

You should see a .venv folder in the output.

  • Activate the virtual environment:

macOS

source .venv/bin/activate

Windows

.venv\Scripts\activate

You should see (.venv) appear at the start of your terminal prompt. This confirms you are working inside the virtual environment.

Don't see (.venv) in your prompt?

On Windows, you may need to allow script execution first. Run Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned in PowerShell, then try activating again.

On macOS, make sure you used source before the path. Running the path alone will not work.

Help me activate my Python virtual environment.

  • Install all project dependencies by running:
pip install scikit-learn==1.9.0 shap==0.52.0 streamlit==1.59.1 pandas==3.0.3 numpy==2.5.1 plotly==6.9.0

What are these libraries?

  • scikit-learn provides the anomaly detection models (Isolation Forest, One-Class SVM, Local Outlier Factor).
  • shap computes feature-level explanations for each detection alert.
  • streamlit builds the live dashboard where you visualize threats.
  • pandas and numpy handle data manipulation and numerical operations.
  • plotly renders interactive charts for model comparison and SHAP explanations.

The install takes a minute or two. You should see Successfully installed followed by all package names at the end of the output.

Install failing or timing out?

Make sure (.venv) is visible in your prompt. Installing without the virtual environment active can cause permission errors.

If you see errors about incompatible Python versions, confirm you are running Python 3.12 or higher with python3 --version. The shap and numpy packages require Python 3.12+.

Help me fix my pip install errors.

Verify your installations

A successful pip install does not guarantee every library loads correctly. A quick import check catches issues like missing system dependencies before you start building.

  • Run this command to import all libraries and print their versions:
python3 -c "
import sklearn, shap, streamlit, pandas, numpy, plotly
print(f'scikit-learn: {sklearn.__version__}')
print(f'shap: {shap.__version__}')
print(f'streamlit: {streamlit.__version__}')
print(f'pandas: {pandas.__version__}')
print(f'numpy: {numpy.__version__}')
print(f'plotly: {plotly.__version__}')
"

You should see all six library names printed with their version numbers. No errors or tracebacks.

Seeing a ModuleNotFoundError?

This usually means the python3 command is pointing to a different Python installation than the one your virtual environment uses. Make sure (.venv) is active in your prompt and re-run the command.

If a single library fails to import, try reinstalling just that package. For example: pip install --force-reinstall shap==0.52.0.

Help me debug my import error.

Confirm Streamlit launches

Streamlit is the framework that powers your dashboard. Confirming it launches now saves you from debugging environment issues later when your app is more complex.

  • Launch the Streamlit sample app by running:

Before you run this, what do you think will happen in your terminal and browser?

streamlit hello

What does this command do?

This command starts a local web server and automatically opens your default browser. Your terminal will show a local URL (http://localhost:8501) and remain occupied while the server runs.

You should see a welcome page titled Welcome to Streamlit with links to demos in the sidebar.

  • Confirm your browser opened a page at localhost:8501 with the Streamlit welcome page.
  • Return to your terminal and press Ctrl+C to stop the Streamlit server.

Browser did not open?

If the browser did not open automatically, copy the URL from your terminal output (http://localhost:8501) and paste it into your browser manually.

If you see a port-in-use error, another process is using port 8501. Close other Streamlit sessions or specify a different port with streamlit hello --server.port 8502.

Help me get Streamlit running.

Your environment is locked and loaded. Next up, you will generate synthetic network traffic and train your first anomaly detector.

Set Up the Project and Install Dependencies

Your environment is ready with all the ML libraries installed. Now you need data to train your detection system on.

Real network traffic logs are sensitive and hard to get. Instead, you will build a synthetic data generator that produces realistic network flow records with known attack patterns baked in. This gives you a labeled dataset where you control the signal.

In this step, get ready to:
  • Open your project in VS Code and create the data generator file.
  • Generate synthetic normal network traffic with realistic feature distributions.
  • Add five attack patterns and verify the full dataset output.
Open the project in VS Code

Your project folder already has a virtual environment with all dependencies installed. Open it in VS Code so you can start writing code.

  • In your terminal (where you activated the virtual environment), open your project folder in VS Code by running:
code .

What does this command do?

The code . command launches VS Code and opens the current directory as a workspace. The dot (.) means the folder you are currently inside.

  • Confirm VS Code opens with your project folder name visible in the Explorer sidebar on the left.

VS Code did not open?

If you see a "command not found" error, you need to add the code command to your PATH. Open VS Code manually, press Cmd+Shift+P (macOS) or Ctrl+Shift+P (Windows), then type Shell Command: Install 'code' command in PATH and select it.

Alternatively, open VS Code from your applications, then use File > Open Folder... and select your project directory.

Help me open my project folder in VS Code.

  • Open the integrated terminal inside VS Code by pressing Ctrl+` (backtick key, same on macOS and Windows).
  • Make sure your virtual environment is still active. You should see (venv) or your environment name at the start of the terminal prompt.
Build the normal traffic generator

Network traffic has measurable features like packet size, flow duration, and connection frequency. Normal traffic follows predictable statistical distributions. Your generator will produce 5000 samples that look like typical network activity.

  • Create a new file called data_generator.py in your project folder. In VS Code, click File > New File, then press Cmd+S (macOS) or Ctrl+S (Windows) and save it as data_generator.py.
  • Add the imports and the normal traffic generation function by pasting this code:
import numpy as np
import pandas as pd


def generate_normal_traffic(n_samples=5000, random_state=42):
    rng = np.random.default_rng(random_state)
    data = {
        "packet_size": rng.normal(512, 128, n_samples).clip(64, 1500),
        "flow_duration": rng.exponential(2.0, n_samples).clip(0.01, 30),
        "bytes_per_second": rng.normal(50000, 15000, n_samples).clip(100, 200000),
        "packets_per_second": rng.normal(100, 30, n_samples).clip(1, 500),
        "port_entropy": rng.normal(3.5, 0.5, n_samples).clip(0, 6),
        "connection_count": rng.poisson(10, n_samples).clip(1, 100),
        "protocol_flag": rng.choice([0, 1, 2], n_samples, p=[0.6, 0.3, 0.1]),
    }
    df = pd.DataFrame(data)
    df["label"] = 0
    df["attack_type"] = "normal"
    return df

What does this code do?

Each feature models a real network measurement. The clip() calls keep values within physically realistic bounds (packets cannot be negative or exceed 1500 bytes on standard Ethernet).

  • Normal distributions model features like packet_size that cluster around a typical value.
  • Exponential distributions model flow_duration because most connections are short with occasional long ones.
  • Poisson distributions model connection_count because connections arrive as discrete events.
  • Every sample gets label = 0 (normal) and attack_type = "normal" so you can evaluate your detector later.
  • Save data_generator.py.
  • Test that the function works by running this quick import check in your terminal:
python -c "from data_generator import generate_normal_traffic; df = generate_normal_traffic(10); print(df.shape); print(df.columns.tolist())"

You should see (10, 9) printed, followed by the list of all 9 column names including the 7 features plus label and attack_type.

Seeing a ModuleNotFoundError?

If you see an error about a missing module like numpy or pandas, your virtual environment may not be active in the VS Code terminal. Look for (venv) at the start of your prompt. If it is missing, activate it again with source venv/bin/activate (macOS) or venv\Scripts\activate (Windows).

Help me fix my import error in data_generator.py.

Add attack patterns and assemble the full dataset

Attacks look different from normal traffic in specific, measurable ways. A port scan sends tiny packets to many ports. Data exfiltration sends large packets over long connections. Your generator encodes these signatures so the detector has clear patterns to learn from.

  • In data_generator.py, add the generate_attack_traffic function below generate_normal_traffic. Start with the function setup and the port scan attack type:
def generate_attack_traffic(n_samples=500, random_state=42):
    rng = np.random.default_rng(random_state)
    attacks = []
    samples_per_type = n_samples // 5

    # Port scan: high port entropy, many connections, small packets
    port_scan = pd.DataFrame({
        "packet_size": rng.normal(64, 10, samples_per_type).clip(40, 128),
        "flow_duration": rng.exponential(0.1, samples_per_type).clip(0.001, 1),
        "bytes_per_second": rng.normal(5000, 2000, samples_per_type).clip(100, 20000),
        "packets_per_second": rng.normal(500, 100, samples_per_type).clip(200, 1000),
        "port_entropy": rng.normal(5.8, 0.2, samples_per_type).clip(5, 6),
        "connection_count": rng.poisson(80, samples_per_type).clip(50, 200),
        "protocol_flag": rng.choice([0, 1, 2], samples_per_type, p=[0.8, 0.1, 0.1]),
    })
    port_scan["label"] = 1
    port_scan["attack_type"] = "port_scan"
    attacks.append(port_scan)

How does a port scan signature differ from normal?

Port scans have extremely high port_entropy (5.8 vs normal 3.5) because they probe many different ports. They also send tiny packets (64 bytes vs normal 512) at very high rates. These statistical differences are exactly what your ML models will learn to detect.

  • Continue inside the same generate_attack_traffic function. Add the brute force attack type directly below the port scan block:
    # Brute force: many connections, moderate packets, repeated patterns
    brute_force = pd.DataFrame({
        "packet_size": rng.normal(256, 30, samples_per_type).clip(128, 512),
        "flow_duration": rng.normal(0.5, 0.1, samples_per_type).clip(0.1, 2),
        "bytes_per_second": rng.normal(30000, 5000, samples_per_type).clip(10000, 60000),
        "packets_per_second": rng.normal(400, 50, samples_per_type).clip(200, 600),
        "port_entropy": rng.normal(0.5, 0.2, samples_per_type).clip(0, 1.5),
        "connection_count": rng.poisson(60, samples_per_type).clip(30, 150),
        "protocol_flag": rng.choice([0, 1, 2], samples_per_type, p=[0.9, 0.05, 0.05]),
    })
    brute_force["label"] = 1
    brute_force["attack_type"] = "brute_force"
    attacks.append(brute_force)

What makes brute force detectable?

Brute force attacks have very low port_entropy (0.5) because they hammer the same port repeatedly. They also show high connection_count (60 vs normal 10) from repeated login attempts. The opposite entropy signature from port scans makes them distinguishable from each other.

  • Still inside generate_attack_traffic, add the data exfiltration and DoS flood attack types below the brute force block:
    # Data exfiltration: large packets, high bytes/sec, long flows
    exfil = pd.DataFrame({
        "packet_size": rng.normal(1400, 50, samples_per_type).clip(1200, 1500),
        "flow_duration": rng.normal(20, 5, samples_per_type).clip(10, 60),
        "bytes_per_second": rng.normal(180000, 20000, samples_per_type).clip(140000, 300000),
        "packets_per_second": rng.normal(150, 20, samples_per_type).clip(80, 300),
        "port_entropy": rng.normal(1.0, 0.3, samples_per_type).clip(0, 2),
        "connection_count": rng.poisson(3, samples_per_type).clip(1, 10),
        "protocol_flag": rng.choice([0, 1, 2], samples_per_type, p=[0.3, 0.6, 0.1]),
    })
    exfil["label"] = 1
    exfil["attack_type"] = "data_exfiltration"
    attacks.append(exfil)

    # DoS flood: tiny packets, extreme packets/sec
    dos = pd.DataFrame({
        "packet_size": rng.normal(64, 5, samples_per_type).clip(40, 100),
        "flow_duration": rng.exponential(0.05, samples_per_type).clip(0.001, 0.5),
        "bytes_per_second": rng.normal(10000, 3000, samples_per_type).clip(5000, 30000),
        "packets_per_second": rng.normal(900, 50, samples_per_type).clip(700, 1000),
        "port_entropy": rng.normal(0.3, 0.1, samples_per_type).clip(0, 1),
        "connection_count": rng.poisson(5, samples_per_type).clip(1, 20),
        "protocol_flag": rng.choice([0, 1, 2], samples_per_type, p=[0.7, 0.2, 0.1]),
    })
    dos["label"] = 1
    dos["attack_type"] = "dos_flood"
    attacks.append(dos)

What makes exfiltration and DoS detectable?

Data exfiltration pushes packet_size near the maximum (1400 bytes) with long flow durations. Someone is sending a lot of data out. DoS floods blast tiny packets at extreme rates (900 packets/sec vs normal 100). Both patterns are statistically far from the normal distribution your model will learn.

  • Add the final attack type (lateral movement), the return statement, the generate_dataset assembly function, and the if __name__ block at the bottom of the file:
    # Lateral movement: normal-ish but unusual connection patterns
    lateral = pd.DataFrame({
        "packet_size": rng.normal(512, 100, samples_per_type).clip(200, 1000),
        "flow_duration": rng.normal(5, 2, samples_per_type).clip(1, 15),
        "bytes_per_second": rng.normal(60000, 10000, samples_per_type).clip(30000, 100000),
        "packets_per_second": rng.normal(120, 20, samples_per_type).clip(50, 200),
        "port_entropy": rng.normal(4.8, 0.3, samples_per_type).clip(4, 5.5),
        "connection_count": rng.poisson(40, samples_per_type).clip(20, 80),
        "protocol_flag": rng.choice([0, 1, 2], samples_per_type, p=[0.4, 0.4, 0.2]),
    })
    lateral["label"] = 1
    lateral["attack_type"] = "lateral_movement"
    attacks.append(lateral)

    return pd.concat(attacks, ignore_index=True)


def generate_dataset(n_normal=5000, n_attack=500, random_state=42):
    normal = generate_normal_traffic(n_normal, random_state)
    attacks = generate_attack_traffic(n_attack, random_state)
    dataset = pd.concat([normal, attacks], ignore_index=True)
    dataset = dataset.sample(frac=1, random_state=random_state).reset_index(drop=True)
    return dataset


if __name__ == "__main__":
    df = generate_dataset()
    print(f"Dataset shape: {df.shape}")
    print(f"Normal: {(df['label'] == 0).sum()}, Attacks: {(df['label'] == 1).sum()}")
    print(f"Attack types: {df[df['label'] == 1]['attack_type'].value_counts().to_dict()}")

What does generate_dataset do?

The generate_dataset function combines normal and attack traffic into one DataFrame, then shuffles it randomly. This simulates how a real network capture looks. Attacks are mixed in with normal packets, and your detector must find them.

Lateral movement is the hardest attack to detect because its features overlap with normal traffic. The key difference is elevated port_entropy (4.8 vs 3.5) and high connection_count (40 vs 10). These subtle shifts test how sensitive your model is.

  • Save data_generator.py.

Before you run the full generator, what do you think the dataset shape will be? Consider: 5000 normal samples plus 500 attack samples, with 7 features plus 2 label columns.

  • Run the data generator to verify everything works:
python data_generator.py

You should see this output:

Dataset shape: (5500, 9)
Normal: 5000, Attacks: 500
Attack types: {'port_scan': 100, 'brute_force': 100, 'data_exfiltration': 100, 'dos_flood': 100, 'lateral_movement': 100}

5500 total samples (5000 normal + 500 attacks), 9 columns (7 features + label + attack_type), and 100 samples for each of the 5 attack types. Your dataset is balanced and ready for training.

Seeing a SyntaxError or IndentationError?

The generate_attack_traffic function is one long function with all five attack blocks indented inside it. Make sure every line from port_scan through return pd.concat(attacks, ignore_index=True) is indented at 4 spaces inside the function body.

Help me fix a SyntaxError in my data_generator.py file.

✔️ Awesome, I've got everything!

Great. Double check you have saved data_generator.py and that running python data_generator.py prints the dataset shape as (5500, 9).

ⓧ I'd like to double check the full code

Your data_generator.py should contain these sections in order. Scroll through your file and confirm each one is present:

  • Imports (numpy and pandas).
  • The generate_normal_traffic function creating 7 features, setting label to 0, and returning a DataFrame.
  • The generate_attack_traffic function with 5 attack DataFrames (port_scan, brute_force, exfil, dos, lateral), each setting label to 1 and its own attack_type string, ending with return pd.concat(attacks, ignore_index=True).
  • The generate_dataset function combining both, shuffling, and returning the full dataset.
  • The if __name__ == "__main__" block that prints shape, counts, and attack type breakdown.

You now have a complete data pipeline that produces labeled network traffic on demand. Next up, you will train a single anomaly detection model on this data and measure how well it separates attacks from normal traffic.

Train a Single Anomaly Detector

Your synthetic dataset is ready with 5500 labeled samples. Now you need a model that can look at a network packet's features and decide: normal traffic, or threat?

You will train a single Isolation Forest model as your baseline detector. Isolation Forest works by randomly partitioning data and measuring how quickly each sample gets isolated. Anomalies are isolated faster because they sit far from normal data. This gives you a working detector, but you will see its limitations firsthand.

In this step, get ready to:
  • Build training and evaluation functions for an Isolation Forest detector.
  • Add ensemble model functions that you will test in the next step.
  • Run the detector and observe the single model's false positive rate.
Create the detector file with single-model functions

Your detector needs to do three things: train a model on scaled features, convert the model's raw predictions into binary labels, and evaluate those predictions against ground truth.

  • Create a new file called detector.py in your project folder.
  • Add the imports, feature column definitions, and single-model functions by pasting this code:
import numpy as np
import pandas as pd
from sklearn.ensemble import IsolationForest
from sklearn.svm import OneClassSVM
from sklearn.neighbors import LocalOutlierFactor
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.preprocessing import StandardScaler
import joblib


FEATURE_COLS = [
    "packet_size", "flow_duration", "bytes_per_second",
    "packets_per_second", "port_entropy", "connection_count", "protocol_flag"
]


def train_single_model(X_train):
    model = IsolationForest(
        n_estimators=100,
        contamination=0.1,
        random_state=42
    )
    model.fit(X_train)
    return model


def predict_single(model, X):
    preds = model.predict(X)
    return np.where(preds == -1, 1, 0)

What does this code do?

FEATURE_COLS lists the 7 numeric features your models will train on. This matches the features your data generator creates.

train_single_model() creates an Isolation Forest with 100 decision trees. The contamination=0.1 tells the model to expect roughly 10% of samples to be anomalies.

predict_single() converts the model's native output (where -1 means anomaly and 1 means normal) into our convention (1 means anomaly, 0 means normal).

  • Below predict_single, add the evaluation function that computes precision, recall, F1, and false positive rate:
def evaluate_model(y_true, y_pred, model_name="Model"):
    cm = confusion_matrix(y_true, y_pred)
    tn, fp, fn, tp = cm.ravel()
    fpr = fp / (fp + tn) if (fp + tn) > 0 else 0
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

    metrics = {
        "model": model_name,
        "precision": round(precision, 4),
        "recall": round(recall, 4),
        "f1": round(f1, 4),
        "fpr": round(fpr, 4),
    }
    return metrics

What does this code do?

evaluate_model() unpacks the confusion matrix into true negatives, false positives, false negatives, and true positives. From these it computes four metrics.

The false positive rate (FPR) is the metric to watch. It tells you what percentage of normal traffic gets incorrectly flagged as a threat. A high FPR means your security team drowns in false alarms.

  • Verify your functions import correctly by running this command:
python -c "from detector import train_single_model, predict_single, evaluate_model; print('Single-model functions ready')"

You should see Single-model functions ready printed with no errors.

Seeing an ImportError?

Make sure you are running the command from inside your project folder (the one containing detector.py). Also confirm your virtual environment is active.

help me fix this ImportError when importing from detector.py

Add ensemble functions for comparison

Your single Isolation Forest will serve as the baseline. To see how much an ensemble improves things, you will also add functions that train three different models and combine their votes. These functions get called in the main script so you can compare results side by side.

  • Below evaluate_model, add the ensemble training function:
def train_ensemble(X_train):
    iso_forest = IsolationForest(
        n_estimators=100,
        contamination=0.1,
        random_state=42
    )
    iso_forest.fit(X_train)

    ocsvm = OneClassSVM(kernel="rbf", gamma="scale", nu=0.1)
    ocsvm.fit(X_train)

    lof = LocalOutlierFactor(
        n_neighbors=20,
        contamination=0.1,
        novelty=True
    )
    lof.fit(X_train)

    return {"isolation_forest": iso_forest, "ocsvm": ocsvm, "lof": lof}

What are these three models?

Each model defines "normal" differently. IsolationForest isolates anomalies by random partitioning depth. OneClassSVM draws a boundary around normal data using kernel density. LocalOutlierFactor compares each point's local density to its neighbors.

The novelty=True parameter on LOF is required so it can score new data after training. Without it, LOF only works on the training set.

  • Below train_ensemble, add the ensemble prediction and risk scoring functions:
def predict_ensemble(models, X):
    iso_preds = np.where(models["isolation_forest"].predict(X) == -1, 1, 0)
    ocsvm_preds = np.where(models["ocsvm"].predict(X) == -1, 1, 0)
    lof_preds = np.where(models["lof"].predict(X) == -1, 1, 0)

    votes = iso_preds + ocsvm_preds + lof_preds
    ensemble_preds = np.where(votes >= 2, 1, 0)
    return ensemble_preds


def ensemble_risk_scores(models, X):
    iso_scores = -models["isolation_forest"].score_samples(X)
    ocsvm_scores = -models["ocsvm"].decision_function(X)
    lof_scores = -models["lof"].decision_function(X)

    iso_norm = (iso_scores - iso_scores.min()) / (iso_scores.max() - iso_scores.min() + 1e-10)
    ocsvm_norm = (ocsvm_scores - ocsvm_scores.min()) / (ocsvm_scores.max() - ocsvm_scores.min() + 1e-10)
    lof_norm = (lof_scores - lof_scores.min()) / (lof_scores.max() - lof_scores.min() + 1e-10)

    risk_scores = (iso_norm + ocsvm_norm + lof_norm) / 3
    return risk_scores

How does the voting work?

predict_ensemble() collects each model's vote (anomaly or normal) and only flags a packet as a threat if at least 2 out of 3 models agree. This majority-vote approach means a single model's false alarm gets overruled by the other two.

ensemble_risk_scores() normalizes each model's raw anomaly score to a 0-1 range, then averages them into a single risk score. Higher scores mean more suspicious.

  • Verify the ensemble functions import correctly by running:
python -c "from detector import train_ensemble, predict_ensemble, ensemble_risk_scores; print('Ensemble functions ready')"

You should see Ensemble functions ready printed.

Run the detector and observe the baseline

Now tie everything together with a main script block that generates data, trains both models, and prints the comparison metrics.

  • At the bottom of detector.py, add the main execution block:
if __name__ == "__main__":
    from data_generator import generate_dataset

    df = generate_dataset()
    X = df[FEATURE_COLS].values
    y = df["label"].values

    scaler = StandardScaler()
    X_scaled = scaler.fit_transform(X)

    single_model = train_single_model(X_scaled)
    single_preds = predict_single(single_model, X_scaled)
    single_metrics = evaluate_model(y, single_preds, "Isolation Forest (Single)")

    ensemble_models = train_ensemble(X_scaled)
    ensemble_preds = predict_ensemble(ensemble_models, X_scaled)
    ensemble_metrics = evaluate_model(y, ensemble_preds, "Ensemble (Majority Vote)")

    print("\n--- Model Comparison ---")
    print(f"{'Metric':<12} {'Single':<12} {'Ensemble':<12}")
    print(f"{'Precision':<12} {single_metrics['precision']:<12} {ensemble_metrics['precision']:<12}")
    print(f"{'Recall':<12} {single_metrics['recall']:<12} {ensemble_metrics['recall']:<12}")
    print(f"{'F1 Score':<12} {single_metrics['f1']:<12} {ensemble_metrics['f1']:<12}")
    print(f"{'FPR':<12} {single_metrics['fpr']:<12} {ensemble_metrics['fpr']:<12}")

    joblib.dump(single_model, "single_model.joblib")
    joblib.dump(ensemble_models, "ensemble_models.joblib")
    joblib.dump(scaler, "scaler.joblib")
    print("\nModels saved.")

What does the main block do?

It generates the full dataset, scales all features using StandardScaler (so no single feature dominates by magnitude), then trains both the single model and the ensemble.

After printing the comparison table, it saves all models and the scaler to .joblib files. You will load these in the dashboard later without retraining.

  • Save detector.py.

Before you run this, what do you think the single Isolation Forest's false positive rate will be? Consider that it works alone with no second opinion to overrule its mistakes.

  • Run the detector by executing:
python detector.py

You should see a comparison table printed. Focus on the FPR row. The single Isolation Forest's false positive rate lands somewhere around 15-20%. That means 1 in 5 normal packets gets flagged as a threat. The ensemble's FPR is dramatically lower.

Why is the single model's FPR so high?

Isolation Forest uses random partitioning to isolate points. Some normal traffic that happens to sit near the edge of the normal distribution gets isolated quickly and mistakenly flagged. With no second opinion, every edge case becomes a false alarm.

The ensemble fixes this because each model defines "normal" differently. A packet that looks suspicious to one model but normal to the other two gets correctly cleared. This is why production intrusion detection systems never rely on a single model.

  • Confirm the model files were saved by running:
ls *.joblib

You should see single_model.joblib, ensemble_models.joblib, and scaler.joblib listed.

Not seeing all three .joblib files?

If the script errored before saving, check the traceback. A common cause is a typo in FEATURE_COLS that does not match the column names in your data generator. The feature names must be identical.

Also confirm your virtual environment is active. If you see a ModuleNotFoundError for scikit-learn or joblib, reactivate your environment and try again.

help me debug why detector.py is failing to save model files

✔️ Awesome, I've got everything!

Great. Double check you have saved detector.py and all three .joblib files are visible in your project folder.

ⓧ I'd like to double check the full code

Your detector.py is built from the chunks above. Scroll up and confirm you have these sections in order:

  • Imports and FEATURE_COLS list (8 import lines plus the list).
  • train_single_model() and predict_single() functions.
  • evaluate_model() function.
  • train_ensemble() function.
  • predict_ensemble() and ensemble_risk_scores() functions.
  • if __name__ == "__main__": block at the bottom.

You now have a working anomaly detector, but its false positive rate is a problem. In the next step, you will dig deeper into the ensemble's improvement and add explainability so every alert tells you exactly which features triggered it.

Build the Ensemble Detector

Your single Isolation Forest detects attacks, but it flags way too many normal packets as threats. That 15-20% false positive rate you saw means roughly 1 in 5 legitimate packets gets flagged.

The fix is an ensemble. Instead of trusting one model's opinion, you require multiple models with different detection strategies to agree before raising an alert. In this step, you will walk through how the three-model ensemble in your detector.py works and verify the false positive improvement.

In this step, get ready to:
  • Understand how three different anomaly detection models complement each other.
  • See how majority voting eliminates false positives from edge-case disagreements.
  • Verify the ensemble achieves under 5% FPR while maintaining detection recall.
Understand the three-model ensemble

Each model in your ensemble uses a fundamentally different mathematical approach to define what "normal" traffic looks like. When they disagree on an edge case, that disagreement is exactly how false positives get filtered out.

  • In VS Code, open detector.py and find the train_ensemble() function.
def train_ensemble(X_train):
    iso_forest = IsolationForest(
        n_estimators=100,
        contamination=0.1,
        random_state=42
    )
    iso_forest.fit(X_train)

    ocsvm = OneClassSVM(kernel="rbf", gamma="scale", nu=0.1)
    ocsvm.fit(X_train)

    lof = LocalOutlierFactor(
        n_neighbors=20,
        contamination=0.1,
        novelty=True
    )
    lof.fit(X_train)

    return {"isolation_forest": iso_forest, "ocsvm": ocsvm, "lof": lof}

What do the three models do differently?

  • IsolationForest isolates anomalies by randomly partitioning data. Outliers require fewer partitions to isolate, so they get short path lengths.
  • OneClassSVM learns a boundary around normal data using kernel density estimation. Anything outside that boundary is flagged.
  • LocalOutlierFactor compares each point's local density to its neighbors. Points in sparse regions relative to their neighbors are flagged. The novelty=True parameter enables prediction on new unseen data.

Because each model uses different math, a packet that tricks one model into a false positive is unlikely to trick all three.

Why these specific parameters?

  • nu=0.1 on OneClassSVM sets an upper bound on the fraction of training errors (roughly matching the 10% contamination in your data).
  • n_neighbors=20 on LOF means each point's density is compared against its 20 nearest neighbors, giving a stable local density estimate.
  • contamination=0.1 tells both IsolationForest and LOF to expect about 10% of samples are anomalies, matching the attack injection rate from your data generator.
Understand majority voting and risk scores

Training three models is only half the solution. The key insight is requiring agreement before flagging a threat.

  • Find the predict_ensemble() function below train_ensemble() in your file.
def predict_ensemble(models, X):
    iso_preds = np.where(models["isolation_forest"].predict(X) == -1, 1, 0)
    ocsvm_preds = np.where(models["ocsvm"].predict(X) == -1, 1, 0)
    lof_preds = np.where(models["lof"].predict(X) == -1, 1, 0)

    votes = iso_preds + ocsvm_preds + lof_preds
    ensemble_preds = np.where(votes >= 2, 1, 0)
    return ensemble_preds

How does majority voting work here?

Each model outputs -1 for anomaly or 1 for normal. The code converts these to 1 (anomaly) or 0 (normal), then sums the votes.

A packet only gets flagged if votes >= 2. This means at least 2 of 3 models must agree it is an anomaly. A single model's false alarm gets outvoted by the other two.

  • Now find the ensemble_risk_scores() function directly below.
def ensemble_risk_scores(models, X):
    iso_scores = -models["isolation_forest"].score_samples(X)
    ocsvm_scores = -models["ocsvm"].decision_function(X)
    lof_scores = -models["lof"].decision_function(X)

    iso_norm = (iso_scores - iso_scores.min()) / (iso_scores.max() - iso_scores.min() + 1e-10)
    ocsvm_norm = (ocsvm_scores - ocsvm_scores.min()) / (ocsvm_scores.max() - ocsvm_scores.min() + 1e-10)
    lof_norm = (lof_scores - lof_scores.min()) / (lof_scores.max() - lof_scores.min() + 1e-10)

    risk_scores = (iso_norm + ocsvm_norm + lof_norm) / 3
    return risk_scores

What does this function do?

Beyond the binary flag, this function gives each packet a continuous risk score between 0 and 1. It normalizes each model's raw anomaly scores to a 0-1 range, then averages them. Higher scores mean more models see the packet as suspicious.

The 1e-10 prevents division by zero when all scores are identical. The negation on each score function flips them so higher values mean more anomalous.

Run the ensemble and verify the improvement

Now that you understand how the ensemble works, run the full detector script and focus on the comparison between the single model and the ensemble.

Before you run it, predict: do you think the ensemble will have higher or lower precision than the single Isolation Forest? What about recall?

  • Run your detector script by running this command:
python detector.py

You should see a comparison table like this:

--- Model Comparison ---
Metric       Single       Ensemble
Precision    ...          ...
Recall       ...          ...
F1 Score     ...          ...
FPR          ~0.15-0.20   ~0.02-0.05

What should I see?

  • The single model's FPR should be roughly 0.15 to 0.20 (15-20% of normal packets falsely flagged).
  • The ensemble's FPR should drop to under 0.05 (below 5%).
  • Recall stays high for the ensemble. The majority vote removes false alarms without losing real detections.
  • You should also see "Models saved." at the bottom, confirming the .joblib files were written.

Not seeing a big FPR improvement?

  • Confirm the predict_ensemble() function uses votes >= 2 (not >= 1). A threshold of 1 would flag anything ANY model flags, making it worse than a single model.
  • Check that novelty=True is set on LocalOutlierFactor. Without it, LOF cannot call predict() on data and will throw an error.
  • Make sure your virtual environment is activated. If you see import errors, the libraries are not available.

Still stuck? Help me debug why my ensemble isn't reducing false positives compared to the single model.

  • Confirm the saved model files exist by running:
ls *.joblib

You should see single_model.joblib, ensemble_models.joblib, and scaler.joblib listed. These saved models will power the dashboard you build later.

Why save models with joblib?

Training takes time (especially as datasets grow). Saving trained models lets your Streamlit dashboard load them instantly instead of retraining every time you refresh the page.

✔️ Awesome, I've got everything!

Great. Double check you've saved detector.py and all three .joblib files are in your project folder.

ⓧ I'd like to double check the full code

Your detector.py should contain these functions in order. Scroll through your file and confirm each one is present:

  • FEATURE_COLS list at the top with 7 feature names.
  • train_single_model() training a single IsolationForest.
  • train_ensemble() training IsolationForest + OneClassSVM + LocalOutlierFactor.
  • predict_single() and predict_ensemble() with majority voting (votes >= 2).
  • ensemble_risk_scores() normalizing and averaging scores from all three models.
  • evaluate_model() computing precision, recall, F1, and FPR.
  • A __main__ block that trains both models, prints the comparison table, and saves .joblib files.

Your ensemble is catching threats with a dramatically lower false alarm rate. Next up, you will add SHAP explainability so every alert tells you exactly which network features triggered it.

Add SHAP Explainability

Your ensemble catches threats with low false positives. But if someone asks "why did it flag this packet?" your system has no answer.

A black-box detector that cannot explain itself is useless in production. In this step, you will add SHAP (SHapley Additive exPlanations) to decompose each alert into per-feature contributions. Every flagged packet will come with a breakdown showing exactly which network features triggered it and by how much.

In this step, get ready to:
  • Build a SHAP explainability module using TreeExplainer on your IsolationForest model.
  • Run SHAP on a flagged anomaly and see per-feature contribution scores with direction indicators.
Create the SHAP explanation functions

SHAP's TreeExplainer has native support for scikit-learn's IsolationForest. It explains the raw anomaly score (average path length) by computing exact, fast Shapley values for each feature.

  • Create a new file called explainer.py in your project folder.
  • Add the following two functions that form the explainability module:
import numpy as np
import shap
from sklearn.ensemble import IsolationForest


def explain_anomaly(model, X_sample, feature_names):
    # Create a TreeExplainer for the IsolationForest model
    explainer = shap.TreeExplainer(model)
    explanation = explainer(X_sample)
    return explanation


def get_top_features(explanation, feature_names, sample_idx=0):
    # Extract SHAP values and pair them with feature names
    shap_values = explanation.values[sample_idx]
    feature_contributions = dict(zip(feature_names, shap_values))
    # Sort by absolute impact so highest contributors come first
    sorted_features = sorted(
        feature_contributions.items(),
        key=lambda x: abs(x[1]),
        reverse=True
    )
    return sorted_features

What do these functions do?

  • explain_anomaly() wraps SHAP's TreeExplainer. It takes your trained IsolationForest model and a sample, then returns a full SHAP explanation object containing per-feature contribution values.
  • get_top_features() takes that explanation object and returns a sorted list of (feature_name, shap_value) tuples. The sorting uses absolute value so the biggest contributors appear first, regardless of direction.

Why TreeExplainer instead of KernelExplainer?

TreeExplainer computes exact SHAP values using the internal tree structure. KernelExplainer is model-agnostic but much slower because it approximates values through sampling. Since IsolationForest is tree-based, TreeExplainer gives you exact answers instantly.

  • Save explainer.py.
Test SHAP on a flagged anomaly

Your helper functions are ready. Now you need a test script that generates data, runs detection, picks a flagged anomaly, and prints the SHAP explanation for it.

  • In explainer.py, add the following __main__ block below your get_top_features function:
if __name__ == "__main__":
    from data_generator import generate_dataset
    from detector import FEATURE_COLS, train_single_model, predict_single
    from sklearn.preprocessing import StandardScaler

    # Generate data and train a fresh model
    df = generate_dataset()
    X = df[FEATURE_COLS].values
    scaler = StandardScaler()
    X_scaled = scaler.fit_transform(X)

    model = train_single_model(X_scaled)
    preds = predict_single(model, X_scaled)

    # Find the first flagged anomaly and explain it
    anomaly_indices = np.where(preds == 1)[0]
    if len(anomaly_indices) > 0:
        sample_idx = anomaly_indices[0]
        sample = X_scaled[sample_idx:sample_idx+1]

        explanation = explain_anomaly(model, sample, FEATURE_COLS)
        top_features = get_top_features(explanation, FEATURE_COLS, 0)

        print(f"\nSHAP Explanation for anomaly at index {sample_idx}:")
        print(f"Attack type: {df.iloc[sample_idx]['attack_type']}")
        print(f"\nFeature contributions (sorted by impact):")
        for feature, value in top_features:
            direction = "toward anomaly" if value > 0 else "toward normal"
            print(f"  {feature}: {value:+.4f} ({direction})")

What does this test block do?

  • It imports your data generator and detector, generates a fresh dataset, and trains an IsolationForest model.
  • It runs detection to find all flagged anomalies, then picks the first one.
  • It passes that flagged sample through explain_anomaly() and get_top_features(), then prints each feature's contribution with a direction label.
  • Save explainer.py.

Before you run the script, think about what you expect to see. Will the SHAP values all point toward anomaly, or will some features push toward normal?

  • Run the explainer script by executing this command in your terminal:
python explainer.py

You should see output showing the attack type of the flagged anomaly and a sorted list of feature contributions. Each feature displays a signed SHAP value and a direction label ("toward anomaly" or "toward normal"). The features with the largest absolute values are listed first.

What does the output mean?

A positive SHAP value means that feature pushed the model's score toward classifying the packet as an anomaly. A negative value means it pushed toward normal. For example, if you see port_entropy: +0.3 (toward anomaly), it means unusually high port entropy was a major reason the model flagged that packet.

Seeing an import error or no output?

  • If you see ModuleNotFoundError: No module named 'shap', make sure your virtual environment is activated and you installed SHAP in Step 1.
  • If the script runs but prints nothing, check that data_generator.py and detector.py are in the same directory as explainer.py.
  • Make sure the if __name__ == "__main__": block is at the top level of the file (not indented inside a function).

Help me debug my SHAP explainer script

✔️ Awesome, I've got everything!

Great. Double check you have saved explainer.py with both functions and the test block.

ⓧ I'd like to double check the full code

Your explainer.py should contain these sections in order. Scroll up and confirm each one is present:

  • Imports: numpy, shap, and IsolationForest from sklearn.
  • explain_anomaly() function: creates a shap.TreeExplainer and returns the explanation object.
  • get_top_features() function: extracts SHAP values, pairs with feature names, sorts by absolute impact.
  • __main__ block: generates data, trains model, finds first anomaly, prints SHAP contributions with direction labels.

Your detection system can now explain every alert it raises. Next up, you will build a live Streamlit dashboard that displays threats, risk scores, and these SHAP explanations in real-time.

Build the Live Streamlit Dashboard

Your detection pipeline and explainability system work perfectly in scripts. But a portfolio piece needs a visual interface that you can demo live in an interview.

In this step, you will build a Streamlit dashboard that lets you generate attacks on the fly, watch the AI detect and explain them, and visually compare single-model vs ensemble performance.

In this step, get ready to:
  • Create the Streamlit app with sidebar controls for traffic generation.
  • Wire the detection pipeline into a live threat feed and SHAP explanation panels.
  • Add a model comparison chart showing single-model vs ensemble metrics side-by-side.
Create the Streamlit app with sidebar controls

The dashboard needs two things from the start: imports to connect your existing modules, and sidebar widgets that let the user control traffic generation parameters.

  • Create a new file called app.py in your project folder and add the following imports, page configuration, and sidebar setup:
import streamlit as st
import numpy as np
import pandas as pd
import plotly.graph_objects as go
import plotly.express as px
from sklearn.preprocessing import StandardScaler
import shap

from data_generator import generate_dataset
from detector import (
    FEATURE_COLS, train_single_model, train_ensemble,
    predict_single, predict_ensemble, ensemble_risk_scores,
    evaluate_model
)
from explainer import explain_anomaly, get_top_features


st.set_page_config(page_title="AI Network IDS", layout="wide")
st.title("AI Network Intrusion Detection System")
st.caption("Ensemble ML + Explainable AI powered threat detection")

# Sidebar controls
st.sidebar.header("Controls")
n_normal = st.sidebar.slider("Normal traffic samples", 1000, 10000, 5000, 500)
contamination_rate = st.sidebar.slider("Attack contamination %", 5, 30, 10, 1)
n_attack = int(n_normal * contamination_rate / 100)

What does this code do?

The imports pull in your three modules (data_generator, detector, explainer) along with Streamlit and Plotly for visualization.

The sidebar gives the user two sliders: one for normal traffic volume and one for attack contamination rate. The n_attack variable dynamically calculates how many attack samples to inject based on the chosen percentage.

Now add the main interaction logic. When the user clicks the generate button, the app runs the full pipeline and stores results in Streamlit's session state.

  • Add the button callback below the sidebar setup in app.py:
if st.sidebar.button("Generate Traffic & Detect"):
    with st.spinner("Generating traffic and training models..."):
        df = generate_dataset(n_normal=n_normal, n_attack=n_attack)
        X = df[FEATURE_COLS].values
        y = df["label"].values

        scaler = StandardScaler()
        X_scaled = scaler.fit_transform(X)

        single_model = train_single_model(X_scaled)
        ensemble_models = train_ensemble(X_scaled)

        single_preds = predict_single(single_model, X_scaled)
        ensemble_preds = predict_ensemble(ensemble_models, X_scaled)
        risk_scores = ensemble_risk_scores(ensemble_models, X_scaled)

        single_metrics = evaluate_model(y, single_preds, "Single Model")
        ensemble_metrics = evaluate_model(y, ensemble_preds, "Ensemble")

        st.session_state["df"] = df
        st.session_state["X_scaled"] = X_scaled
        st.session_state["single_model"] = single_model
        st.session_state["ensemble_models"] = ensemble_models
        st.session_state["ensemble_preds"] = ensemble_preds
        st.session_state["risk_scores"] = risk_scores
        st.session_state["single_metrics"] = single_metrics
        st.session_state["ensemble_metrics"] = ensemble_metrics

What does this code do?

When the button is clicked, Streamlit runs the full detection pipeline: generate data, scale it, train both the single model and ensemble, compute predictions and risk scores, then evaluate both models.

All results go into st.session_state so they persist across Streamlit reruns. Without session state, the data would disappear every time the user interacts with a widget.

  • Save app.py.
  • Launch the dashboard by running this command in your terminal:
streamlit run app.py

Your browser opens to localhost:8501. You should see the page title "AI Network Intrusion Detection System" and a sidebar on the left with two sliders and a "Generate Traffic & Detect" button.

Don't see the sidebar?

Make sure your virtual environment is activated. Streamlit needs access to all your installed packages.

Check that app.py is in the same folder as data_generator.py and detector.py. The imports require them to be in the same directory.

Help me debug my Streamlit app not launching

Wire the detection results into the display

The sidebar generates data and stores it, but the main area is still blank. You need to read from session state and render the model comparison metrics and chart.

  • Add the model comparison panel below the button logic in app.py:
if "df" in st.session_state:
    df = st.session_state["df"]
    ensemble_preds = st.session_state["ensemble_preds"]
    risk_scores = st.session_state["risk_scores"]
    single_metrics = st.session_state["single_metrics"]
    ensemble_metrics = st.session_state["ensemble_metrics"]

    # Model comparison panel
    st.header("Model Comparison")
    col1, col2, col3, col4 = st.columns(4)
    col1.metric("Ensemble Precision", f"{ensemble_metrics['precision']:.2%}")
    col2.metric("Ensemble Recall", f"{ensemble_metrics['recall']:.2%}")
    col3.metric("Ensemble F1", f"{ensemble_metrics['f1']:.2%}")
    col4.metric("FPR Reduction", f"{(single_metrics['fpr'] - ensemble_metrics['fpr']):.2%}")

    metrics_df = pd.DataFrame([single_metrics, ensemble_metrics])
    fig_compare = go.Figure()
    for metric in ["precision", "recall", "f1", "fpr"]:
        fig_compare.add_trace(go.Bar(
            name=metric.upper(),
            x=metrics_df["model"],
            y=metrics_df[metric],
            text=[f"{v:.2%}" for v in metrics_df[metric]],
            textposition="auto"
        ))
    fig_compare.update_layout(
        barmode="group",
        title="Single Model vs Ensemble Performance",
        yaxis_title="Score"
    )
    st.plotly_chart(fig_compare, use_container_width=True)

What does this code do?

The if "df" in st.session_state check ensures the display panels only render after the user has clicked the generate button.

Four metric cards show the ensemble's headline numbers at a glance. The grouped bar chart uses Plotly to render precision, recall, F1, and false positive rate side-by-side for both models so the improvement is immediately visible.

Next, add the threat feed table. This shows every packet the ensemble flagged as malicious, ranked by risk score.

  • Add the threat feed panel directly below the st.plotly_chart(fig_compare, use_container_width=True) line, still inside the if "df" in st.session_state: block:
    # Threat feed panel
    st.header("Threat Feed")
    flagged_mask = ensemble_preds == 1
    flagged_df = df[flagged_mask].copy()
    flagged_df["risk_score"] = risk_scores[flagged_mask]
    flagged_df = flagged_df.sort_values("risk_score", ascending=False).reset_index(drop=True)

    st.write(f"Detected {len(flagged_df)} threats out of {len(df)} packets")
    st.dataframe(
        flagged_df[FEATURE_COLS + ["attack_type", "risk_score"]].head(20),
        use_container_width=True
    )

What does this code do?

The threat feed filters to only packets where the ensemble predicted an anomaly. It attaches the normalized risk score, sorts highest-risk first, and displays the top 20 in an interactive table. The attack_type column reveals what kind of attack each flagged packet actually is.

  • Save app.py and switch to your browser.
  • Click Generate Traffic & Detect in the sidebar.

After a few seconds of spinner, you should see four metric cards at the top, a grouped bar chart comparing Single Model vs Ensemble, and a table of flagged threats sorted by risk score.

Seeing an error after clicking the button?

Check that all the indentation inside the if "df" in st.session_state: block uses exactly 4 spaces. The model comparison and threat feed code must be indented one level inside that if-block.

Help me fix my Streamlit dashboard error

Add the SHAP explanation panel

The dashboard shows what was flagged, but not why. The SHAP panel lets you select any alert and see exactly which network features triggered it.

  • Add the SHAP explanation section below the threat feed code, still inside the if "df" in st.session_state: block:
    # SHAP explanation panel
    st.header("SHAP Explanation")
    if len(flagged_df) > 0:
        alert_idx = st.selectbox(
            "Select alert to explain",
            range(min(20, len(flagged_df))),
            format_func=lambda i: f"Alert {i+1} (Risk: {flagged_df.iloc[i]['risk_score']:.3f}, Type: {flagged_df.iloc[i]['attack_type']})"
        )

        X_scaled = st.session_state["X_scaled"]
        single_model = st.session_state["single_model"]

        original_idx = flagged_df.index[alert_idx]
        sample = X_scaled[original_idx:original_idx+1]

        explanation = explain_anomaly(single_model, sample, FEATURE_COLS)
        top_features = get_top_features(explanation, FEATURE_COLS, 0)

        features = [f[0] for f in top_features]
        values = [f[1] for f in top_features]
        colors = ["red" if v > 0 else "blue" for v in values]

What does this code do?

The selectbox lets the user pick any of the top 20 alerts. For the selected alert, the code retrieves the original scaled sample and runs it through explain_anomaly() to get SHAP values from the TreeExplainer.

The colors list assigns red to features pushing toward anomaly (positive SHAP) and blue to features pushing toward normal (negative SHAP).

Now render the horizontal bar chart that visualizes those SHAP contributions, and add a fallback message for when no data has been generated yet.

  • Add the chart rendering directly below the colors line you just wrote, keeping the indentation inside the if len(flagged_df) > 0: block:
        fig_shap = go.Figure(go.Bar(
            x=values,
            y=features,
            orientation="h",
            marker_color=colors,
            text=[f"{v:+.3f}" for v in values],
            textposition="outside"
        ))
        fig_shap.update_layout(
            title=f"SHAP Feature Contributions (Alert {alert_idx+1})",
            xaxis_title="SHAP Value (red = toward anomaly, blue = toward normal)",
            yaxis_title="Feature",
            height=350
        )
        st.plotly_chart(fig_shap, use_container_width=True)
else:
    st.info("Click 'Generate Traffic & Detect' in the sidebar to start.")

What does this code do?

The horizontal bar chart shows each feature's SHAP value. Red bars point right (pushing toward anomaly) and blue bars point left (pushing toward normal). This is the "why" behind every alert.

The else clause at the bottom closes the if "df" in st.session_state: block. It displays an info message when the app first loads, before any data exists.

  • Save app.py.
  • Switch to your browser. Streamlit should auto-reload. Click Generate Traffic & Detect again.

Before you look at the result, predict: what do you expect to see in the SHAP chart for a port scan alert? Which features should have the largest red bars?

Scroll down past the threat feed. You should see a "SHAP Explanation" header with a dropdown listing alerts by risk score and attack type. Select any alert to see the horizontal bar chart showing per-feature SHAP contributions colored red and blue.

What should I see?

For a port scan alert, you should see port_entropy and connection_count with large red bars pushing toward anomaly. For a data exfiltration alert, expect bytes_per_second and packet_size to dominate. Each attack type has its own signature.

SHAP chart not appearing?

Check that the else: at the bottom has NO indentation (it pairs with the outer if "df" in st.session_state: block, not the inner if len(flagged_df) > 0: block).

If you see an index error, ensure original_idx = flagged_df.index[alert_idx] uses flagged_df.index (not just alert_idx directly) to map back to the correct row in the full dataset.

Help me fix my SHAP explanation panel

✔️ Awesome, I've got everything!

Great. Double check you have saved app.py and all three panels (Model Comparison, Threat Feed, SHAP Explanation) render correctly.

ⓧ I'd like to double check the full code

Your app.py should contain the following sections in order. Scroll up and confirm each one is present:

  • Imports (streamlit, numpy, pandas, plotly, sklearn, shap, plus your three local modules).
  • Page config (st.set_page_config, st.title, st.caption).
  • Sidebar controls (header, two sliders, n_attack calculation).
  • Button callback storing 8 items into st.session_state.
  • Session state retrieval + Model Comparison (4 metric columns + grouped bar chart).
  • Threat Feed (flagged_df sorted by risk_score, top 20 displayed).
  • SHAP Explanation (selectbox, explain_anomaly call, horizontal bar chart with red/blue colors).
  • The else: st.info(...) fallback at the bottom.

Secret mission

Add an Autoencoder for Zero-Day Threat Detection

Your ensemble catches known attack patterns with low false positives. But what about attacks it has never seen before? In this mission, you will add a neural autoencoder that learns the shape of normal traffic and flags anything that deviates from it, catching novel zero-day threats your tree-based models miss entirely.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally, so there are no ongoing costs.

Resources you used:

  • Project directory on your Desktop containing all Python files and saved models.
  • Python virtual environment with installed packages (scikit-learn, shap, streamlit, pandas, numpy, plotly).
  • Saved model files (single_model.joblib, ensemble_models.joblib, scaler.joblib).
  • Streamlit process (if still running in your terminal).

Keep everything running

No action needed. Choose this if you are still actively building or want to keep experimenting with the dashboard.

  • Your project folder stays on your Desktop with all files intact.
  • You can relaunch the dashboard at any time by activating your virtual environment and running streamlit run app.py.
  • The virtual environment and all installed packages remain available.

Pause - I'll come back to this later

Shut down running processes to free up memory. Your files remain on disk for later.

  • If Streamlit is still running in your terminal, press Ctrl+C to stop it.
  • Deactivate your virtual environment by running:
deactivate
  • Your project folder and all files stay on your Desktop. When you are ready to return, activate the virtual environment again and relaunch Streamlit.

Delete - I don't want to use this again

Remove all project resources and start fresh. This deletes your code, models, and virtual environment permanently.

  • If Streamlit is still running in your terminal, press Ctrl+C to stop it.
  • Deactivate your virtual environment if it is active:
deactivate
  • Navigate to your Desktop and delete the project folder. On macOS/Linux, run:
cd ~/Desktop
rm -rf network-ids
  • On Windows, run:
cd ~\Desktop
rmdir /s /q network-ids
  • This removes all Python files, saved model files, and the virtual environment with all installed packages.

Nice Work!

Nice Work!

You just built an AI-powered network intrusion detection system that detects, classifies, and explains threats in real-time.

You've learned how to:

  • Engineer an ensemble anomaly detector combining IsolationForest, OneClassSVM, and LocalOutlierFactor with majority voting to dramatically reduce false positives over a single model.
  • Add SHAP explainability using TreeExplainer to decompose every anomaly alert into per-feature contributions that tell you exactly why a packet was flagged.
  • Build a live Streamlit dashboard with a threat feed, interactive SHAP waterfall charts, and side-by-side model comparison metrics.
  • Secret Mission: Added an autoencoder zero-day detector that catches novel attacks the ensemble cannot see.

Ready to quiz yourself?