AI-Powered Phishing Domain Detector

Train an ML model to catch phishing domains from live certificate logs.

Introduction

30 Second Summary

Every day, attackers register thousands of fake domains designed to trick people into handing over passwords and credit card numbers. By the time these phishing sites hit a blacklist, the damage is already done.

In this project, you will build a machine learning pipeline that catches phishing domains at the moment they register a TLS certificate, before they send a single email. You'll train two competing models on real phishing data, pick the winner, and wire it to a live stream of newly issued certificates.

What You'll Build

You can feed any domain into your terminal and get back an instant phishing probability with a plain-English explanation of why the model flagged it, or point it at a live certificate feed and watch suspicious domains get called out in real-time as they appear.

By the end of this project, you'll have:

  • A trained classifier that scores any domain string as phishing or benign, with SHAP-powered explanations showing exactly which lexical features triggered the verdict.
  • A head-to-head model comparison between XGBoost and Random Forest on real phishing data, evaluated with precision, recall, and F1 metrics.
  • A live terminal tool that connects to the Certificate Transparency log stream and flags suspicious domains within seconds of their certificate being issued.
  • Secret Mission: Add DNS enrichment that resolves flagged domains and checks for hallmarks of throwaway phishing infrastructure.

Are there any prerequisites?

You need basic Python familiarity (variables, functions, loops) and VS Code installed on your machine. No prior machine learning experience is required.

Before We Start

Before We Start...

Before we dive in, let's lock in what you're about to build and why it matters. Phishing domains get registered and issued TLS certificates every minute. Blacklists catch them hours or days later. You're going to build a detector that catches them at the exact moment their certificate appears, before a single phishing email goes out.

Set Up Your Environment

The phishing detector you are building relies on machine learning libraries for training, explainability tools for interpreting predictions, and a streaming client for connecting to live certificate feeds.

Before any of that can work, you need the right Python version and an isolated environment with all six packages installed. This step confirms your Python version, creates a clean workspace, and installs everything the project needs.

In this step, get ready to:
  • Confirm Python 3.12 or higher is installed.
  • Create a project directory with a virtual environment.
  • Install all ML and streaming packages.
Check your Python version

This project requires Python 3.12 or higher because both XGBoost and SHAP depend on it. Running an older version causes silent install failures with confusing error messages.

  • Open your terminal by pressing Cmd+Space (macOS) or the Windows key (Windows), typing Terminal (macOS) or PowerShell (Windows), and pressing Enter.
  • Check your Python version by running this command:
python3 --version

You should see output like Python 3.12.x or higher.

✔️ I see version 3.12 or higher

Your Python version is ready. Move on to the next substep.

ⓧ I see an older version

macOS often ships with an older Python that shadows newer installs. You need to install Python 3.12+ alongside it.

On macOS, install via Homebrew:

brew install python@3.12

After the install, Homebrew places the new binary at a versioned path. Add it to your PATH by running:

echo 'export PATH="$HOMEBREW_PREFIX/opt/python@3.12/libexec/bin:$PATH"' >> ~/.zshrc
source ~/.zshrc

On Windows, download and run the latest Python installer from python.org/downloads. Make sure to check Add python.exe to PATH during installation.

After installing, close and reopen your terminal, then run python3 --version again to confirm 3.12 or higher appears.

ⓧ Command not found

Python is not installed or not in your PATH.

On macOS, install Python via Homebrew:

brew install python@3.12

If you do not have Homebrew, install it first from brew.sh.

On Windows, download and run the installer from python.org/downloads. Check Add python.exe to PATH during installation. On Windows, use python instead of python3 for all commands in this project.

After installing, close and reopen your terminal, then run python3 --version again.

Create your project directory and virtual environment

A virtual environment isolates this project's packages from the rest of your system. Nothing you install here touches your global Python.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir phish-detector
cd phish-detector
  • Create a virtual environment inside this folder by running:
python3 -m venv venv
  • Confirm the virtual environment was created by running:
ls

You should see a venv folder listed in the output.

  • Activate the virtual environment:

macOS

  • Run this activation command:
source venv/bin/activate

Windows

  • Run this activation command:
venv\Scripts\activate

Your terminal prompt now shows (venv) at the beginning. This confirms the virtual environment is active.

Don't see (venv) in your prompt?

Make sure you ran the activation command from inside the phish-detector folder. On Windows, if you get a permissions error, run Set-ExecutionPolicy -Scope CurrentUser RemoteSigned in PowerShell first, then retry.

Help me fix my virtual environment activation.

Install dependencies and verify

With the virtual environment active, every package you install stays scoped to this project.

  • Install all six project packages by running:
pip install xgboost scikit-learn shap pandas certstream matplotlib

What are these packages?

  • xgboost trains the gradient-boosted phishing classifier.
  • scikit-learn provides Random Forest, train/test splitting, and evaluation metrics.
  • shap generates human-readable explanations of model predictions.
  • pandas handles the labeled dataset as DataFrames.
  • certstream streams newly issued TLS certificates in real time.
  • matplotlib renders the SHAP summary plot.

The install may take a minute or two. You will see packages downloading and a final line confirming successful installation.

Before you run the next command, what do you think will happen if any package failed to install?

  • Verify all packages import correctly by running:
python3 -c "import xgboost; import shap; import certstream; print('All imports OK')"

You should see All imports OK printed to the terminal.

Seeing an ImportError?

Make sure your virtual environment is active. You should see (venv) in your prompt. If not, re-run the activation command from the phish-detector folder.

If xgboost or shap failed to install, confirm your Python version is 3.12+ by running python3 --version inside the venv. Both packages require Python 3.12 or higher.

Help me debug my import error.

  • Open VS Code in your project folder by running:
code .

If code command is not found

Open VS Code manually, press Cmd+Shift+P (macOS) or Ctrl+Shift+P (Windows), type Shell Command: Install 'code' command in PATH, and select it. Then retry code . in your terminal.

VS Code opens with the phish-detector folder loaded in the Explorer sidebar. You will see the venv folder listed there.

Your environment is ready with all six packages installed and verified. Next up, you will download real phishing and benign domain data to train the classifier on.

Collect and Label Training Data

Your environment is ready with all the ML and streaming packages installed. Now you need to give your model something to learn from.

Every classifier starts with labeled data. You need real examples of phishing domains and benign domains so the model can learn the difference between them. Without this training data, there is nothing to learn from.

In this step, get ready to:
  • Download real phishing domains from a live threat feed.
  • Download legitimate domains from the Tranco top-sites list.
  • Balance and label both sets into a training-ready CSV file.
Write the URL parser and phishing download function

The phishunt.io feed provides active phishing URLs in a text file, one per line. Some entries are full URLs like https://evil-login.com/page, so you need a parser that extracts just the domain portion.

  • In VS Code, create a new file called collect_data.py in your phish-detector folder.
  • Add the imports, data source URLs, and the domain extraction helper by pasting this code:
import urllib.request
import csv
import random
from urllib.parse import urlparse

PHISHUNT_URL = "https://raw.githubusercontent.com/0xDanielLopez/phishunt-feed/main/feed.txt"
TRANCO_URL = "https://tranco-list.eu/top-1m.csv.zip"

def extract_domain(url_or_domain):
    """Extract just the domain from a URL or return as-is if already a domain."""
    if url_or_domain.startswith("http"):
        parsed = urlparse(url_or_domain)
        return parsed.netloc or parsed.path
    return url_or_domain.strip()

What does this code do?

PHISHUNT_URL points to the phishunt.io GitHub feed, a public-domain (CC0) list of active phishing URLs updated every few hours.

TRANCO_URL points to the Tranco top-1M list, a research-grade ranking of legitimate sites that aggregates data from multiple providers.

extract_domain() handles both raw domains and full URLs. If the input starts with http, it uses Python's urlparse to pull out just the hostname.

  • Now add the function that downloads and cleans phishing domains from the feed, below extract_domain:
def download_phishing_domains():
    """Download active phishing domains from phishunt.io feed."""
    print("Downloading phishing domains from phishunt.io...")
    response = urllib.request.urlopen(PHISHUNT_URL)
    lines = response.read().decode("utf-8").strip().split("\n")
    domains = []
    for line in lines:
        domain = extract_domain(line.strip())
        if domain and "." in domain:
            domain = domain.split("/")[0]
            if ":" in domain:
                domain = domain.split(":")[0]
            domains.append(domain.lower())
    print(f"  Retrieved {len(domains)} phishing domains")
    return list(set(domains))

What does this function do?

It fetches the live phishunt feed over HTTP, then loops through each line. For each entry, it extracts the domain, strips any trailing path or port number, lowercases it, and adds it to the list.

The final list(set(domains)) deduplicates the results, since multiple phishing URLs can share the same domain.

  • Save collect_data.py.
  • Verify your parser works by running this quick test in your terminal:
python3 -c "from collect_data import extract_domain; print(extract_domain('https://evil-login.com/page'))"

You should see evil-login.com printed, confirming the parser strips the URL down to just the domain.

Seeing an ImportError?

Make sure your virtual environment is activated. Your terminal prompt should show (venv) at the beginning.

Also confirm you saved the file as collect_data.py (not collect-data.py) in the phish-detector folder.

help me fix my ImportError when testing collect_data.py

Add the benign download and assemble the dataset

You have the phishing side covered. Now you need the opposite: a set of domains that are definitely legitimate. The Tranco list ranks the top million sites by traffic across five data providers, making it ideal for this purpose.

  • Add the benign domain download function below download_phishing_domains in collect_data.py:
def download_benign_domains():
    """Download top benign domains from Tranco list."""
    import zipfile
    import io
    print("Downloading benign domains from Tranco list...")
    response = urllib.request.urlopen(TRANCO_URL)
    zip_data = io.BytesIO(response.read())
    with zipfile.ZipFile(zip_data) as zf:
        csv_filename = zf.namelist()[0]
        with zf.open(csv_filename) as f:
            reader = csv.reader(io.TextIOWrapper(f, encoding="utf-8"))
            domains = [row[1] for row in reader if len(row) >= 2]
    print(f"  Retrieved {len(domains)} benign domains")
    return domains

What does this function do?

The Tranco list is distributed as a ZIP file containing a CSV with columns rank,domain. This function downloads the ZIP into memory, extracts the CSV, and pulls out the domain column from each row.

You end up with up to 1 million confirmed-legitimate domains to contrast against the phishing set.

  • Now add the main() function that ties everything together. It samples 2000 from each class, labels them, shuffles, and saves to CSV. Paste this below download_benign_domains:
def main():
    phishing = download_phishing_domains()
    benign = download_benign_domains()

    sample_size = min(2000, len(phishing), len(benign))
    phishing_sample = random.sample(phishing, sample_size)
    benign_sample = random.sample(benign[:10000], sample_size)

    dataset = [(d, 1) for d in phishing_sample] + [(d, 0) for d in benign_sample]
    random.shuffle(dataset)

    with open("domains_labeled.csv", "w", newline="") as f:
        writer = csv.writer(f)
        writer.writerow(["domain", "label"])
        writer.writerows(dataset)

    print(f"\nDataset ready: {sample_size} phishing, {sample_size} benign domains")
    print(f"Saved to domains_labeled.csv ({len(dataset)} total rows)")
    print("\nSample phishing domains:")
    for d in phishing_sample[:5]:
        print(f"  [PHISH] {d}")
    print("\nSample benign domains:")
    for d in benign_sample[:5]:
        print(f"  [SAFE]  {d}")

if __name__ == "__main__":
    main()

What does this code do?

  • It calls both download functions, then samples 2000 domains from each class so the dataset is balanced.
  • Phishing domains get label 1, benign domains get label 0.
  • The combined list is shuffled so the model does not see all phishing followed by all benign during training.
  • It saves to domains_labeled.csv and prints 5 samples from each class so you can sanity-check the data.

Why balance the dataset?

If you had 500 phishing domains and 50,000 benign ones, the model could achieve 99% accuracy by simply predicting "benign" every time. Equal class sizes force the model to actually learn the difference.

  • Save collect_data.py.

Before you run this, what do you think the script will print if it successfully downloads and processes both feeds?

  • Run the data collection script:
python3 collect_data.py

The script will take 10-20 seconds to download both feeds. You should see output like this:

Downloading phishing domains from phishunt.io...
  Retrieved 427 phishing domains
Downloading benign domains from Tranco list...
  Retrieved 1000000 benign domains

Dataset ready: 427 phishing, 427 benign domains
Saved to domains_labeled.csv (854 total rows)

Sample phishing domains:
  [PHISH] login-facebook-facebook.blogspot.com
  [PHISH] metamaskrewards.com
  [PHISH] secure-paypal-verify.xyz
  [PHISH] microsoft365-login.top
  [PHISH] appleid-confirm.buzz

Sample benign domains:
  [SAFE]  google.com
  [SAFE]  github.com
  [SAFE]  amazon.com
  [SAFE]  wikipedia.org
  [SAFE]  reddit.com

Why might the numbers vary?

The phishunt feed is a rolling 24-hour snapshot of active phishing sites. The exact count changes every few hours as new sites appear and old ones are taken down. If the feed has fewer than 2000 domains, the script automatically adjusts the sample size to match what is available.

Seeing a connection or download error?

Make sure you have an active internet connection. Both feeds require downloading data from GitHub and tranco-list.eu.

If you see an HTTP 403 error, a corporate firewall or VPN may be blocking the request. Try disconnecting from your VPN and running again.

help me debug my collect_data.py download error

  • Confirm the CSV file was created by running:
ls

You should see collect_data.py and domains_labeled.csv listed in the output.

✔️ Awesome, I've got everything!

Great. Double check you saved collect_data.py and that domains_labeled.csv appears in your project folder.

ⓧ I'd like to double check the full code

Here is the complete collect_data.py file. Scroll through and confirm each section matches what you have:

  • Imports and URL constants at the top.
  • extract_domain() helper function.
  • download_phishing_domains() function.
  • download_benign_domains() function.
  • main() function and if __name__ == "__main__" entry point.

You now have a balanced, labeled dataset of real phishing and benign domains. Next up, you will turn these raw domain strings into numerical features that a machine learning model can actually learn from.

Engineer Lexical Features

You have 4000 labeled domains sitting in a CSV, but a classifier cannot learn from raw text strings. It needs numbers.

In this step, you will convert each domain into 12 numerical features that capture the patterns attackers use. These include random character sequences, excessive length, brand-name impersonation, and suspicious TLDs.

In this step, get ready to:
  • Build a feature extraction module with lexical pattern detection.
  • Apply features to the full dataset and confirm class separation.
Write the feature extraction module

Phishing domains have telltale patterns. They tend to be long, full of random characters, and they impersonate trusted brands. Your feature extractor will measure each of these signals as a number.

  • Create a new file called features.py in VS Code by pressing Cmd+N (macOS) or Ctrl+N (Windows).
  • Save the empty file as features.py inside your phish-detector folder.
  • Add the imports and pattern-detection constants by pasting this code into features.py:
import math
import re
from collections import Counter

# Brands that phishing domains commonly impersonate
BRAND_KEYWORDS = [
    "google", "facebook", "amazon", "microsoft", "paypal",
    "netflix", "apple", "instagram", "linkedin", "dropbox",
    "coinbase", "binance", "metamask", "outlook", "icloud"
]

# Action words that phishing domains use to create urgency
SUSPICIOUS_KEYWORDS = [
    "login", "secure", "account", "verify", "update",
    "banking", "confirm", "suspend", "alert", "security"
]

# TLDs commonly abused for phishing (cheap, minimal verification)
HIGH_RISK_TLDS = [
    "xyz", "top", "club", "buzz", "tk", "ml", "ga", "cf",
    "gq", "wang", "info", "vip", "icu", "cam", "surf"
]

# TLDs with higher registration barriers
LOW_RISK_TLDS = ["com", "org", "edu", "gov", "net", "co.uk", "ac.uk"]

What do these constants do?

Each list defines a detection signal the model will use.

  • BRAND_KEYWORDS catches impersonation attempts like secure-paypal-login.xyz.
  • SUSPICIOUS_KEYWORDS flags urgency language attackers use to trick victims into clicking.
  • HIGH_RISK_TLDS lists TLDs that are cheap and require minimal identity verification, making them popular for throwaway phishing sites.
  • Next, add the entropy helper function below the constants in features.py:
def shannon_entropy(s):
    """Calculate Shannon entropy of a string."""
    if not s:
        return 0.0
    counter = Counter(s)
    length = len(s)
    return -sum((count / length) * math.log2(count / length) for count in counter.values())

What is Shannon entropy?

Shannon entropy measures how random a string looks. A domain like google.com has low entropy because it uses common, repeated characters. A domain like xk7qm3.xyz has high entropy because attackers generate random strings to avoid detection.

  • Save features.py.
  • Test the entropy function by running this in your terminal:
python3 -c "from features import shannon_entropy; print(shannon_entropy('google')); print(shannon_entropy('xk7qm3p'))"

You should see two numbers: a lower value (around 2.3) for google and a higher value (around 3.0) for the random string.

Seeing ModuleNotFoundError?

  • Make sure your terminal is in the phish-detector folder where features.py lives.
  • Confirm your virtual environment is active. You should see (venv) at the start of your terminal prompt.

Still stuck? Help me debug this ModuleNotFoundError when importing features

Now you will build the main feature extraction function. This function takes a single domain string and returns a dictionary of 12 numerical features.

  • Add the first part of the extract_features function below shannon_entropy in features.py:
def extract_features(domain):
    """Extract lexical features from a domain string."""
    domain_lower = domain.lower().strip()
    parts = domain_lower.split(".")
    sld = parts[0] if parts else domain_lower
    tld = parts[-1] if len(parts) > 1 else ""

    # Count character types for ratio calculations
    letters = re.findall(r"[a-z]", domain_lower)
    digits = re.findall(r"[0-9]", domain_lower)
    vowels = re.findall(r"[aeiou]", domain_lower)
    consonants = re.findall(r"[bcdfghjklmnpqrstvwxyz]", domain_lower)

    # Find longest run of consonants (random strings have long runs)
    consonant_sequences = re.findall(r"[bcdfghjklmnpqrstvwxyz]+", domain_lower)
    longest_consonant_seq = max((len(seq) for seq in consonant_sequences), default=0)

    # Check for brand impersonation and urgency keywords
    has_brand = int(any(brand in domain_lower for brand in BRAND_KEYWORDS))
    has_suspicious_kw = int(any(kw in domain_lower for kw in SUSPICIOUS_KEYWORDS))

    # Score TLD risk level
    if tld in HIGH_RISK_TLDS:
        tld_risk = 1.0
    elif tld in LOW_RISK_TLDS:
        tld_risk = 0.0
    else:
        tld_risk = 0.5

What does this code do?

  • It splits the domain into parts (subdomain, second-level domain, TLD) so each piece can be analyzed separately.
  • It counts character types (digits, vowels, consonants) to compute ratios that distinguish random gibberish from real words.
  • It checks whether the domain contains any known brand name or urgency keyword, both strong phishing indicators.
  • It assigns a TLD risk score (1.0 for high-risk, 0.0 for low-risk, 0.5 for everything else).
  • Complete the function by adding the return statement directly below the TLD risk block:
    # Return all 12 features as a dictionary
    return {
        "length": len(domain_lower),
        "entropy": round(shannon_entropy(sld), 4),
        "digit_ratio": round(len(digits) / max(len(domain_lower), 1), 4),
        "hyphen_count": domain_lower.count("-"),
        "dot_count": domain_lower.count("."),
        "subdomain_depth": len(parts) - 2 if len(parts) > 2 else 0,
        "longest_consonant_seq": longest_consonant_seq,
        "has_brand_keyword": has_brand,
        "has_suspicious_keyword": has_suspicious_kw,
        "tld_risk_score": tld_risk,
        "vowel_ratio": round(len(vowels) / max(len(letters), 1), 4),
        "domain_char_diversity": len(set(domain_lower)),
    }

What does each feature capture?

  • length and entropy detect randomly generated domains. Phishing domains tend to be longer and more random.
  • digit_ratio and longest_consonant_seq flag strings that do not look like real words.
  • has_brand_keyword and has_suspicious_keyword are direct impersonation signals.
  • tld_risk_score encodes how commonly the TLD is abused for phishing.
  • Save features.py.
  • Test the full extractor on a known phishing domain by running:
python3 -c "from features import extract_features; import json; print(json.dumps(extract_features('secure-paypal-login.xyz'), indent=2))"

You should see a JSON dictionary with 12 features. Notice the high entropy, the brand keyword flag set to 1, the suspicious keyword flag set to 1, and a TLD risk score of 1.0.

Seeing a SyntaxError or IndentationError?

  • The return statement must be indented inside the extract_features function (one level of indentation from the def line).
  • Make sure there is no blank line between the TLD risk block and the return statement that breaks the function scope.

Help me fix this Python error in my features.py file

✔️ Awesome, I've got everything!

Great. Double check you have saved features.py and that the test printed all 12 features.

ⓧ I'd like to double check the full code

Scroll up and confirm your features.py contains these chunks in order:

  • Imports (math, re, Counter) plus the four constant lists.
  • The shannon_entropy function.
  • The extract_features function with parsing, character counting, keyword checks, TLD risk, and the 12-key return dictionary.
Apply features to the full dataset

A single domain test proves the function works. Now you need to run it across all 4000 labeled domains to confirm that phishing and benign domains produce genuinely different feature values.

  • Run the feature extractor across your full dataset and print summary statistics by running:
python3 -c "
import pandas as pd
from features import extract_features

df = pd.read_csv('domains_labeled.csv')
features_list = [extract_features(d) for d in df['domain']]
X = pd.DataFrame(features_list)
X['label'] = df['label'].values

print('Feature matrix shape:', X.shape)
print()
print('First 5 rows:')
print(X.head().to_string())
print()
print('Mean features by class (0=benign, 1=phishing):')
print(X.groupby('label').mean().round(3).to_string())
"

What should I see?

You should see a feature matrix with shape (4000, 13) (12 features plus the label column). The first 5 rows show numeric values for each domain.

The mean-by-class table is the key output. You should see clear differences: phishing domains (label 1) have higher mean entropy, longer mean length, higher digit ratio, and higher TLD risk score compared to benign domains (label 0). These differences prove the features capture real patterns the model can learn from.

Seeing all zeros or identical means?

  • Make sure domains_labeled.csv has a domain column with real domain strings, not empty values.
  • Re-run collect_data.py if the CSV looks empty or malformed.

Help me understand why my feature means look identical for phishing and benign

Your feature extractor can now convert any domain string into 12 numbers that reveal whether it smells like phishing. Next up, you will feed these features into two competing classifiers and see which one catches more phishing domains.

Train and Compare XGBoost vs Random Forest

You have 4,000 labeled domains transformed into 12 numerical features. The data is ready for a classifier to learn from.

The question is: which model architecture best separates phishing from benign? Rather than blindly picking one, you will train both XGBoost and Random Forest on the same data and compare their precision, recall, and F1 scores head-to-head.

In this step, get ready to:
  • Train an XGBoost and a Random Forest classifier on your labeled feature set.
  • Compare both models with classification reports and tricky edge-case domains.
  • Select the winner by F1 score and save it for live scoring.
Load data and prepare the train/test split

Before training any model, you need to load the labeled CSV, convert every domain into its feature vector, and split the result into training and test sets.

  • Create a new file called train_model.py in your phish-detector folder and paste the following code:
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report, f1_score
from xgboost import XGBClassifier
import joblib
from features import extract_features

def main():
    print("Loading dataset...")
    df = pd.read_csv("domains_labeled.csv")

    print("Extracting features...")
    features_list = [extract_features(d) for d in df["domain"]]
    X = pd.DataFrame(features_list)
    y = df["label"]

    print(f"Feature matrix shape: {X.shape}")
    print(f"\nMean features by class:")
    X_with_label = X.copy()
    X_with_label["label"] = y.values
    print(X_with_label.groupby("label").mean().round(3).to_string())

    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.2, random_state=42, stratify=y
    )

if __name__ == "__main__":
    main()

What does this code do?

  • The imports pull in both classifiers (XGBoost and Random Forest), evaluation metrics, and joblib for saving the trained model to disk.
  • extract_features (from your features.py) runs on every domain in the CSV, producing a 4000x12 feature matrix.
  • train_test_split holds out 20% of data for testing, using stratify=y to keep the phishing/benign ratio equal in both halves.
  • Save train_model.py and run it to confirm the data loads correctly:
python3 train_model.py

You should see output showing Feature matrix shape: (4000, 12) followed by a table of mean feature values for each class. The phishing class should show higher entropy, more hyphens, and higher TLD risk scores than benign.

Seeing a ModuleNotFoundError?

Make sure your virtual environment is activated. You should see (venv) at the start of your terminal prompt. If not, run source venv/bin/activate (macOS/Linux) or venv\Scripts\activate (Windows) and try again.

help me fix a ModuleNotFoundError when running train_model.py

Train both classifiers

Now you will train XGBoost and Random Forest on the same training set and immediately evaluate each on the held-out test data.

  • In train_model.py, find the if __name__ == "__main__": line at the bottom. Add the following code directly above it (still inside the main() function, indented with 4 spaces):
    print("\n--- Training XGBoost ---")
    xgb_model = XGBClassifier(
        n_estimators=100, max_depth=6, learning_rate=0.1,
        eval_metric="logloss", random_state=42
    )
    xgb_model.fit(X_train, y_train)
    xgb_preds = xgb_model.predict(X_test)
    xgb_f1 = f1_score(y_test, xgb_preds)
    print(classification_report(y_test, xgb_preds, target_names=["Benign", "Phishing"]))

    print("\n--- Training Random Forest ---")
    rf_model = RandomForestClassifier(n_estimators=100, max_depth=10, random_state=42)
    rf_model.fit(X_train, y_train)
    rf_preds = rf_model.predict(X_test)
    rf_f1 = f1_score(y_test, rf_preds)
    print(classification_report(y_test, rf_preds, target_names=["Benign", "Phishing"]))

What does this code do?

  • XGBClassifier builds 100 gradient-boosted trees. Each tree corrects the mistakes of the previous ones, which is why boosting often outperforms a single decision tree.
  • RandomForestClassifier grows 100 independent trees on random subsets of features, then takes a majority vote. The two approaches tackle the same problem from different angles.
  • classification_report prints precision, recall, and F1 for each class so you can compare both models at a glance.

Why compare two models instead of just picking one?

XGBoost and Random Forest represent two fundamentally different ensemble strategies: sequential correction vs. independent voting. On phishing datasets, gradient boosting typically edges ahead on F1 because it focuses training effort on the hardest-to-classify domains.

Running both on the same split proves this on your data rather than taking it on faith. The loser still serves as a sanity check that the winner is not overfitting.

  • Save train_model.py and run it again:
python3 train_model.py

You should now see two classification reports printed side-by-side. Both should show F1 scores above 0.85 for the Phishing class. Notice the precision and recall values for each model.

Training takes too long or freezes?

With 4,000 samples and 12 features, training should finish in under 10 seconds. If it hangs, check that your CSV has exactly 4,000 rows. Run wc -l domains_labeled.csv to confirm.

help me debug why my model training is hanging or running slowly

Test tricky edge cases

Balanced test-set accuracy looks good on paper. But phishing attackers craft domains that deliberately skirt detection. A hand-picked set of tricky domains reveals whether the models handle real-world ambiguity.

  • In train_model.py, add the following code above the if __name__ == "__main__": line (still inside main(), after the Random Forest report):
    print("\n--- Edge Case Testing ---")
    edge_cases = [
        ("secure-paypal-login.xyz", 1),
        ("mycompanyhr.com", 0),
        ("g00gle-verify.top", 1),
        ("amazon.co.uk", 0),
        ("netflix-account-update.buzz", 1),
    ]
    print(f"{'Domain':<35} {'Expected':<10} {'XGBoost':<10} {'RF':<10}")
    print("-" * 65)
    for domain, expected in edge_cases:
        feats = pd.DataFrame([extract_features(domain)])
        xgb_pred = xgb_model.predict(feats)[0]
        rf_pred = rf_model.predict(feats)[0]
        label_map = {0: "Benign", 1: "Phishing"}
        print(f"{domain:<35} {label_map[expected]:<10} {label_map[xgb_pred]:<10} {label_map[rf_pred]:<10}")

What makes these domains tricky?

  • secure-paypal-login.xyz has a brand keyword AND a high-risk TLD. Obvious phishing to a human, but will features alone catch it?
  • amazon.co.uk contains a brand keyword but is completely legitimate. The model needs to avoid flagging it as phishing.
  • g00gle-verify.top uses character substitution (zeros for 'o') to evade exact brand-name matching.
  • Save train_model.py and run it again.

Before you look at the output, predict: do you think both models will get all 5 edge cases correct?

python3 train_model.py

Look at the edge case table. You will likely see at least one mismatch between the expected label and a model's prediction. One or both models may misclassify a domain like amazon.co.uk or miss the character-substitution trick in g00gle-verify.top. This is the designed lesson: raw accuracy on balanced test data does not tell the whole story. Edge cases reveal the model's blind spots.

Why edge cases matter for phishing detection

Attackers actively design domains to evade classifiers. A model with 94% test-set F1 can still miss a cleverly crafted phishing domain. In the next step, you will use SHAP to look inside the model and understand exactly which features cause these misclassifications.

Pick the winner and save the model

You have seen both reports and the edge cases. Now your script needs to programmatically choose the better model by F1 score and save it for the live scoring pipeline.

  • In train_model.py, add this final block above the if __name__ == "__main__": line (after the edge case loop):
    print(f"\n--- Results ---")
    print(f"XGBoost F1: {xgb_f1:.4f}")
    print(f"Random Forest F1: {rf_f1:.4f}")

    if xgb_f1 >= rf_f1:
        winner = xgb_model
        winner_name = "XGBoost"
    else:
        winner = rf_model
        winner_name = "Random Forest"

    print(f"\nWinner: {winner_name} with F1={max(xgb_f1, rf_f1):.4f}")
    joblib.dump(winner, "phish_model.joblib")
    joblib.dump(list(X.columns), "feature_names.joblib")
    print("Model saved to phish_model.joblib")

What does this code do?

  • The F1 comparison picks whichever model balances precision and recall better. A tie goes to XGBoost (>=).
  • joblib.dump serializes the trained model to phish_model.joblib so you can load it later without retraining.
  • The feature names list is saved to feature_names.joblib so the live scoring script knows which columns to expect.
  • Save train_model.py.

Before you run this final version, predict: based on the classification reports you saw earlier, which model do you think will win?

  • Run the complete training script:
python3 train_model.py

At the bottom of the output, you should see a line like Winner: XGBoost with F1=0.94 (your exact numbers will vary depending on the data download). The model file is now saved to disk.

  • Confirm the model files were created by running:
ls *.joblib

You should see phish_model.joblib and feature_names.joblib listed.

Don't see the joblib files?

Check that you are running ls from inside the phish-detector directory. Also scroll up in your terminal output to confirm the script printed Model saved to phish_model.joblib. If you see an error instead, the script crashed before reaching the save step.

help me figure out why my model files were not saved

✔️ Awesome, I've got everything!

Great. Double check you have saved train_model.py and that both .joblib files appear when you run ls *.joblib.

ⓧ I'd like to double check the full code

Your train_model.py should contain the following sections in this order. Scroll up and confirm each is present:

  • 8 import lines (pandas, numpy, sklearn, xgboost, joblib, features).
  • main() function opening with CSV loading and feature extraction.
  • Mean features print and train_test_split call.
  • XGBoost training block with classification report.
  • Random Forest training block with classification report.
  • Edge case testing with 5 domains and the comparison table.
  • Winner selection, joblib.dump for model and feature names.
  • if __name__ == "__main__": main() at the very end.

You now have a trained phishing classifier saved to disk. But the edge-case misses proved that a raw score is not enough. Next up, you will use SHAP to crack open the model and see exactly which features drive each prediction, turning a black-box score into a plain-English explanation.

Explain Predictions and Score Live Domains

You have a trained model that scores domains, but the edge-case failures from the last step revealed a problem. A bare score tells you nothing about why the model made its decision.

In this step, you will use SHAP to crack open the model and see which features drive each prediction. Then you will wire the model up to the live Certificate Transparency log stream, scoring brand-new domains the instant their TLS certificates are issued.

In this step, get ready to:
  • Generate SHAP explanations that reveal exactly which features make a domain look suspicious.
  • Connect to the CertStream firehose and score domains in real-time.
Generate SHAP explanations for the trained model

SHAP (SHapley Additive exPlanations) assigns each feature a contribution score for every individual prediction. Instead of guessing why the model flagged a domain, you will see the exact math behind the decision.

  • Create a new file called explain_and_score.py in your phish-detector folder and add the imports plus the first section of the explain_model() function by pasting this code:
import sys
import pandas as pd
import numpy as np
import shap
import joblib
import certstream
from features import extract_features

def explain_model():
    """Generate SHAP explanations for the trained model."""
    print("Loading model and data...")
    model = joblib.load("phish_model.joblib")
    feature_names = joblib.load("feature_names.joblib")
    df = pd.read_csv("domains_labeled.csv")

    features_list = [extract_features(d) for d in df["domain"]]
    X = pd.DataFrame(features_list)
    y = df["label"]

    print("Computing SHAP values (this may take a moment)...")
    explainer = shap.TreeExplainer(model)
    shap_values = explainer.shap_values(X)

    if isinstance(shap_values, list):
        shap_vals = shap_values[1]
    else:
        shap_vals = shap_values

What does this code do?

TreeExplainer uses the Shapley value algorithm from game theory to calculate how much each feature contributed to a given prediction. The if isinstance check handles a difference between XGBoost (returns a 2D array) and Random Forest (returns a list of arrays per class).

  • Still inside the explain_model() function, add the feature importance ranking directly below the code you just pasted:
    mean_abs_shap = np.abs(shap_vals).mean(axis=0)
    feature_importance = sorted(
        zip(feature_names, mean_abs_shap), key=lambda x: x[1], reverse=True
    )

    print("\n--- SHAP Feature Importance Ranking ---")
    for i, (name, importance) in enumerate(feature_importance, 1):
        print(f"  {i:>2}. {name:<25} {importance:.4f}")

What does this code do?

This averages the absolute SHAP values across all 4000 domains to produce a global feature importance ranking. Features with higher mean absolute SHAP values have a bigger influence on the model's phishing/benign decisions overall.

  • Save explain_and_score.py and confirm the module loads without errors by running:
python3 -c "from explain_and_score import explain_model; explain_model()"

You should see the model load, SHAP values compute (this takes 10-20 seconds), and then a ranked list of all 12 features printed to the terminal. Features like tld_risk_score and has_suspicious_keyword will likely rank near the top.

Seeing a ModuleNotFoundError?

Make sure your virtual environment is still activated. You should see (venv) at the start of your terminal prompt. If not, re-activate it with source venv/bin/activate.

If you see an error about features not being found, confirm you are running the command from inside the phish-detector folder where features.py lives.

help me debug a ModuleNotFoundError in my explain_and_score.py script

Now you will add the edge case explanations that show exactly why the model gets confused on tricky domains.

  • Add the following code directly below the importance ranking loop you just wrote, still inside explain_model():
    print("\n--- Edge Case Explanations ---")
    edge_cases = [
        "secure-paypal-login.xyz",
        "mycompanyhr.com",
        "g00gle-verify.top",
        "amazon.co.uk",
        "netflix-account-update.buzz",
    ]
    for domain in edge_cases:
        feats = pd.DataFrame([extract_features(domain)])
        sv = explainer.shap_values(feats)
        if isinstance(sv, list):
            domain_shap = sv[1][0]
        else:
            domain_shap = sv[0]
        prob = model.predict_proba(feats)[0][1]

        top_features = sorted(
            zip(feature_names, domain_shap), key=lambda x: abs(x[1]), reverse=True
        )[:3]

        direction = "PHISHING" if prob > 0.5 else "BENIGN"
        print(f"\n  {domain} -> {direction} (prob={prob:.3f})")
        for fname, sval in top_features:
            arrow = "+" if sval > 0 else "-"
            print(f"    {arrow} {fname}: {sval:+.4f}")

What does this code do?

For each edge-case domain from Step 4, this computes individual SHAP values and prints the top 3 features pushing the prediction. A + means the feature pushed toward phishing. A - means it pushed toward benign.

For example, amazon.co.uk might show has_brand_keyword pushing toward phishing, but low entropy and tld_risk_score=0 pulling it back toward benign. That is exactly the kind of insight a raw score cannot give you.

  • Add the SHAP summary plot saving code directly below the edge case loop, still inside explain_model():
    print("\n\nSaving SHAP summary plot to shap_summary.png...")
    import matplotlib
    matplotlib.use("Agg")
    import matplotlib.pyplot as plt
    shap.summary_plot(shap_vals, X, feature_names=feature_names, show=False)
    plt.tight_layout()
    plt.savefig("shap_summary.png", dpi=150, bbox_inches="tight")
    plt.close()
    print("Done! Check shap_summary.png")

What does this code do?

This saves a SHAP beeswarm plot as a PNG image. Each dot represents one domain in the dataset, colored by feature value (red = high, blue = low). Features at the top of the plot have the greatest impact on predictions. The Agg backend lets matplotlib render without a display window.

  • Save explain_and_score.py and re-run the explanation function to see the full output:
python3 -c "from explain_and_score import explain_model; explain_model()"

You should now see the importance ranking, followed by each edge-case domain with its top 3 SHAP contributions, and finally the message confirming shap_summary.png was saved.

  • Confirm the plot file was created by running:
ls shap_summary.png

You should see shap_summary.png listed in the output.

Not seeing the plot file?

If the script errored on the matplotlib import, run pip install matplotlib to ensure it is installed in your virtual environment.

If the script completed but shap_summary.png is missing, check that you are running ls from the phish-detector directory.

help me troubleshoot why shap_summary.png is not being created

Build the live CertStream scoring script

The Certificate Transparency log is a public ledger of every TLS certificate issued on the internet. CertStream taps into this ledger as a real-time WebSocket feed. You will wire your model directly to this stream so it scores domains the instant their certificates appear.

  • Add the first section of the live_score() function below explain_model() in explain_and_score.py:
def live_score():
    """Connect to CertStream and score domains in real-time."""
    print("\nStarting live CertStream scoring...")
    model = joblib.load("phish_model.joblib")
    feature_names = joblib.load("feature_names.joblib")
    explainer = shap.TreeExplainer(model)

    context_data = {"count": 0, "flagged": 0}

    def score_callback(message, context):
        if message["message_type"] != "certificate_update":
            return

        all_domains = message["data"]["leaf_cert"]["all_domains"]
        for domain in all_domains:
            if domain.startswith("*."):
                domain = domain[2:]

            context_data["count"] += 1
            feats = pd.DataFrame([extract_features(domain)])
            prob = model.predict_proba(feats)[0][1]

What does this code do?

The score_callback function is called by CertStream every time a new certificate is issued. It filters for certificate_update messages (ignoring heartbeats), extracts all domains listed on the certificate, strips wildcard prefixes (*.), and scores each domain with the trained model.

The context_data dictionary acts as a shared counter between the outer function and the callback, tracking how many domains have been processed and how many were flagged.

  • Add the decision logic, counter, connection call, and the script's entry point directly below the code you just pasted:
            if prob > 0.7:
                context_data["flagged"] += 1
                sv = explainer.shap_values(feats)
                if isinstance(sv, list):
                    domain_shap = sv[1][0]
                else:
                    domain_shap = sv[0]
                top_features = sorted(
                    zip(feature_names, domain_shap),
                    key=lambda x: abs(x[1]), reverse=True
                )[:3]
                reasons = ", ".join(f"{n}={v:+.3f}" for n, v in top_features)
                print(f"  [SUSPICIOUS] {domain} (score={prob:.3f}) | {reasons}")
            else:
                sys.stdout.write(f"  [ OK ] {domain} ({prob:.2f})\r")
                sys.stdout.flush()

            if context_data["count"] % 50 == 0:
                print(f"\n  --- Processed {context_data['count']} domains, "
                      f"{context_data['flagged']} flagged suspicious ---\n")

    print("Connecting to CertStream (wss://certstream.calidog.io/)...")
    print("Press Ctrl+C to stop.\n")
    certstream.listen_for_events(score_callback, url="wss://certstream.calidog.io/")

if __name__ == "__main__":
    if len(sys.argv) > 1 and sys.argv[1] == "--live":
        live_score()
    else:
        explain_model()
        print("\n" + "=" * 60)
        print("To start live scoring, run: python3 explain_and_score.py --live")
        print("=" * 60)

What does this code do?

Domains scoring above 0.7 probability get flagged as [SUSPICIOUS] with their top 3 SHAP reasons printed inline. Low-scoring domains flash briefly on the same line using \r (carriage return) so your terminal is not flooded.

Every 50 domains, a progress summary prints showing how many have been processed and how many were flagged. The __main__ block makes the script run explain_model() by default, or live_score() when you pass the --live flag.

  • Save explain_and_score.py.
  • First, confirm the explanation mode still works by running:
python3 explain_and_score.py

You should see the full SHAP output followed by the line: To start live scoring, run: python3 explain_and_score.py --live.

Before you run the live scorer, what do you expect to see? Do you think domains will appear quickly or slowly? Will most be flagged or will suspicious ones be rare?

  • Start the real-time CertStream scorer by running:
python3 explain_and_score.py --live

Within 10-20 seconds you will see domains streaming rapidly. Most will flash by as [ OK ], but suspicious ones will stay on screen marked [SUSPICIOUS] with their score and SHAP feature reasons. Every 50 domains you will see a progress counter.

  • Let it run for a minute or two to see some flagged domains, then press Ctrl+C to stop.

Not seeing any domains scroll?

CertStream connects over a WebSocket. If you are behind a corporate proxy or restrictive firewall, the connection to wss://certstream.calidog.io/ may be blocked. Try from a different network or a personal hotspot.

If you see a connection error immediately, confirm certstream is installed by running pip show certstream and checking the version is listed.

help me debug my CertStream connection issue in explain_and_score.py

✔️ Awesome, I've got everything!

Great. Double check you have saved explain_and_score.py and that shap_summary.png exists in your phish-detector folder.

ⓧ I'd like to double check the full code

Your explain_and_score.py file is over 30 lines, so confirm each section is present in order by scrolling through the file:

  • Imports (sys, pandas, numpy, shap, joblib, certstream, extract_features).
  • explain_model() function: loads model/data, computes SHAP values, prints importance ranking, loops through 5 edge-case domains printing top-3 SHAP reasons, saves shap_summary.png.
  • live_score() function: loads model, creates TreeExplainer, defines score_callback that filters certificate_update messages, scores domains, prints [SUSPICIOUS] for prob > 0.7 with SHAP reasons, prints counter every 50 domains, connects to wss://certstream.calidog.io/.
  • __main__ block: runs explain_model() by default, or live_score() with the --live argument.

Your detector now explains its reasoning for every prediction and scores live domains in real-time. Next up, you will extend it with DNS enrichment in the Secret Mission to corroborate suspicious scores with network evidence.

Secret mission

DNS Enrichment for Flagged Domains

Your ML model flags suspicious domains, but a high score alone can produce false positives. Real phishing infrastructure has telltale DNS characteristics: brand-new A records, no MX record, free nameservers. Can you add DNS resolution that corroborates the ML verdict and assigns a confidence level to each alert?

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally, so there are no ongoing costs.

Resources you used:

  • A Python virtual environment in phish-detector/venv/ (local, no cost).
  • Downloaded and generated data files: domains_labeled.csv, shap_summary.png (local, no cost).
  • Trained model files: phish_model.joblib, feature_names.joblib (local, no cost).

Keep everything running

No action needed. Choose this if you're still actively building or want to experiment with different feature sets, model hyperparameters, or detection thresholds.

  • Your virtual environment stays active until you close the terminal or run deactivate.
  • All project files remain in your phish-detector folder on the Desktop.
  • You can re-run the live scorer any time with python3 explain_and_score.py --live.

Pause - I'll come back to this later

Shut down running processes and deactivate the virtual environment. Your files stay on disk so you can pick up where you left off.

  • If the live scorer is still running, press Ctrl+C to stop it.
  • Deactivate your virtual environment by running:
deactivate

When you come back, reactivate the environment from inside the project folder:

cd ~/Desktop/phish-detector
source venv/bin/activate

Delete - I don't want to use this again

Remove all project files, the virtual environment, and all generated data. This cannot be undone.

  • If the live scorer is still running, press Ctrl+C to stop it.
  • Deactivate the virtual environment if it is still active:
deactivate
  • Delete the entire project folder. On macOS/Linux, run:
rm -rf ~/Desktop/phish-detector
  • On Windows, run:
rmdir /s /q %USERPROFILE%\Desktop\phish-detector
  • Confirm the folder is gone by running:
ls ~/Desktop/phish-detector

You should see a "No such file or directory" message. Everything has been removed.

Nice Work!

Nice Work!

You just built a complete machine learning pipeline that catches phishing domains at the moment they register a TLS certificate. Your detector spots malicious domains before they send a single phishing email, backed by explainable predictions you can actually trust.

You've learned how to:

  • Engineer lexical features from raw domain strings and train competing XGBoost and Random Forest classifiers, selecting a winner based on F1 score.
  • Use SHAP to explain exactly which features drive the model's phishing or benign decision for any domain.
  • Connect to the Certificate Transparency log stream and flag suspicious domains in real-time within seconds of certificate issuance.
  • Secret Mission: Added DNS enrichment that corroborates ML predictions with network evidence, producing high-confidence phishing alerts.

Ready to quiz yourself?