Build a URL Safety Checker

Build a bloom filter URL checker using the same pattern as Chrome Safe Browsing.

Introduction

30 Second Summary

Every time you click a link, your browser checks it against millions of known malicious URLs in milliseconds. It does this without sending every URL you visit to a remote server and without storing the entire threat database in memory.

In this project, you will build a URL safety checker that uses the same two-tier architecture as Chrome Safe Browsing. You'll construct a bloom filter from scratch in pure Python, load 100,000 synthetic malicious URLs, and measure the 80x memory reduction it achieves over a naive approach.

What You'll Build

Run a URL through your checker and watch it answer "definitely safe" or "maybe malicious" in microseconds, using just ~117KB of memory instead of ~10MB.

By the end of this project, you'll have:

  • A working URL safety checker where you can check any URL against 100,000 known threats in microseconds using just ~117KB of memory.
  • A two-tier architecture that eliminates 99%+ of expensive database lookups, proving that the bloom filter catches "definitely safe" URLs before they ever hit the full database.
  • A side-by-side memory comparison showing your bloom filter's ~117KB footprint versus ~10MB for a naive Python set holding the same data.
  • Secret Mission: Implement a Counting Bloom Filter that supports removing delisted URLs without rebuilding from scratch.

Are there any prerequisites?

You should be comfortable with basic Python concepts like functions, classes, and loops. Familiarity with sets and dictionaries will help, since you'll be comparing your bloom filter against a Python set.

Before We Start

Before We Start...

Before we dive in, let's lock in what you're about to build and why it matters.

You're building a URL safety checker that uses the same two-tier pattern as Chrome Safe Browsing. The entire architecture hinges on one key property: a bloom filter can say "definitely not present" but only "maybe present."

Set Up Your Project

The bloom filter you're about to build relies entirely on Python's standard library. That means zero external dependencies to install.

All you need is a confirmed Python installation and a clean project folder. Once those are ready, you can start building immediately.

In this step, get ready to:
  • Verify Python 3.8+ is installed and working.
  • Create the bloom-filter-safety project folder.
  • Confirm all required standard library modules import successfully.
Confirm your Python version
  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type VS Code and press Enter to open it.
  • Open the integrated terminal by pressing Ctrl+` (backtick key, top-left of your keyboard).
  • Check your Python version by running this command:
python3 --version

You should see Python 3.8 or higher printed (e.g. Python 3.11.4).

Seeing command not found?

If your terminal doesn't recognize python3, Python may not be installed or not added to your PATH. Visit python.org/downloads to install the latest version.

help me install Python 3.8 or higher on my machine

Create your project folder

You'll create a dedicated folder on your Desktop to keep all your project files together.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir bloom-filter-safety
cd bloom-filter-safety
  • Confirm the folder was created by running:
pwd

You should see a path ending in bloom-filter-safety.

Now open this folder in VS Code so the Explorer sidebar shows your project.

  • Click File in the VS Code menu bar.
  • Click Open Folder.
  • Navigate to your Desktop, select the bloom-filter-safety folder, and click Open.

What should I see?

The VS Code Explorer sidebar now shows BLOOM-FILTER-SAFETY as your workspace root. The folder is empty for now.

Create and run the test file

Your bloom filter will use three standard library modules: hashlib for hashing, math for computing optimal parameters, and tracemalloc for measuring memory. A quick test file confirms they all import cleanly.

  • In the VS Code Explorer sidebar, click the new file icon (page with a plus sign) next to the folder name.
  • Name the file test_setup.py and press Enter.
  • Add the following code to test_setup.py:
import hashlib
import math
import tracemalloc
print('All imports successful. Ready to build!')

What does this code do?

  • hashlib provides cryptographic hash functions (SHA-256, MD5) that you'll use to map URLs to bit positions in the bloom filter.
  • math gives you log and ceil for computing optimal bloom filter size.
  • tracemalloc measures memory usage so you can compare bloom filter memory against a naive Python set.
  • Save test_setup.py by pressing Cmd+S (macOS) or Ctrl+S (Windows).

Before you run this, what do you expect to see if all three modules are available on your system?

  • Run the test file in your terminal:
python3 test_setup.py

You should see: All imports successful. Ready to build!.

Seeing a ModuleNotFoundError?

All three modules (hashlib, math, tracemalloc) are part of Python's standard library and ship with any Python 3.4+ installation. If one is missing, your Python install may be incomplete.

Try reinstalling Python from python.org/downloads using the official installer.

help me fix a missing standard library module in Python

✔️ Awesome, I've got everything!

Great. Double check you've saved test_setup.py and your terminal printed the success message.

ⓧ I'd like to double check the full code

Here's the complete test_setup.py file:

import hashlib
import math
import tracemalloc
print('All imports successful. Ready to build!')

Your environment is locked in. Next up, you'll build a naive URL checker that loads 100,000 malicious URLs into memory and see exactly how expensive that is.

Build the Naive URL Checker

Before you can appreciate what a bloom filter gives you, you need to see what "expensive" looks like. The brute-force approach to URL safety checking is simple: load every known malicious URL into memory and check incoming URLs against that list.

In this step, you will generate 100,000 synthetic malicious URLs and build a naive checker that loads them all into a Python set. It works perfectly. But you will see exactly how much memory it costs.

In this step, get ready to:
  • Generate a dataset of 100,000 synthetic malicious URLs and 1,000 labeled test URLs.
  • Build a naive URL safety checker that loads every URL into memory.
  • Measure the memory cost of the brute-force approach.
Generate the malicious URL dataset

You need a large dataset of "known malicious" URLs to simulate a real threat database. You will generate 100,000 synthetic phishing-style URLs and a separate file of 1,000 labeled test URLs for accuracy checking.

  • Create a new file called generate_urls.py in your bloom-filter-safety folder.
  • Add the imports and the generate_malicious_urls function by pasting this code:
import random
import string
import os


def generate_malicious_urls(count=100_000):
    """Generate synthetic malicious URLs for testing."""
    schemes = ["http://", "https://"]
    tlds = [".com", ".net", ".org", ".xyz", ".ru", ".cn", ".tk"]
    patterns = [
        "phishing-{word}{tld}/login",
        "malware-download{tld}/{word}",
        "{word}-free-gift{tld}/claim",
        "account-verify-{word}{tld}/secure",
        "{word}{tld}/wp-admin/hack",
        "free-{word}{tld}/download.exe",
    ]

    urls = set()
    while len(urls) < count:
        word = "".join(random.choices(string.ascii_lowercase, k=random.randint(4, 10)))
        tld = random.choice(tlds)
        pattern = random.choice(patterns)
        scheme = random.choice(schemes)
        path = pattern.format(word=word, tld=tld)
        urls.add(scheme + path)

    return list(urls)

What does this code do?

This function creates fake malicious URLs by combining random words with phishing-style patterns like phishing-{word}.com/login. It uses a set internally to guarantee all 100,000 URLs are unique.

The patterns mimic real-world phishing URLs: fake login pages, malware downloads, and gift scams.

  • Below the generate_malicious_urls function, add the generate_test_urls function:

def generate_test_urls(malicious_urls, safe_count=500, malicious_count=500):
    """Generate test URLs: mix of known malicious and definitely safe."""
    test_malicious = random.sample(malicious_urls, min(malicious_count, len(malicious_urls)))

    safe_domains = [
        "google.com", "github.com", "stackoverflow.com", "python.org",
        "wikipedia.org", "youtube.com", "reddit.com", "amazon.com",
        "microsoft.com", "apple.com", "nextwork.org", "bbc.co.uk",
    ]
    safe_urls = []
    for i in range(safe_count):
        domain = random.choice(safe_domains)
        path = "".join(random.choices(string.ascii_lowercase, k=6))
        safe_urls.append(f"https://{domain}/{path}")

    return test_malicious, safe_urls

What does this function do?

This creates the test data you will use to measure accuracy. It picks 500 URLs that ARE in the malicious database, and generates 500 URLs from well-known safe domains like google.com and github.com that are definitely NOT malicious.

This 50/50 split lets you verify both that malicious URLs are caught and that safe URLs are not falsely flagged.

  • Verify both functions work by running this quick test:
python3 -c "from generate_urls import generate_malicious_urls; urls = generate_malicious_urls(5); print(urls)"

You should see a short list of 5 synthetic phishing-style URLs printed to the terminal.

Seeing an import error?

Make sure you saved generate_urls.py before running the test. Also confirm your terminal is inside the bloom-filter-safety folder.

help me debug an import error when running generate_urls.py

  • Now add the main block at the bottom of generate_urls.py that writes both data files:

if __name__ == "__main__":
    print("Generating 100,000 malicious URLs...")
    malicious_urls = generate_malicious_urls(100_000)

    with open("malicious_urls.txt", "w") as f:
        for url in malicious_urls:
            f.write(url + "\n")
    print(f"Saved {len(malicious_urls)} malicious URLs to malicious_urls.txt")

    print("Generating test URLs...")
    test_malicious, test_safe = generate_test_urls(malicious_urls)

    with open("test_urls.txt", "w") as f:
        for url in test_malicious:
            f.write(f"MALICIOUS|{url}\n")
        for url in test_safe:
            f.write(f"SAFE|{url}\n")
    print(f"Saved {len(test_malicious)} malicious + {len(test_safe)} safe test URLs to test_urls.txt")

    file_size = os.path.getsize("malicious_urls.txt") / 1024 / 1024
    print(f"\nDataset file size: {file_size:.2f} MB")

What does the main block do?

This is the entry point that ties everything together. It generates all 100,000 URLs, writes them to malicious_urls.txt (one per line), then creates 1,000 labeled test URLs in test_urls.txt.

Each test URL line starts with MALICIOUS| or SAFE| so the checker can measure its own accuracy.

  • Save generate_urls.py.
  • Generate both data files by running:
python3 generate_urls.py

You should see output confirming 100,000 malicious URLs were saved, 1,000 test URLs were created, and the dataset file size in MB.

  • Confirm the data files were created by running:
ls

You should see malicious_urls.txt and test_urls.txt alongside generate_urls.py.

Script takes too long or hangs?

Generating 100,000 unique URLs takes a few seconds. If it hangs beyond 30 seconds, check that the while len(urls) < count loop uses a set() to avoid duplicates. A typo here can cause an infinite loop.

help me debug why my URL generation script is hanging

✔️ Awesome, I've got everything!

Great. Double check you have saved generate_urls.py and can see both .txt files in your folder.

ⓧ I'd like to double check the full code

Your generate_urls.py should contain these sections in order. Scroll up and confirm each one is present:

  • Imports: random, string, os.
  • The generate_malicious_urls function with URL patterns and a while loop.
  • The generate_test_urls function with safe domains and random sampling.
  • The if __name__ == "__main__": block that writes both files and prints the dataset size.
Build the naive checker that loads everything into memory

Now you have 100,000 malicious URLs on disk. The simplest possible safety checker loads them all into a Python set and checks incoming URLs against it. You will use tracemalloc to measure exactly how much memory this costs.

  • Create a new file called naive_checker.py in your bloom-filter-safety folder.
  • Add the imports and helper functions by pasting this code:
import tracemalloc
import time


def load_malicious_urls():
    """Load all malicious URLs into a Python set."""
    with open("malicious_urls.txt", "r") as f:
        return set(line.strip() for line in f)


def load_test_urls():
    """Load test URLs with their labels."""
    test_urls = []
    with open("test_urls.txt", "r") as f:
        for line in f:
            label, url = line.strip().split("|", 1)
            test_urls.append((label, url))
    return test_urls


def check_url(url, malicious_set):
    """Check if a URL is malicious using the full set lookup."""
    return url in malicious_set

What do these helpers do?

  • load_malicious_urls() reads all 100K URLs from disk into a Python set. This is the expensive in-memory database.
  • load_test_urls() parses the labeled test file so you can measure accuracy.
  • check_url() performs a single membership check against the full set.
  • Below the helper functions, add the main block that loads URLs and measures memory:

if __name__ == "__main__":
    print("=== Naive URL Safety Checker ===")
    print("Strategy: Load ALL malicious URLs into memory as a Python set\n")

    tracemalloc.start()

    print("Loading malicious URL database...")
    start = time.time()
    malicious_set = load_malicious_urls()
    load_time = time.time() - start

    current, peak = tracemalloc.get_traced_memory()
    print(f"Loaded {len(malicious_set):,} URLs in {load_time:.2f}s")
    print(f"Memory used: {current / 1024 / 1024:.2f} MB")
    print(f"Peak memory: {peak / 1024 / 1024:.2f} MB")

What does this section do?

tracemalloc.start() begins tracking every memory allocation Python makes. After loading the set, get_traced_memory() reports exactly how many megabytes the 100K URLs occupy in RAM.

This is the number you will compare against the bloom filter later.

  • Save naive_checker.py and run it to see the memory cost:
python3 naive_checker.py

You should see the set load 100,000 URLs and report memory usage of several MB. The script exits cleanly after printing memory stats.

  • Now add the accuracy-checking loop below the memory measurement section in naive_checker.py:

    print("\nChecking test URLs...")
    test_urls = load_test_urls()

    correct = 0
    total = len(test_urls)
    start = time.time()

    for label, url in test_urls:
        is_malicious = check_url(url, malicious_set)
        expected = label == "MALICIOUS"
        if is_malicious == expected:
            correct += 1

    check_time = time.time() - start

    print(f"Checked {total} URLs in {check_time:.4f}s")
    print(f"Accuracy: {correct}/{total} ({100 * correct / total:.1f}%)")
    print(f"\nTotal lookups against database: {total}")
    print("Every single check required the full URL database in memory.")

    tracemalloc.stop()

What does the checking loop do?

  • For each test URL, it calls check_url() which does a full set lookup against all 100K entries.
  • It compares each result against the expected label (MALICIOUS or SAFE) and tallies correct answers.
  • The final line drives the point home: every single check required the entire URL database to sit in memory.
  • Save naive_checker.py.

Before you run this, what accuracy do you expect the naive checker to achieve? Think about it: if every malicious URL is in the set, can it ever miss one?

  • Run the full naive checker:
python3 naive_checker.py

You should see 100,000 URLs loaded, memory usage of several MB, and 100% accuracy (1000/1000 correct). Every single URL check required the full database in memory.

Why is accuracy 100%?

A Python set is deterministic. If a URL is in the set, membership returns True. If it is not, membership returns False. There is no guessing involved.

The tradeoff is memory. Holding all 100K URLs costs several MB. Imagine scaling this to Chrome's full malicious URL database of hundreds of millions of entries. That is the problem bloom filters solve.

Seeing a FileNotFoundError?

This means malicious_urls.txt or test_urls.txt does not exist in your current folder. Run python3 generate_urls.py first to create the data files, then try again.

help me debug a FileNotFoundError in naive_checker.py

✔️ Awesome, I've got everything!

Great. Double check you have saved naive_checker.py and can see memory usage plus 100% accuracy in the output.

ⓧ I'd like to double check the full code

Your naive_checker.py should contain these sections in order. Scroll up and confirm each one is present:

  • Imports: tracemalloc and time.
  • Three helper functions: load_malicious_urls(), load_test_urls(), and check_url().
  • The if __name__ == "__main__": block with tracemalloc tracking, URL loading, memory reporting, the accuracy-checking loop, and final stats.

You now have a baseline. The naive checker is 100% accurate, but it holds every single URL in memory at all times. Next up, you will build a bloom filter that achieves nearly the same result using 80x less memory.

Build a Bloom Filter from Scratch

You now have a baseline: the naive checker loads every malicious URL into a Python set that eats several MB of memory. The bloom filter will replace that expensive set with a tiny bit array that answers the same question in microseconds.

You will build the bloom filter entirely from Python's standard library. A bytearray holds the bits, and hashlib provides the hashing. No external packages needed.

In this step, get ready to:
  • Create a BloomFilter class with compact bit-level storage.
  • Implement double-hashing to generate multiple positions from one item.
  • Add an optimal classmethod that computes perfect parameters from the math.
Create the BloomFilter class with bit storage

A bloom filter stores membership information as individual bits in a compact array. Instead of storing the full URL strings, it flips bits at specific positions. This is why it uses so much less memory.

  • Create a new file called bloom_filter.py in your bloom-filter-safety folder.
  • Add the imports, class definition, constructor, and bit-manipulation methods by pasting this code:
import hashlib
import math


class BloomFilter:
    """A space-efficient probabilistic data structure for membership testing."""

    def __init__(self, size, num_hashes):
        self.size = size
        self.num_hashes = num_hashes
        self.bit_array = bytearray(math.ceil(size / 8))
        self.count = 0

    def _set_bit(self, position):
        byte_index = position // 8
        bit_index = position % 8
        self.bit_array[byte_index] |= (1 << bit_index)

    def _get_bit(self, position):
        byte_index = position // 8
        bit_index = position % 8
        return (self.bit_array[byte_index] >> bit_index) & 1

What does this code do?

  • The constructor takes size (total bits) and num_hashes (how many positions to check per item). It creates a bytearray with just enough bytes to hold that many bits.
  • _set_bit turns on a single bit using bitwise OR. _get_bit reads a single bit using bitwise shift and AND. This is how you pack 8 boolean values into one byte.
  • Save bloom_filter.py.
  • Verify the class imports and bit operations work by running this command:
python3 -c "from bloom_filter import BloomFilter; bf = BloomFilter(1000, 7); bf._set_bit(5); print(bf._get_bit(5), bf._get_bit(6))"

You should see 1 0 printed. Bit 5 is on, bit 6 is off.

Seeing an ImportError?

Make sure your file is named exactly bloom_filter.py (underscores, not hyphens) and that you are running the command from inside your bloom-filter-safety folder.

Help me debug my bloom filter import error.

Implement double-hashing and the core methods

A bloom filter needs to map each item to multiple bit positions. The double-hashing technique generates all positions from just two base hashes, avoiding the need for many independent hash functions.

  • In bloom_filter.py, add the hashing method, add(), __contains__(), and memory_bytes() below the _get_bit method:
    def _get_positions(self, item):
        """Generate bit positions using double hashing (Kirsch-Mitzenmacher)."""
        data = item.encode("utf-8")
        h1 = int.from_bytes(hashlib.sha256(data).digest()[:8], "big")
        h2 = int.from_bytes(hashlib.md5(data).digest()[:8], "big")
        for i in range(self.num_hashes):
            yield (h1 + i * h2) % self.size

    def add(self, item):
        for position in self._get_positions(item):
            self._set_bit(position)
        self.count += 1

    def __contains__(self, item):
        for position in self._get_positions(item):
            if not self._get_bit(position):
                return False
        return True

    def memory_bytes(self):
        return len(self.bit_array)

What does this code do?

  • _get_positions uses SHA-256 and MD5 as two base hashes, then combines them with the formula h1 + i * h2 to produce num_hashes unique positions. This is the Kirsch-Mitzenmacher optimization used in production systems like Cassandra.
  • add() sets bits at all hashed positions. __contains__() checks if ALL positions are set. If any position is zero, the item was definitely never added.
  • memory_bytes() returns the actual byte count of the underlying array so you can compare it to the naive set later.

Why can a bloom filter say 'definitely not' but only 'maybe yes'?

If even one bit position is zero, the item was never added. That is a guarantee. But if all positions are set, another item might have set those same bits by coincidence. That is a false positive.

  • Save bloom_filter.py.
  • Test that the bloom filter can add and check items by running this command:
python3 -c "from bloom_filter import BloomFilter; bf = BloomFilter(1000, 7); bf.add('test'); print('test' in bf, 'nope' in bf)"

You should see True False printed. The string 'test' was added so it is found. The string 'nope' was never added so all its bit positions cannot all be set.

Seeing True True instead?

Check that your __contains__ method returns False when a bit is NOT set. The condition should be if not self._get_bit(position): return False. If you have the logic inverted, every lookup returns True.

Help me fix my bloom filter __contains__ method.

Add the optimal parameter calculator

Choosing the right number of bits and hash functions by gut feeling does not work. There are exact formulas that compute the optimal parameters from just two inputs: how many items you expect, and what false positive rate you want.

  • In bloom_filter.py, add this classmethod below the memory_bytes method:
    @classmethod
    def optimal(cls, expected_items, false_positive_rate):
        m = math.ceil(-expected_items * math.log(false_positive_rate) / (math.log(2) ** 2))
        k = max(1, round((m / expected_items) * math.log(2)))
        return cls(m, k)

What do the formulas compute?

  • m = ceil(-n * log(p) / (log(2)^2)) computes the optimal number of bits. For 100K items at 1% false positive rate, this gives ~958,506 bits (~117 KB).
  • k = round((m/n) * log(2)) computes the optimal number of hash functions. For those same parameters, this gives 7.
  • The classmethod returns a fully configured BloomFilter instance so you never have to pick parameters by hand.
  • Save bloom_filter.py.
  • Test the optimal calculator by running this command:
python3 -c "from bloom_filter import BloomFilter; bf = BloomFilter.optimal(100_000, 0.01); print(f'Bits: {bf.size}, Hashes: {bf.num_hashes}, Memory: {bf.memory_bytes()} bytes')"

You should see the filter computed ~958,506 bits, 7 hash functions, and about 119,814 bytes (~117 KB). That is the entire memory budget for checking 100,000 URLs.

Getting a different number of hashes?

Make sure you have max(1, round(...)) wrapping the k calculation. Without max(1, ...), very small inputs could round down to zero hash functions.

Help me troubleshoot my optimal method.

✔️ Awesome, I've got everything!

Great. Double check you have saved bloom_filter.py and your file contains all three sections you built above.

ⓧ I'd like to double check the full code

Scroll up and confirm your bloom_filter.py contains these three sections in order:

  • Imports (hashlib, math) plus the class definition with __init__, _set_bit, and _get_bit.
  • The _get_positions double-hashing method, add(), __contains__(), and memory_bytes().
  • The optimal classmethod at the bottom.

If any section is missing, scroll up to the matching chunk and paste it in at the correct position inside the class.

Before you run the final verification, what do you think will happen if you check for a string you never added to the filter? Will it ever say True?

  • Run the end-of-step verification:
python3 -c "from bloom_filter import BloomFilter; bf = BloomFilter(1000, 7); bf.add('test'); print('test' in bf, 'nope' in bf)"

You see True False. The filter correctly identifies that 'test' was added and that 'nope' was definitely never added. With only 1000 bits and 1 item, a false positive on a random string is very unlikely.

Your bloom filter is complete. Next up, you will plug it into the two-tier safety checker pattern and see what happens when you pick parameters by gut feeling instead of trusting the math.

Wire the Two-Tier Safety Checker

You have a working bloom filter. Now it's time to plug it into the same architecture Google Chrome uses to protect billions of users from malicious URLs.

The pattern is called two-tier checking. A bloom filter acts as the fast first layer, and an expensive full-database lookup serves as the second layer. You'll wire this up with parameters that seem reasonable at first glance, then see how the system actually performs.

In this step, get ready to:
  • Build a two-tier URL checker using the Chrome Safe Browsing pattern.
  • Run the checker with hand-picked parameters and observe the results.
Create safety_checker.py with data loading

The two-tier architecture needs three helpers: one to load the malicious URL database, one to load labeled test URLs, and one to simulate the expensive fallback lookup that happens when the bloom filter says "maybe."

  • Create a new file called safety_checker.py in your bloom-filter-safety folder.
  • Add the imports and helper functions by pasting this code:
import tracemalloc
import time
from bloom_filter import BloomFilter


# Load all malicious URLs as a list
def load_malicious_urls():
    with open("malicious_urls.txt", "r") as f:
        return [line.strip() for line in f]


# Load labeled test URLs for accuracy checking
def load_test_urls():
    test_urls = []
    with open("test_urls.txt", "r") as f:
        for line in f:
            label, url = line.strip().split("|", 1)
            test_urls.append((label, url))
    return test_urls


# Simulate the expensive lookup the bloom filter aims to avoid
def expensive_database_lookup(url, malicious_set):
    return url in malicious_set

What does this code do?

  • load_malicious_urls() reads every URL from malicious_urls.txt into a list. It returns a list (not a set) so you can measure set memory separately later.
  • load_test_urls() parses the labeled test file, splitting each line on the | delimiter to get the label and URL.
  • expensive_database_lookup() simulates the second-tier network call Chrome would make. In production, this would be a remote API call costing time and bandwidth.
  • Below the helper functions, add the main block that loads URLs and builds the bloom filter by pasting this code:

if __name__ == "__main__":
    print("=== Bloom Filter URL Safety Checker ===")
    print("Pattern: Chrome Safe Browsing (two-tier architecture)\n")

    tracemalloc.start()

    malicious_urls = load_malicious_urls()
    malicious_set = set(malicious_urls)

    # Build the Bloom Filter with BAD manually-chosen parameters
    bf = BloomFilter(100_000, 3)

    for url in malicious_urls:
        bf.add(url)

Why 100,000 bits and 3 hashes?

You have 100,000 URLs to store, so 100,000 bits seems like a natural fit. Three hash functions is a common default you might see in tutorials.

These are gut-feel numbers. The whole point of this step is to see whether intuition is enough, or whether the math matters.

  • Save safety_checker.py.
  • Run the file to confirm it loads without errors by running:
python3 safety_checker.py
  • You should see the header printed, then the script exits. No errors means the bloom filter loaded all 100K URLs successfully.

Seeing a ModuleNotFoundError?

  • Make sure bloom_filter.py is in the same folder as safety_checker.py. Python imports from the current directory.
  • Check that your terminal is inside the bloom-filter-safety folder. Run ls and confirm you see both files listed.

Why can't Python find my bloom_filter module?

Implement the two-tier check loop

The two-tier pattern works like this: the bloom filter is the bouncer at the door. If it says "definitely not in the set," the URL is safe and you skip the expensive lookup entirely. If it says "maybe in the set," you fall back to the full database check.

  • In safety_checker.py, add the two-tier check loop below the bf.add(url) loop by pasting this code:
    test_urls = load_test_urls()

    expensive_lookups = 0
    false_positives = 0
    safe_count = 0
    malicious_missed = 0

    for label, url in test_urls:
        if url in bf:
            # Bloom filter says "maybe" - need expensive check
            expensive_lookups += 1
            actually_malicious = expensive_database_lookup(url, malicious_set)
            if not actually_malicious:
                false_positives += 1
        else:
            # Bloom filter says "definitely not" - skip expensive check
            if label == "MALICIOUS":
                malicious_missed += 1
        if label == "SAFE":
            safe_count += 1

    total = len(test_urls)
    lookups_saved = total - expensive_lookups

    print("--- Results (BAD parameters) ---")
    print(f"Total URLs checked: {total}")
    print(f"Expensive lookups needed: {expensive_lookups}/{total}")
    print(f"Expensive lookups saved: {lookups_saved} ({100 * lookups_saved / total:.1f}%)")
    print(f"False positives: {false_positives}/{safe_count}")
    print(f"False positive rate: {100 * false_positives / max(1, safe_count):.1f}%")
    print(f"Bloom filter memory: {bf.memory_bytes() / 1024:.1f} KB")

    tracemalloc.stop()

What does this check loop do?

  • url in bf triggers your __contains__ method. If all hashed bits are set, it returns True ("maybe present").
  • When the bloom filter says "maybe," the code does the expensive fallback lookup against the full set. If the URL turns out to be safe, that's a false positive.
  • malicious_missed tracks false negatives. Bloom filters mathematically guarantee this stays at zero.
  • lookups_saved counts how many expensive checks the bloom filter eliminated. The higher this number, the more useful the filter is.
  • Save safety_checker.py.

Before you run this, make a prediction. You chose 100,000 bits for 100,000 URLs and 3 hash functions. Do you think the bloom filter will save most expensive lookups, or barely any?

  • Run the safety checker by running:
python3 safety_checker.py

You should see a false positive rate of around 86%. The bloom filter flagged almost every safe URL as "maybe malicious," forcing expensive lookups on nearly all of them.

That result is intentional. With only 1 bit per URL and 3 hashes, the bit array is approximately 95% full. Almost any combination of bit positions will already be set, so nearly every URL looks "present."

Why is 86% so catastrophic?

The whole point of the bloom filter was to avoid expensive lookups. At 86% false positives, you are still doing expensive lookups for almost every URL. The filter is barely better than not having one at all.

This proves that intuition fails for bloom filter sizing. You need the math. The next step fixes this with optimal parameters that drop the false positive rate to 1%.

Seeing a KeyError or IndexError?

  • Make sure the indentation of the new code is consistent with the if __name__ block (4 spaces in from the if line).
  • Confirm that malicious_urls.txt and test_urls.txt exist in the same folder. Run ls *.txt to check.

Help me debug my safety_checker.py output

✔️ Awesome, I've got everything!

Great. Double check you've saved safety_checker.py and that your output shows a false positive rate around 86%.

ⓧ I'd like to double check the full code

Here is the complete safety_checker.py file. Compare yours against it:

import tracemalloc
import time
from bloom_filter import BloomFilter


def load_malicious_urls():
    with open("malicious_urls.txt", "r") as f:
        return [line.strip() for line in f]


def load_test_urls():
    test_urls = []
    with open("test_urls.txt", "r") as f:
        for line in f:
            label, url = line.strip().split("|", 1)
            test_urls.append((label, url))
    return test_urls


def expensive_database_lookup(url, malicious_set):
    return url in malicious_set
if __name__ == "__main__":
    print("=== Bloom Filter URL Safety Checker ===")
    print("Pattern: Chrome Safe Browsing (two-tier architecture)\n")

    tracemalloc.start()

    malicious_urls = load_malicious_urls()
    malicious_set = set(malicious_urls)

    # Build the Bloom Filter with BAD manually-chosen parameters
    bf = BloomFilter(100_000, 3)

    for url in malicious_urls:
        bf.add(url)

    test_urls = load_test_urls()

    expensive_lookups = 0
    false_positives = 0
    safe_count = 0
    malicious_missed = 0

    for label, url in test_urls:
        if url in bf:
            expensive_lookups += 1
            actually_malicious = expensive_database_lookup(url, malicious_set)
            if not actually_malicious:
                false_positives += 1
        else:
            if label == "MALICIOUS":
                malicious_missed += 1
        if label == "SAFE":
            safe_count += 1
    total = len(test_urls)
    lookups_saved = total - expensive_lookups

    print("--- Results (BAD parameters) ---")
    print(f"Total URLs checked: {total}")
    print(f"Expensive lookups needed: {expensive_lookups}/{total}")
    print(f"Expensive lookups saved: {lookups_saved} ({100 * lookups_saved / total:.1f}%)")
    print(f"False positives: {false_positives}/{safe_count}")
    print(f"False positive rate: {100 * false_positives / max(1, safe_count):.1f}%")
    print(f"Bloom filter memory: {bf.memory_bytes() / 1024:.1f} KB")

    tracemalloc.stop()

Your bloom filter is technically working. It has zero false negatives. But with an 86% false positive rate, it is nearly useless as a performance optimization. Next up, you will apply the sizing math and watch that 86% collapse down to 1%.

Tune for Production

In the previous step, your bloom filter flagged ~86% of safe URLs as suspicious. That is worse than useless.

The problem is not the bloom filter itself. The problem is that 100,000 bits for 100,000 items fills the array to 95%, leaving almost no room for the math to work. In this step, you will let the formulas choose the right size and watch the false positive rate snap to exactly where you target it.

In this step, get ready to:
  • Replace gut-feel parameters with the optimal classmethod that computes the right size and hash count.
  • Add a memory comparison showing the ~80x reduction over a naive Python set.
  • Confirm zero false negatives and a ~1% false positive rate.
Replace bad parameters with optimal sizing

The optimal classmethod you built in Step 3 computes the ideal bit-array size (m) and hash count (k) from a target false positive rate. Time to use it.

  • In safety_checker.py, find this section inside the if __name__ == "__main__": block:
    # Build the Bloom Filter with BAD manually-chosen parameters
    bf = BloomFilter(100_000, 3)

    for url in malicious_urls:
        bf.add(url)
  • Replace that entire section with this code:
    # Build the Bloom Filter with OPTIMAL parameters
    print("Building Bloom Filter with optimal parameters...")
    print(f"  Target false positive rate: 1%")
    print(f"  Expected items (n): {len(malicious_urls):,}")

    bf = BloomFilter.optimal(
        expected_items=len(malicious_urls),
        false_positive_rate=0.01
    )

    print(f"  Optimal bits (m): {bf.size:,}")
    print(f"  Optimal hashes (k): {bf.num_hashes}")

    start = time.time()
    for url in malicious_urls:
        bf.add(url)
    build_time = time.time() - start

    print(f"  Built in {build_time:.2f}s")
    print(f"  Filter memory: {bf.memory_bytes() / 1024:.1f} KB")

What does this code do?

  • BloomFilter.optimal(expected_items=100_000, false_positive_rate=0.01) uses the formulas m = ceil(-n * log(p) / (log(2)^2)) and k = round((m/n) * log(2)) to compute the exact array size and hash count needed for a 1% false positive rate.
  • The print statements reveal the computed parameters so you can see what the math chose. For 100K items at 1% FP rate, it picks ~958,506 bits and 7 hash functions.
  • Build timing shows the filter loads all 100K URLs in a couple of seconds.

Before you run this, what do you think will happen to the false positive rate now that the math is choosing the parameters?

  • Save safety_checker.py and run it to see the immediate improvement:
python3 safety_checker.py

You should see the "Building Bloom Filter with optimal parameters" output showing ~958,506 bits and 7 hashes. The false positive rate in the results section drops from ~86% to around 1%.

Still seeing ~86% false positive rate?

  • Make sure you deleted the old bf = BloomFilter(100_000, 3) line entirely. If both lines exist, the second one overwrites the first.
  • Check that you saved the file before running it.
  • Help me debug my bloom filter parameters
Add the memory comparison and full results summary

The false positive rate is fixed, but the output does not yet show the most impressive part: how much memory the bloom filter saves compared to the naive set from Step 2.

  • In safety_checker.py, find the line that prints the filter memory:
    print(f"  Filter memory: {bf.memory_bytes() / 1024:.1f} KB")
  • Add these lines directly below it to compute and print the memory comparison:
    naive_memory = sum(len(url.encode("utf-8")) + 49 for url in malicious_urls)
    print(f"  Naive set would use: ~{naive_memory / 1024 / 1024:.1f} MB")
    print(f"  Memory reduction: ~{naive_memory // bf.memory_bytes()}x smaller\n")

    # Two-tier checking
    print("Checking test URLs (two-tier architecture)...\n")

What does this code do?

  • naive_memory estimates what a Python set would use: each URL's byte length plus ~49 bytes of set-entry overhead (hash, pointer, bucket). This gives you a direct apples-to-apples comparison.
  • The memory reduction line divides the naive estimate by the bloom filter's actual byte count, revealing the ~80x savings.

Now update the results section to include timing and the zero-false-negatives guarantee.

  • Find the existing check loop and results section. It starts with test_urls = load_test_urls() and ends at tracemalloc.stop(). Replace that entire section with this code:
    test_urls = load_test_urls()

    expensive_lookups = 0
    false_positives = 0
    safe_count = 0
    malicious_missed = 0

    start = time.time()
    for label, url in test_urls:
        if url in bf:
            expensive_lookups += 1
            actually_malicious = expensive_database_lookup(url, malicious_set)
            if not actually_malicious:
                false_positives += 1
        else:
            if label == "MALICIOUS":
                malicious_missed += 1

        if label == "SAFE":
            safe_count += 1

    check_time = time.time() - start
    total = len(test_urls)
    lookups_saved = total - expensive_lookups
  • Immediately below the lookups_saved line, add the final results printout:
    print("--- Results ---")
    print(f"Total URLs checked: {total}")
    print(f"Expensive lookups needed: {expensive_lookups}/{total}")
    print(f"Expensive lookups saved: {lookups_saved} ({100 * lookups_saved / total:.1f}%)")
    print(f"False positives (safe URLs wrongly flagged): {false_positives}/{safe_count}")
    print(f"False positive rate: {100 * false_positives / max(1, safe_count):.1f}%")
    print(f"Malicious URLs missed (false negatives): {malicious_missed}")
    print(f"Check time: {check_time:.4f}s")
    print(f"\nBloom filter memory: {bf.memory_bytes() / 1024:.1f} KB")
    print(f"Guarantee: Zero false negatives. Every malicious URL is always caught.")

    tracemalloc.stop()

What changed in the results section?

  • start = time.time() wraps the check loop so you can measure how fast the two-tier lookup runs.
  • The summary now explicitly prints malicious_missed to prove the bloom filter's guarantee: it never misses a genuinely malicious URL.
  • The final "Guarantee" line states the fundamental property in plain English for anyone reading the output.
  • Save safety_checker.py.

Before you run the final version, predict: how much memory do you think 100K URLs take in a naive Python set, and what fraction of expensive lookups do you think the bloom filter eliminates?

  • Run the completed safety checker:
python3 safety_checker.py

You should see output showing: optimal bits ~958,506 with 7 hashes, filter memory ~117 KB, naive set memory ~several MB, memory reduction ~80x, false positive rate ~1%, expensive lookups saved ~99%+, and zero malicious URLs missed.

Why does this matter in production?

This is exactly how Chrome Safe Browsing works. A tiny bloom filter ships to your browser and checks every URL locally in microseconds. Only the ~1% that trigger a "maybe malicious" response require an expensive network round-trip to Google's servers. The other 99% are confirmed safe instantly with zero network cost.

Seeing a higher false positive rate or errors?

  • If you see an IndentationError, make sure all lines inside the if __name__ block use exactly 4 spaces of indentation.
  • If the false positive rate is much higher than 1%, confirm you removed the old BloomFilter(100_000, 3) line and are using BloomFilter.optimal(...) instead.
  • If you see NameError: name 'check_time' is not defined, make sure start = time.time() appears directly before the for label, url in test_urls: loop.
  • Help me fix my safety_checker.py output

✔️ Awesome, I've got everything!

Great. Double check you have saved safety_checker.py and confirmed the output shows ~1% false positive rate and ~80x memory reduction.

ⓧ I'd like to double check the full code

Your safety_checker.py should have these sections in order. Scroll up and confirm each is present:

  • Imports: tracemalloc, time, and BloomFilter from bloom_filter.
  • Three helper functions: load_malicious_urls(), load_test_urls(), and expensive_database_lookup().
  • Optimal bloom filter construction: BloomFilter.optimal(expected_items=len(malicious_urls), false_positive_rate=0.01) with print statements showing computed m and k, build time, and filter memory.
  • Memory comparison: naive_memory calculation followed by prints showing the naive set estimate and the ~80x reduction.
  • Two-tier check loop: Wrapped in time.time() timing, checking the bloom filter first, falling back to expensive_database_lookup() on positives.
  • Results summary: Prints total checked, expensive lookups saved, false positives, false positive rate, malicious missed, check time, bloom filter memory, and the zero-false-negatives guarantee.

You now have a production-pattern URL safety system that checks 100K threats using 80x less memory, eliminates 99% of expensive lookups, and mathematically guarantees zero false negatives. Next up: the Secret Mission challenges you to support URL removal without rebuilding the entire filter.

Secret mission

Implement a Counting Bloom Filter

Your bloom filter catches malicious URLs, but what happens when a URL gets delisted? Standard bloom filters cannot delete items because clearing a bit might destroy another item's entry. Your mission: build a Counting Bloom Filter that supports removal, and measure the memory tradeoff.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally using Python's standard library, so there are no ongoing costs or cloud resources to worry about. Decide whether to keep your project files, or delete them entirely.

Resources you used:

  • Project folder bloom-filter-safety on your Desktop (contains all Python scripts and generated data files).

Keep everything running

No action needed. Choose this if you want to keep experimenting with your bloom filter, tweak parameters, or extend the project further.

  • Your project folder stays on your Desktop with all scripts and data files intact.
  • You can re-run any script at any time with python3.

Pause - I'll come back to this later

Since this is a local project with no running processes, there is nothing to pause. Your files sit on disk and use minimal space.

  • You can optionally delete the generated data files to free up a few MB by running this from inside your project folder:
rm malicious_urls.txt test_urls.txt

You can regenerate them any time by running python3 generate_urls.py.

Delete - I don't want to use this again

Remove the entire project folder and all its contents.

  • Delete the project folder by running the following command in your terminal.

macOS

cd ~/Desktop
rm -rf bloom-filter-safety

Windows

cd ~\Desktop
rmdir /s /q bloom-filter-safety
  • Confirm the folder is gone by running ls on your Desktop. You should no longer see bloom-filter-safety in the output.

Nice Work!

Nice Work!

You just built a production-pattern URL safety checker that protects against 100,000 malicious URLs using the same architecture that Chrome Safe Browsing uses to protect billions of users.

You've learned how to:

  • Build a bloom filter entirely from scratch using Python stdlib (bytearray, hashlib, math) and measure an 80x memory reduction from ~10MB to ~117KB while maintaining sub-1% false positive rates.
  • Implement the Chrome Safe Browsing two-tier architecture pattern where a bloom filter eliminates 99%+ of expensive database lookups, answering "definitely safe" in microseconds.
  • Experience firsthand how bad parameters destroy a bloom filter (86% false positive rate), then apply the optimal sizing math to prove the fundamental guarantee: zero false negatives, ever.
  • Secret Mission: Implemented a Counting Bloom Filter that supports removing delisted URLs without rebuilding from scratch, at the cost of 8x more memory.

Ready to quiz yourself?