Build a Duplicate-Matching Benchmark

Create a labeled Python benchmark for duplicate-matching precision and recall.

Introduction

30 Second Summary

A contact list can quietly hold the same person more than once. Those extra entries can distort reports or send the same customer repeated messages.

In this project, you will build a reproducible duplicate-matching evaluation harness using Python, pandas, plus Microsoft Excel. The finished workflow turns synthetic data plus human labels into honest precision and recall measurements while preserving an untouched held-out test set.

What You'll Build

You will run the finished workflow to turn a messy CRM-style contact list into a scored benchmark that shows exactly where a duplicate-matching rule succeeds or fails.

By the end of this project, you'll have:

  • A reproducible generator you can run to produce 1,040 synthetic contacts plus a 64-pair labeling worksheet.
  • A human-labeled gold set where every side-by-side pair has a duplicate decision plus a reason.
  • An exact-match benchmark that reports confusion counts, precision, plus recall from 48 tuning pairs. Its evidence files confirm that all 16 held-out pairs remain unscored.
  • Secret Mission: Add confidence labels to your decisions before creating a human-review queue for uncertain pairs.

Are there any prerequisites?

You need a 64-bit device running Windows 10 or 11. You also need internet access for setup plus Microsoft Excel.

Before We Start

Before any fuzzy matching or record merging begins, you need a trustworthy benchmark for measuring accuracy. This is your moment to commit to testing an exact email-or-phone matcher against human labels while keeping 16 held-out pairs untouched.

Set Up Your Python Workspace

Global Python packages can collide across projects. A clean workspace keeps this benchmark reproducible on another Windows computer.

You will use Visual Studio Code to keep the project files together. A virtual environment will isolate pandas from every other Python project on your device.

In this step, get ready to:
  • Install Python 3.15.2 through the Python install manager.
  • Install Visual Studio Code 1.143.1 before opening the lead-data-integrity-1a folder.
  • Create an isolated environment with pandas 3.1.0.
Install Python 3.15.2

The Python install manager handles Python runtimes on Windows. A version check reveals whether the required runtime is already available.

  • Press the Windows key to open Windows search.
  • Type Terminal and press Enter to open a fresh terminal.
  • Check your installed Python 3 runtime by running this command:
py -3 --version

What does this check do?

The py launcher selects an installed Python 3 runtime. The version output determines which setup path you need.

✔️ I see the required Python version

Python 3.15.2 is ready. You can keep this terminal open while you install the editor.

ⓧ I see an older Python version

An older runtime can remain on your device. The Python install manager adds 3.15.2 beside it.

  • Press the Windows key to open Windows search.
  • Type Microsoft Store and press Enter to open it.
  • Search for Python install manager.
  • Select the Python install manager from the results.
  • Click Install.
  • Return to the terminal after the installation completes.
  • Install the required runtime beside the older version by running this command:
py install 3.15.2

What does this command do?

The install manager downloads the standard Python runtime tagged 3.15.2. Your existing runtime remains available.

  • Confirm the new runtime by running the version check again:
py -3 --version

What should I see?

You should see Python 3.15.2 in the output. The launcher now selects the required Python 3 runtime.

Runtime installation failing?

Confirm that the Python install manager finished installing before retrying the runtime command.

Check that your device has an internet connection for the runtime download.

Ask for help with the installation error.

ⓧ The command is not found

Windows needs the Python install manager before it can provide the py command. The Microsoft Store supplies the supported manager.

  • Press the Windows key to open Windows search.
  • Type Microsoft Store and press Enter to open it.
  • Search for Python install manager.
  • Select the Python install manager from the results.
  • Click Install.
  • Close the current terminal after the installation completes.
  • Open a fresh terminal through Windows search.
  • Start Python for the first time by running this command:
python

What does the first launch do?

The first python launch installs the current runtime when no Python runtime exists. The first launch can take a few minutes while the runtime downloads.

  • Close the terminal after you see the Python interpreter prompt.
  • Open another fresh terminal through Windows search.
  • Confirm the installed runtime by running this command:
py -3 --version

What should I see?

You should see Python 3.15.2 in the output. The runtime is now available from fresh terminals.

Still missing the Python commands?

Confirm that the Store shows the Python install manager as installed.

Close every terminal window before opening a new one so Windows can load the new commands.

Ask for help with the missing commands.

Install Visual Studio Code and open the folder

Visual Studio Code gives this project one editor plus an integrated terminal. Checking its version first keeps the workspace aligned with the project instructions.

  • Press the Windows key to open Windows search.
  • Type Visual Studio Code and press Enter.
  • Click Help in the top menu if Visual Studio Code opens.
  • Select About to read the first listed version number.

✔️ I see the required VS Code version

Visual Studio Code 1.143.1 is ready. Keep the editor open for the project folder.

ⓧ I see an older VS Code version

The installed editor needs an update before you build the workspace. Release 1.143.1 provides the version used throughout this project.

  • Open the official Visual Studio Code 1.143 release page in your browser.
  • Download the Windows x64 build for release 1.143.1.
  • Open the downloaded installer.
  • Follow the installer prompts with the default options.
  • Press the Windows key after installation.
  • Type Visual Studio Code and press Enter.
  • Click Help in the top menu.
  • Select About to confirm version 1.143.1.

Update not showing in About?

Close every Visual Studio Code window before reopening the updated installation.

Ask for help if the older version remains.

ⓧ Visual Studio Code is not installed

The Windows user installer provides the editor without requiring a system-wide setup. You can install the pinned release from its official release page.

  • Open the official Visual Studio Code 1.143 release page in your browser.
  • Download the Windows x64 build for release 1.143.1.
  • Open the downloaded installer.
  • Follow the installer prompts with the default options.
  • Press the Windows key after installation.
  • Type Visual Studio Code and press Enter.
  • Click Help in the top menu.
  • Select About to confirm version 1.143.1.

Installer not opening?

Confirm that you downloaded the Windows x64 build from the release page.

Ask for help with the installer.

Your editor is installed. Now the project needs a named home on your Desktop.

  • Press the Windows key to open Windows search.
  • Type File Explorer and press Enter to open it.
  • Select Desktop in the navigation pane.
  • Click New in the toolbar.
  • Select Folder.
  • Type lead-data-integrity-1a and press Enter.
  • Switch back to Visual Studio Code.
  • Click File in the top menu.

Your project has a home

The lead-data-integrity-1a folder keeps the script and its generated evidence together. Opening the folder as a workspace also starts new terminals in the correct location.

  • Click Open Folder.
  • Select Desktop in the folder dialog.
  • Select the lead-data-integrity-1a folder.
  • Confirm the folder selection.

Asked whether you trust the folder?

Confirm that you trust the folder because you created it yourself. Visual Studio Code then enables the workspace features needed for the project.

You should see LEAD-DATA-INTEGRITY-1A at the top of the Explorer sidebar. The empty workspace is ready for its dependency file.

Create the isolated environment

A requirements file records the exact package version used by the benchmark. The virtual environment installs that package inside this project only.

  • Hover over the LEAD-DATA-INTEGRITY-1A heading in the Explorer sidebar.
  • Click the New File icon.
  • Type requirements.txt and press Enter.
  • Add the pinned pandas dependency by pasting this line:
pandas==3.1.0

What does this file do?

The double equals sign pins pandas to exactly 3.1.0. Reinstalling from this file gives the project the same dependency version each time.

  • Save requirements.txt by pressing Ctrl+S.
  • Confirm that requirements.txt appears in the Explorer sidebar.

✔️ Awesome, I've got everything!

Your dependency file is saved with the required pandas pin.

ⓧ I'd like to double check the full code

Compare your saved requirements.txt file with this complete version:

pandas==3.1.0

Requirements file missing?

Check that the filename is exactly requirements.txt. Windows must not add another file extension.

Ask for help if the file does not appear.

The activation command in this project targets Command Prompt. Setting that terminal profile avoids PowerShell execution-policy changes.

  • Press Ctrl+Shift+P to open the Command Palette.
  • Type Terminal: Select Default Profile.
  • Select Command Prompt.
  • Click Terminal in the top menu.
  • Click New Terminal.
  • Create the project environment by running this command:
python -m venv tutorial-env

What does this command do?

The venv module creates an isolated Python environment named tutorial-env. Packages installed after activation stay inside that environment.

  • Confirm that the tutorial-env folder appears in the Explorer sidebar.

Environment folder not appearing?

Confirm that the terminal path ends with lead-data-integrity-1a before retrying the command.

Ask for help with the environment creation error.

  • Activate the new environment by running this command:
tutorial-env\Scripts\activate

What does activation change?

Activation makes the terminal use the Python interpreter inside tutorial-env. The terminal prompt shows which environment receives new packages.

  • Confirm that the terminal prompt starts with (tutorial-env).

Activation command blocked?

Confirm that the terminal tab is using Command Prompt. A PowerShell terminal can apply different script rules.

Ask for help without changing your execution policy.

  • Install the dependency recorded in requirements.txt by running this command:
python -m pip install -r requirements.txt

What does this installation do?

The command asks the active environment's Python interpreter to run pip. Pip reads requirements.txt before installing pandas 3.1.0 with its required packages.

The package download can take a minute. Installation progress in the terminal shows that the environment is receiving the dependency.

Package installation failing?

Confirm that the terminal prompt still starts with (tutorial-env).

Confirm that requirements.txt contains the exact pandas pin from the full-code check.

Ask for help with the installation output.

Before you run the final check, do you expect both the Python runtime and the pinned pandas package to appear?

  • Verify the runtime plus the installed packages by running these commands:
py -3 --version
python -m pip list

What does the final check prove?

  • The first command confirms the Python 3 runtime available through the Windows launcher.
  • The second command lists packages inside the active project environment.

You should see Python 3.15.2 in the first result. The package list should include pandas version 3.1.0.

That is the setup hurdle cleared: your isolated workspace now has the exact runtime and dependency this benchmark needs.

Your workspace is ready. Next up, you will generate the messy lead dataset and see the first candidate duplicate pairs.

Generate the Lead Dataset

Your isolated Python workspace is ready. A duplicate matcher now needs trustworthy data with known patterns before you can measure its accuracy.

In this step, you will build a deterministic generator with pandas. It creates 1,040 synthetic contacts plus 64 candidate duplicate pairs for human review.

In this step, get ready to:
  • Add the complete data-generation script to your workspace.
  • Run the generator to create two reproducible CSV files.
  • Inspect the raw contacts plus the side-by-side candidate pairs.
Add the deterministic generator

A deterministic generator produces the same controlled records each time it runs. This gives every later evaluation the same starting point.

The script also contains the validation plus evaluation commands used in later steps. Keeping the workflow in one file makes the benchmark reproducible.

  • In the Explorer sidebar in Visual Studio Code, create lead_integrity_1a.py inside the lead-data-integrity-1a folder.
  • Choose the tab that matches the current contents of lead_integrity_1a.py:

✔️ Awesome, I've got everything!

Great. Your complete script is ready for a final save before you generate the dataset.

ⓧ I'd like to double check the full code

Use this complete file to populate lead_integrity_1a.py.

import json
import re
import sys
from pathlib import Path

import pandas as pd

BASE_DIR = Path(__file__).resolve().parent
DATA_DIR = BASE_DIR / "data"
OUTPUT_DIR = BASE_DIR / "outputs"
RAW_PATH = DATA_DIR / "raw_contacts.csv"
LABELS_PATH = DATA_DIR / "candidate_labels.csv"
TUNING_PATH = DATA_DIR / "tuning_pairs.csv"
HELD_OUT_PATH = DATA_DIR / "held_out_pairs_locked.csv"

RAW_COLUMNS = [
    "record_id",
    "first_name",
    "last_name",
    "email",
    "company_name",
    "company_domain",
    "phone",
    "source_system",
    "updated_at",
]

FIRST_NAMES = [
    "Avery", "Blake", "Casey", "Devon", "Emery",
    "Finley", "Gray", "Harper", "Indigo", "Jordan",
    "Kai", "Logan", "Morgan", "Noel", "Oakley",
    "Parker", "Quinn", "Riley", "Sage", "Taylor",
    "Uma", "Val", "Winter", "Xen", "Yael",
]

LAST_NAMES = [
    "Adams", "Baker", "Carter", "Diaz", "Evans",
    "Foster", "Garcia", "Hayes", "Irwin", "Jones",
    "Khan", "Lee", "Martin", "Nguyen", "Owens",
    "Patel", "Quinn", "Reed", "Singh", "Turner",
    "Usher", "Vega", "Walker", "Xu", "Young",
    "Zimmer", "Brooks", "Chen", "Davis", "Ellis",
    "Franklin", "Green", "Hill", "Ibrahim", "Kim",
    "Lopez", "Moore", "Nolan", "Price", "Ross",
]

COMPANY_PREFIXES = [
    "Northstar", "Bluefield", "Clearpath", "Daybreak", "Evergreen",
    "Fairview", "Goldline", "Highpoint", "Ironwood", "Juniper",
    "Keystone", "Lakeside", "Moonrise", "Newbridge", "Oakridge",
    "Pinecrest", "Quickstone", "Redwood", "Silverline", "Truepoint",
]

COMPANY_NOUNS = [
    "Analytics", "Systems", "Labs", "Advisory", "Software",
    "Networks", "Logistics", "Services", "Works", "Partners",
]


def ensure_directories():
    DATA_DIR.mkdir(parents=True, exist_ok=True)
    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)


def make_base_records():
    records = []
    for index in range(1000):
        first_name = FIRST_NAMES[index % len(FIRST_NAMES)]
        last_name = LAST_NAMES[(index // len(FIRST_NAMES)) % len(LAST_NAMES)]
        company_number = index % 200
        company_name = (
            f"{COMPANY_PREFIXES[company_number % len(COMPANY_PREFIXES)]} "
            f"{COMPANY_NOUNS[(company_number // len(COMPANY_PREFIXES)) % len(COMPANY_NOUNS)]} "
            f"{company_number:03d}"
        )
        company_domain = f"company-{company_number:03d}.example.com"
        records.append(
            {
                "record_id": f"C{index + 1:04d}",
                "first_name": first_name,
                "last_name": last_name,
                "email": (
                    f"{first_name.lower()}.{last_name.lower()}.{index:04d}"
                    f"@{company_domain}"
                ),
                "company_name": company_name,
                "company_domain": company_domain,
                "phone": f"000-000-{index:04d}",
                "source_system": ["web_form", "event_list", "partner_upload"][index % 3],
                "updated_at": f"2026-09-{(index % 28) + 1:02d}",
            }
        )
    return records


def mutate_duplicate(base_record, duplicate_number):
    duplicate = base_record.copy()
    duplicate["record_id"] = f"D{duplicate_number + 1:04d}"
    duplicate["source_system"] = "legacy_import"
    duplicate["updated_at"] = f"2026-10-{(duplicate_number % 27) + 1:02d}"

    mutation_type = duplicate_number % 4
    phone_digits = re.sub(r"\D", "", base_record["phone"])

    if mutation_type == 0:
        duplicate["email"] = f" {base_record['email'].upper()} "
        duplicate["phone"] = base_record["phone"].replace("-", " ")
    elif mutation_type == 1:
        local_part, domain = base_record["email"].split("@")
        duplicate["email"] = f"{local_part.replace('.', '', 1)}@{domain}"
        next_digit = str((int(phone_digits[-1]) + 1) % 10)
        duplicate["phone"] = phone_digits[:-1] + next_digit
    elif mutation_type == 2:
        duplicate["email"] = ""
        duplicate["phone"] = (
            f"({phone_digits[:3]}) {phone_digits[3:6]} {phone_digits[6:]}"
        )
    else:
        local_part, domain = base_record["email"].split("@")
        duplicate["email"] = f"{local_part}+alias@{domain}"
        duplicate["phone"] = ""
        duplicate["first_name"] = base_record["first_name"][:-1]
        duplicate["company_name"] = base_record["company_name"].replace(" ", "-", 1)

    return duplicate


def make_pair_row(pair_number, left_record, right_record):
    row = {"pair_id": f"P{pair_number:03d}"}
    for side, record in (("left", left_record), ("right", right_record)):
        for column in RAW_COLUMNS:
            row[f"{side}_{column}"] = record[column]
    row["label"] = ""
    row["label_reason"] = ""
    return row


def build_dataset_and_candidates():
    records = make_base_records()
    positive_pairs = []

    for duplicate_number in range(40):
        source_index = (duplicate_number * 23) % 1000
        base_record = records[source_index]
        duplicate = mutate_duplicate(base_record, duplicate_number)
        records.append(duplicate)
        positive_pairs.append((base_record, duplicate))

    negative_pairs = []
    for index in range(12):
        left_index = index * 25
        negative_pairs.append((records[left_index], records[left_index + 1]))

    for index in range(12):
        left_index = 12 + index
        negative_pairs.append((records[left_index], records[left_index + 200]))

    ordered_pairs = []
    for index, positive_pair in enumerate(positive_pairs):
        ordered_pairs.append(positive_pair)
        if index < len(negative_pairs):
            ordered_pairs.append(negative_pairs[index])

    candidates = [
        make_pair_row(pair_number, left_record, right_record)
        for pair_number, (left_record, right_record) in enumerate(ordered_pairs, start=1)
    ]
    return records, candidates


def generate():
    ensure_directories()
    records, candidates = build_dataset_and_candidates()
    pd.DataFrame(records, columns=RAW_COLUMNS).to_csv(
        RAW_PATH, index=False, encoding="utf-8-sig"
    )
    pd.DataFrame(candidates).to_csv(
        LABELS_PATH, index=False, encoding="utf-8-sig"
    )
    print(f"Created {len(records):,} raw rows: {RAW_PATH}")
    print(f"Created {len(candidates)} candidate pairs: {LABELS_PATH}")
    print("Next: label every pair in Excel, then run the validate command.")


def load_labeled_pairs(path):
    if not path.exists():
        raise FileNotFoundError(f"Missing required file: {path}")

    frame = pd.read_csv(path, dtype=str, keep_default_na=False, encoding="utf-8-sig")
    required_columns = {"pair_id", "label", "label_reason"}
    missing_columns = required_columns.difference(frame.columns)
    if missing_columns:
        raise ValueError(f"Missing columns: {sorted(missing_columns)}")

    rows = frame.to_dict("records")
    errors = []
    for row in rows:
        pair_id = row["pair_id"]
        row["label"] = row["label"].strip()
        row["label_reason"] = row["label_reason"].strip()
        if row["label"] not in {"0", "1"}:
            errors.append(f"{pair_id}: label must be 0 or 1")
        if not row["label_reason"]:
            errors.append(f"{pair_id}: label_reason is blank")

    if errors:
        preview = "\n".join(errors[:10])
        raise ValueError(f"Fix these labeling errors:\n{preview}")
    return rows


def validate_labels():
    rows = load_labeled_pairs(LABELS_PATH)
    duplicate_count = sum(row["label"] == "1" for row in rows)
    nonduplicate_count = len(rows) - duplicate_count
    if duplicate_count == 0 or nonduplicate_count == 0:
        raise ValueError("The gold set must contain both duplicate and non-duplicate labels.")
    print(f"Validated {len(rows)} complete labels.")
    print(f"Duplicate labels: {duplicate_count}")
    print(f"Non-duplicate labels: {nonduplicate_count}")
    print("No blank reasons found.")


def split_labels():
    ensure_directories()
    rows = load_labeled_pairs(LABELS_PATH)
    tuning_rows = []
    held_out_rows = []

    for row in rows:
        pair_number = int(row["pair_id"].removeprefix("P"))
        if pair_number % 8 in {0, 1}:
            row["split"] = "held_out"
            held_out_rows.append(row)
        else:
            row["split"] = "tuning"
            tuning_rows.append(row)

    pd.DataFrame(tuning_rows).to_csv(
        TUNING_PATH, index=False, encoding="utf-8-sig"
    )
    pd.DataFrame(held_out_rows).to_csv(
        HELD_OUT_PATH, index=False, encoding="utf-8-sig"
    )
    print(f"Tuning pairs: {len(tuning_rows)}")
    print(f"Held-out pairs locked: {len(held_out_rows)}")
    print("Predictions created: 0")


def normalize_email(value):
    return re.sub(r"\s+", "", value.strip().lower())


def normalize_phone(value):
    return re.sub(r"\D", "", value)


def baseline_prediction(row):
    left_email = normalize_email(row["left_email"])
    right_email = normalize_email(row["right_email"])
    left_phone = normalize_phone(row["left_phone"])
    right_phone = normalize_phone(row["right_phone"])

    exact_email = bool(left_email) and left_email == right_email
    exact_phone = bool(left_phone) and left_phone == right_phone
    return int(exact_email or exact_phone)


def confusion_counts(scored_rows):
    counts = {"tp": 0, "fp": 0, "fn": 0, "tn": 0}
    for row in scored_rows:
        actual = row["actual_duplicate"]
        predicted = row["predicted_duplicate"]
        if actual == 1 and predicted == 1:
            counts["tp"] += 1
        elif actual == 0 and predicted == 1:
            counts["fp"] += 1
        elif actual == 1 and predicted == 0:
            counts["fn"] += 1
        else:
            counts["tn"] += 1
    return counts


def evaluate():
    ensure_directories()
    if not TUNING_PATH.exists() or not HELD_OUT_PATH.exists():
        raise FileNotFoundError("Run the split command before evaluate.")

    tuning_rows = load_labeled_pairs(TUNING_PATH)
    raw_frame = pd.read_csv(
        RAW_PATH, dtype=str, keep_default_na=False, encoding="utf-8-sig"
    )
    held_out_frame = pd.read_csv(
        HELD_OUT_PATH, dtype=str, keep_default_na=False, encoding="utf-8-sig"
    )

    scored_rows = []
    for row in tuning_rows:
        scored = dict(row)
        scored["actual_duplicate"] = int(row["label"])
        scored["predicted_duplicate"] = baseline_prediction(row)
        scored_rows.append(scored)

    counts = confusion_counts(scored_rows)
    precision_denominator = counts["tp"] + counts["fp"]
    recall_denominator = counts["tp"] + counts["fn"]
    precision = counts["tp"] / precision_denominator if precision_denominator else 0.0
    recall = counts["tp"] / recall_denominator if recall_denominator else 0.0

    report_row = {
        "split": "tuning",
        "pairs": len(scored_rows),
        "true_positives": counts["tp"],
        "false_positives": counts["fp"],
        "false_negatives": counts["fn"],
        "true_negatives": counts["tn"],
        "precision": round(precision, 4),
        "recall": round(recall, 4),
        "precision_target": 0.95,
        "recall_target": 0.80,
        "precision_target_met": precision >= 0.95,
        "recall_target_met": recall >= 0.80,
    }

    scored_path = OUTPUT_DIR / "baseline_scored_tuning.csv"
    report_path = OUTPUT_DIR / "baseline_report.csv"
    manifest_path = OUTPUT_DIR / "run_manifest.json"

    pd.DataFrame(scored_rows).to_csv(
        scored_path, index=False, encoding="utf-8-sig"
    )
    pd.DataFrame([report_row]).to_csv(
        report_path, index=False, encoding="utf-8-sig"
    )

    manifest = {
        "project": "Lead-Data Integrity Engine 1A",
        "python_version": sys.version.split()[0],
        "pandas_version": pd.__version__,
        "raw_row_count": len(raw_frame),
        "expected_raw_row_count": 1040,
        "row_count_preserved": len(raw_frame) == 1040,
        "tuning_pairs_scored": len(scored_rows),
        "held_out_pair_count": len(held_out_frame),
        "held_out_pairs_scored": 0,
        "baseline_rule": (
            "Predict duplicate when normalized email matches or normalized phone matches."
        ),
        "precision_target": 0.95,
        "recall_target": 0.80,
    }
    manifest_path.write_text(
        json.dumps(manifest, indent=2), encoding="utf-8"
    )

    print(f"Tuning precision: {precision:.1%}")
    print(f"Tuning recall: {recall:.1%}")
    print(f"Held-out pairs scored: {manifest['held_out_pairs_scored']}")
    print(f"Row count preserved: {manifest['row_count_preserved']}")
    print(f"Created: {scored_path}")
    print(f"Created: {report_path}")
    print(f"Created: {manifest_path}")


def main():
    commands = {
        "generate": generate,
        "validate": validate_labels,
        "split": split_labels,
        "evaluate": evaluate,
    }
    if len(sys.argv) != 2 or sys.argv[1] not in commands:
        available = ", ".join(commands)
        raise SystemExit(f"Usage: python lead_integrity_1a.py [{available}]")
    commands[sys.argv[1]]()


if __name__ == "__main__":
    main()

This reference keeps the generator plus the later benchmark commands in one file. Only generate runs during this step.

How does the generator work?

  • The path constants keep generated data inside the data folder.
  • The make_base_records() function creates 1,000 structured contact records.
  • The mutate_duplicate() function produces 40 altered copies with controlled differences.
  • The make_pair_row() function places each left record beside its right record.
  • The build_dataset_and_candidates() function combines known duplicate pairs with known non-duplicate pairs.
  • The generate() function writes the two datasets as CSV files.
  • Save lead_integrity_1a.py by pressing Ctrl+S.
Generate the contact files

Synthetic data gives you realistic data-quality problems without exposing real contact details. The fixed name lists plus mutation rules make the output repeatable.

Before you run the generator, what two files do you expect it to create inside data?

  • Switch back to the activated VS Code terminal.
  • Generate the raw contacts plus candidate pairs by running:
python lead_integrity_1a.py generate

What does this command do?

The main() function routes the generate argument to generate().

The generator creates the required folders. It then writes the raw contacts plus labeling worksheet.

You should see Created 1,040 raw rows: followed by the raw-contact path. You should also see Created 64 candidate pairs: followed by the labeling path.

That is your first working data pipeline. One command now rebuilds the benchmark inputs from the same rules.

Did the generator fail?

  • Confirm the terminal prompt begins with (tutorial-env).
  • Confirm lead_integrity_1a.py is saved inside lead-data-integrity-1a.
  • Compare the bottom of your script with the full-code reference to catch missing indentation.

Ask for help with the exact terminal output: Help me debug my lead dataset generator.

Inspect the side-by-side pairs

The terminal confirms the file counts. Inspecting the files shows what those counts represent before you start labeling.

  • Expand the data folder in the Explorer sidebar.
  • Select raw_contacts.csv.

You should see the columns record_id, first_name, last_name, email, company_name, company_domain, phone, source_system, plus updated_at.

  • Scroll to the final record in raw_contacts.csv.

The first 1,000 records use IDs from C0001 through C1000. The final 40 records use IDs from D0001 through D0040.

Before you inspect the candidate worksheet, how do you expect one pair to present its two contacts?

  • Select candidate_labels.csv in the data folder.
  • Scroll horizontally across the candidate columns.
  • Scroll to pair P064 at the bottom of the file.

You should see pair IDs from P001 through P064. Each row presents left_ fields beside matching right_ fields.

The label plus label_reason fields are blank. That visible gap is where your human judgment enters the benchmark in the next step.

You now have 1,040 synthetic contacts plus 64 review-ready pairs. Next, you will use Excel to turn those candidate pairs into a labeled gold set.

Label the Gold Set

The generator has given you 64 candidate duplicate pairs. Each pair needs a human decision before the file can serve as a gold set.

Microsoft Excel gives you a side-by-side surface for those decisions. You will classify each pair as the same person or different people.

You will also record the evidence behind each choice. The tuning examples still share one CSV file with the final test examples.

Using this combined file for rule changes would leak test information. Rule changes must wait until the final test rows are separated.

In this step, get ready to:
  • Classify all 64 candidate pairs in Microsoft Excel.
  • Document a reason for every human decision.
  • Confirm the completed gold set passes validation after saving.
Decide which pairs represent one person

A duplicate label means both records represent the same person despite a data-quality difference. A non-duplicate label means the records represent different people.

How to judge a pair

  • Treat matching identity details as evidence for 1 even when spaces or punctuation differ.
  • Treat conflicting person details as evidence for 0 even when the contacts share a company.
  • Use several fields when one value is blank or ambiguous.

The worksheet is wide, so finding the two labeling columns is fiddly. They sit after right_updated_at at the far right.

  • Switch back to data\candidate_labels.csv from earlier in Microsoft Excel.
  • Scroll horizontally to the label column.
  • Compare the left and right identity fields for P001.
  • Enter the appropriate value in label. Use 1 for the same person. Use 0 for different people.
  • Repeat the comparison for every row through P064.
  • Review the completed label column for values outside 0 or 1.

You should see all 64 cells in the label column filled with either 0 or 1.

Record reasons and save the CSV

A label records your decision. The label_reason column preserves the evidence that led to it.

  • Type a short explanation into the label_reason cell for every pair.
  • Review all 64 label_reason cells for blanks.

You should now see a completed decision beside a completed reason for every pair.

Replacing the blank candidate file can feel risky. Your labels belong in that exact file while data\raw_contacts.csv stays unchanged.

  • Click File in Microsoft Excel.
  • Select Save As.

You will see Excel's save options for the active worksheet.

  • Select the lead-data-integrity-1a\data folder as the save location.
  • Enter candidate_labels.csv in the File name field.
  • Choose CSV (Comma delimited) (*.csv) as the file type.

The save target should now point to the existing data\candidate_labels.csv file.

  • Click Save.
  • Approve the prompt to replace the existing candidate_labels.csv file.
  • Confirm Excel's CSV warning so the active worksheet stays in comma-delimited format.
Validate every completed label

The validator checks the structure and completeness of your labels. Your side-by-side comparison provides the human judgment behind them.

Before you run the check, do you expect all 64 rows to pass on the first attempt? Hold that prediction for the result.

  • Return to the Visual Studio Code terminal from earlier.
  • Validate the completed candidate file by running this command:
python lead_integrity_1a.py validate

What does validation check?

  • The load_labeled_pairs() function reads data\candidate_labels.csv.
  • Each label value must be either 0 or 1.
  • Each label_reason value must contain text.
  • The validate_labels() function confirms that both label classes are present.

When every label is complete, you will see Validated 64 complete labels. followed by Duplicate labels: and Non-duplicate labels: with nonzero counts.

The final line says No blank reasons found. That is the key checkpoint done: your 64-pair gold set now has complete human decisions.

Did validation name a pair?

  • Note every pair ID named in the terminal.
  • Switch back to data\candidate_labels.csv in Microsoft Excel.
  • Correct each named label or reason.
  • Save the active worksheet as the same comma-delimited CSV again.
  • Return to the Visual Studio Code terminal.
  • Run the same validation command again.

If the command still reports a problem, help me diagnose the failed label validation.

Your gold set now contains 64 labeled decisions with supporting reasons. Next, you will separate the tuning pairs from a locked held-out test set.

Lock the Held-Out Test Set

Your 64 candidate pairs now have human labels. Keeping all 64 decisions in one CSV leaves the final test cases exposed to future tuning.

A held-out test set gives you an untouched benchmark. This step uses a deterministic pair ID rule to separate 48 tuning pairs from 16 test pairs.

In this step, get ready to:
  • Create a 48-pair tuning file using the deterministic split rule.
  • Create a locked 16-pair held-out test file.
  • Confirm that neither file contains baseline predictions.
Split the labeled pairs

A deterministic split assigns the same pair to the same file every time. This reproducibility prevents a convenient reshuffle after you see the results.

Before you run the split, what pair counts do you expect the deterministic rule to produce?

  • Create the two split files by running this command in the VS Code terminal:
python lead_integrity_1a.py split

What does this command do?

  • The split command calls split_labels() to load the completed labels.
  • The modulo rule places pair numbers with remainders 0 or 1 in data\held_out_pairs_locked.csv.
  • Every other pair goes into data\tuning_pairs.csv.
  • The command adds the split column without creating prediction columns.

You will see Tuning pairs: 48. You will also see Held-out pairs locked: 16.

The terminal reports Predictions created: 0. That split is the safeguard your benchmark needed.

Split command failed?

  • Confirm the VS Code terminal is still inside the lead-data-integrity-1a folder.
  • Confirm data\candidate_labels.csv still contains all 64 completed labels.
  • Rerun the validation command from the previous step if the terminal points to incomplete labels.

Help me diagnose why the deterministic split command failed.

Inspect only the tuning file

The tuning file is safe to inspect during rule development. The held-out file stays closed so its labels cannot influence future changes.

  • In VS Code's Explorer sidebar, expand the data folder.
  • Click tuning_pairs.csv to open it.
  • Inspect the split column to confirm each visible value is tuning.
  • Inspect the header row to confirm no actual_duplicate or predicted_duplicate columns exist.
  • Close the tuning_pairs.csv tab.
  • Keep held_out_pairs_locked.csv closed.

Before the final run, do you expect the pair counts to change when the same deterministic split runs again?

  • Verify the split remains reproducible by running the command again:
python lead_integrity_1a.py split

Why run the split twice?

The same source labels pass through the same pair ID rule. Matching counts prove that rerunning the workflow does not reshuffle the benchmark.

You will see Tuning pairs: 48. You will also see Held-out pairs locked: 16.

The final line says Predictions created: 0. Your held-out test set is now locked away from baseline scoring.

Counts changed after the rerun?

  • Return to data\candidate_labels.csv to confirm its pair IDs still run from P001 through P064.
  • Compare split_labels() in lead_integrity_1a.py with the Complete Code if the modulo rule was edited.

Help me find why my split counts are no longer 48 tuning pairs and 16 held-out pairs.

Your benchmark now has a protected final test set. Next, you will measure the exact-match baseline using only the tuning file.

Measure the Exact-Match Baseline

Your gold set now has 48 tuning pairs. The 16 held-out pairs remain locked away from scoring.

An exact-match baseline gives you a transparent starting point. Its conservative rule exposes the typo variants that slip through while protecting the held-out test for later.

In this step, get ready to:
  • Score only the tuning pairs with the normalized exact-email-or-phone rule.
  • Inspect confusion counts plus precision plus recall plus one false negative.
  • Confirm the row-count evidence plus the untouched held-out status in the run manifest.
Run the tuning-only baseline

Normalization removes email whitespace plus letter-case differences. It also removes non-digit phone formatting before the rule checks for exact equality.

The rule predicts a duplicate when either normalized value is nonempty plus equal on both sides. Only the 48 rows in data/tuning_pairs.csv receive predictions.

Before you run this, predict whether the conservative rule will find every human-labeled duplicate.

  • Evaluate the tuning split from the activated tutorial-env terminal by running:
python lead_integrity_1a.py evaluate

What does this evaluation do?

  • The baseline_prediction() function compares normalized email values plus normalized phone values.
  • The confusion_counts() function separates correct predictions from missed duplicates plus incorrect matches.
  • The evaluation writes predictions to outputs/baseline_scored_tuning.csv. The original data/tuning_pairs.csv remains unchanged.
  • The held-out rows are counted for evidence. They never pass through the prediction rule.

You will see tuning precision of 100.0%. You will also see tuning recall of 40.0%.

The terminal also reports Held-out pairs scored: 0 plus Row count preserved: True.

You now have your first trustworthy benchmark. It measures the tuning data while preserving the held-out test.

Seeing an Evaluation Error?

  • Confirm the terminal prompt begins with (tutorial-env).
  • Check that data/tuning_pairs.csv exists inside lead-data-integrity-1a.
  • Check that data/held_out_pairs_locked.csv exists inside the same folder.

Help me troubleshoot this evaluation command.

Review the metrics and a false negative

A confusion matrix separates correct duplicate predictions from false alarms plus missed duplicates. This makes the baseline's strengths plus gaps visible.

Precision uses tp / (tp + fp). Recall uses tp / (tp + fn).

  • Return to Microsoft Excel from the labeling step.
  • Open outputs/baseline_report.csv from the lead-data-integrity-1a folder using Excel's file-opening screen.
  • Read the single tuning row from left to right.

You should see 48 pairs. The confusion counts are 12 true positives plus 0 false positives plus 18 false negatives plus 18 true negatives.

Precision is 1.0. The precision_target_met field is True.

Recall is 0.4. The recall_target_met field is False.

  • Open outputs/baseline_scored_tuning.csv in Excel.
  • Locate the row with pair_id set to P003.
  • Compare its actual_duplicate value with its predicted_duplicate value.
  • Review the left plus right email fields.
  • Review the left plus right phone fields.

The P003 row has actual_duplicate set to 1. Its predicted_duplicate value is 0.

This row is a false negative. Its altered email separator plus changed phone digit leave the exact-match rule with no equal value.

Verify the reproducibility guardrails

A JSON run manifest records the software versions plus the dataset counts behind the result. This gives a later run a fixed reference for comparison.

Before you inspect the manifest, predict whether its safeguards will match the terminal output.

  • Return to Visual Studio Code from earlier.
  • Open outputs/run_manifest.json from the file sidebar.
  • Read the version fields near the top of the manifest.
  • Read the raw-row plus tuning-pair count fields.
  • Read the held-out count plus held-out scoring fields.
  • Read the value beside baseline_rule.

The manifest should record Python 3.15.2 plus pandas 3.1.0.

You should see raw_row_count set to 1040. The row_count_preserved field should be true.

You should see tuning_pairs_scored set to 48. The held_out_pair_count field should be 16.

The final safeguard is held_out_pairs_scored set to 0. This proves the baseline never consumed the locked test cases.

Your evaluation evidence is complete. It preserves all 1,040 raw rows.

The tuning metrics are reproducible. The held-out test remains untouched for the later final evaluation.

Secret mission

Build a Human-Review Queue

Some duplicate decisions carry more uncertainty than others. Add confidence to every tuning label. Turn the uncertain decisions into a review queue without changing the baseline or scoring the held-out pairs.

Clean Up Your Resources

Clean Up Your Resources

Your benchmark lives on your local Windows device with no ongoing cloud costs. Choose whether to keep the workspace available, pause your local apps, or delete the project folder entirely.

Resources you used:

  • The lead-data-integrity-1a project folder on your Windows device.
  • The tutorial-env virtual environment stored inside the project folder.
  • The data folder containing the generated contacts plus the labeled tuning and held-out files.
  • The outputs folder containing the baseline evidence plus the human-review queue.

Keep everything running

Keeping the workspace needs no action. Choose this option if you are still reviewing the benchmark or preparing to continue with it.

Keep data/held_out_pairs_locked.csv unchanged so its 16 pairs remain suitable for a later final evaluation.

Pause - I'll come back to this later

Shut down the open apps to free local memory while keeping every project file on disk.

  • Close Microsoft Excel.
  • Close Visual Studio Code.

Closing Visual Studio Code ends the active terminal session. Your script plus every CSV and JSON artifact remain available for your next session.

Delete - I don't want to use this again

Deleting the project folder is permanent. Your separately installed apps remain available.

The full folder path can be fiddly to spot because Windows may shorten its display. A correct path ends with lead-data-integrity-1a.

  • Record the full project folder path from where you saved it: full path to lead-data-integrity-1a.
  • Close Microsoft Excel.
  • Close Visual Studio Code.
  • Press the Windows key to open the search bar.
  • Type PowerShell into the search bar.
  • Press Enter to open PowerShell.
  • Delete the entire project folder by running this command:
Remove-Item -Recurse -Force "[[PROJECT_FOLDER_PATH=\"full path to lead-data-integrity-1a\"]]"

What Does This Command Remove?

This command removes the lead-data-integrity-1a folder plus its virtual environment and every project artifact. Python plus your separately installed apps remain available.

  • Return to the folder location where you saved the project in File Explorer.

You should no longer see the lead-data-integrity-1a folder. Your local benchmark has now been removed.

Folder Still There?

  • Close any window that still has a project file open.
  • Run the same PowerShell command again.

Help me remove the local project folder.

Nice Work!

Nice Work!

You did it! Your duplicate-matching evaluation harness now uses Python and pandas to test a conservative matcher against human labels while protecting the held-out set.

You've learned how to:

  • Generate a deterministic synthetic dataset with 1,040 CRM-style contact records. You also produced 64 candidate pairs for human review.
  • Create a human-labeled gold set with explicit reasons for every decision. You protected the final test by locking away 16 held-out pairs.
  • Measure an exact-match baseline on 48 tuning pairs. You recorded confusion matrix counts. You preserved precision, recall, row-count checks, and run metadata as reusable evidence.
  • Complete the optional Secret Mission by adding confidence labels to every tuning pair. You created a human-review queue containing only medium- or low-confidence cases. You kept all 16 held-out pairs unchanged and unscored.

Ready to quiz yourself?