Expose Misleading AI Accuracy

Build a reproducible AI evaluation lab and cloud-ready approval pipeline.

Introduction

30 Second Summary

A headline score can make an automated decision look dependable. The same system may still fail every rare case that matters most.

In this project, you will build a reproducible experiment in Python that exposes how high accuracy can hide total failure on a minority class. You will publish the evidence as a GitHub case study with a provider-neutral cloud evaluation architecture.

What You'll Build

Picture opening your GitHub case study to show a rendered cloud approval pipeline built around evidence that catches hidden model failures.

By the end of this project, you'll have:

  • A reproducible baseline experiment you can run to expose high accuracy beside zero minority-class detections.
  • An evidence-based model comparison that produces JSON evidence plus a readable Markdown report with balanced accuracy and confusion matrices.
  • A public GitHub case study that shows how your evidence becomes a rendered cloud approval pipeline.
  • Secret Mission: Test whether balanced accuracy still tells the truth when the class distribution shifts from 90:10 to 97:3.

Are there any prerequisites?

Bring basic Python familiarity, Visual Studio Code, and a GitHub account. The guide walks you through dependency setup without using a cloud account.

Before We Start

This checkpoint gives you a clear research question before the hands-on work begins. You will commit to testing whether high ordinary accuracy reflects useful minority-class behavior using local Python experiments published through GitHub.

Set Up a Reproducible Lab

An impressive metric only means something when another person can recreate the experiment. This lab needs an isolated dependency with an exact version before its results can be trusted.

You will prepare the project in Visual Studio Code. You will verify Python before installing a pinned version of scikit-learn.

In this step, get ready to:
  • Verify that Python 3.11 or newer is available.
  • Create the project folder with an isolated virtual environment.
  • Install scikit-learn 1.9.1 from a pinned dependency file.
Create your project workspace and verify Python

A Visual Studio Code workspace keeps the project files with the terminal that runs them. Creating the workspace first gives every later command the correct folder context.

  • Press Cmd+Space on macOS or the Windows key on Windows to open your search bar.
  • Type VS Code and press Enter to open it.
  • Select File in the menu bar.
  • Select Open Folder....
  • Select your Desktop in the folder dialog.
  • Select New Folder.
  • Type misleading-accuracy-lab as the folder name.
  • Press Enter to create the folder.

Why Open a Workspace?

A workspace gives the editor with its terminal the same folder context. New terminals start inside misleading-accuracy-lab once that folder is open.

  • Select Open on macOS or Select Folder on Windows.
  • Select Yes, I trust the authors if the Workspace Trust dialog appears.
  • Select View in the menu bar.
  • Select Terminal to open the integrated terminal.
  • Check your installed Python version by running:
python3 --version

What Does This Check?

This command asks the Python 3 executable to report its version. Scikit-learn 1.9.1 requires Python 3.11 or newer.

  • Choose the tab that matches the version result in your terminal.

✔️ I see the required version or higher

Your Python version meets the project requirement. That clears the first reproducibility check.

  • Continue to the dependency setup below.

ⓧ I see an older version

This project needs Python 3.11 or newer. Install a current 64-bit Python 3 release before creating the environment.

  • Visit the official Python downloads page.
  • Choose a current 64-bit Python 3 release.
  • Complete the Python installer.
  • Restart Visual Studio Code after the installation.
  • Return to the misleading-accuracy-lab workspace.
  • Open a fresh integrated terminal from the View menu.
  • Check the new Python version by running:
python3 --version

What Should Change?

The fresh terminal reads the Python installation available to Visual Studio Code. Continue when it reports Python 3.11 or newer.

Still Seeing the Older Version?

Your terminal may still be finding an earlier Python installation. Help me find why Visual Studio Code still uses my older Python version or share the version output with the NextWork community.

ⓧ Command not found

Your terminal cannot find a Python 3 executable. Install a current 64-bit Python 3 release before continuing.

  • Visit the official Python downloads page.
  • Choose a current 64-bit Python 3 release.
  • Complete the Python installer.
  • Restart Visual Studio Code after the installation.
  • Return to the misleading-accuracy-lab workspace.
  • Open a fresh integrated terminal from the View menu.
  • Confirm that Python is now available by running:
python3 --version

What Should Change?

The command now reports the installed Python version. Continue when it shows Python 3.11 or newer.

Still Missing Python?

The installer may not be visible to the shell used by Visual Studio Code. Help me make Python 3 available in the Visual Studio Code terminal or share the terminal output with the NextWork community.

Pin the dependency and create the environment

A virtual environment keeps this lab's packages separate from other Python projects. The dependency file records the exact package version needed to reproduce the experiment.

  • Select New File in the Visual Studio Code Explorer.
  • Type requirements.txt as the file name.
  • Press Enter to create the file.
  • Pin the project dependency by pasting this exact line into requirements.txt:
scikit-learn==1.9.1

What Does This Pin Do?

The double equals sign tells pip to install exactly scikit-learn 1.9.1. This prevents a future release from changing the experiment unexpectedly.

  • Save requirements.txt.
  • Check that requirements.txt remains visible beneath misleading-accuracy-lab in the Explorer.

You should see requirements.txt in the project folder with one pinned dependency.

Dependency File Missing?

Check that the Explorer shows misleading-accuracy-lab as the open folder. Make sure the file name is exactly requirements.txt.

Help me locate or fix my requirements file

✔️ Awesome, I've got everything!

Your dependency pin is saved. The project now records the package version used by the experiment.

ⓧ I'd like to double check the full code

  • Compare your complete requirements.txt file with this reference.
scikit-learn==1.9.1

What Should Match?

The file contains one line that pins scikit-learn to 1.9.1. No other dependencies belong in this file.

  • Create the isolated sklearn-env environment by running:
python3 -m venv sklearn-env

What Does This Command Do?

Python creates an isolated environment inside the sklearn-env folder. Packages installed after activation stay inside this project environment.

  • Check the Visual Studio Code Explorer for the new sklearn-env folder.

You should see sklearn-env beneath the project folder. That folder is the visible boundary around this lab's Python packages.

Environment Folder Missing?

Confirm that the terminal is inside the misleading-accuracy-lab workspace. A missing folder can also mean the environment creation command ended with an error.

Help me diagnose why sklearn-env was not created

  • Activate the project environment by running:
source sklearn-env/bin/activate

What Does Activation Change?

The source command updates the current terminal session to use the environment's Python tools. Future package commands now target sklearn-env.

  • Check the beginning of your terminal prompt for sklearn-env.

You should see (sklearn-env) at the start of the prompt. The isolated environment is now active.

Environment Not Active?

Check that sklearn-env appears in the Explorer. Confirm that the activation command ran from inside misleading-accuracy-lab.

Help me activate sklearn-env in my terminal

Install and verify scikit-learn

The active environment is ready to receive the pinned dependency. Installing from requirements.txt ties the package on your computer to the version recorded in the project.

  • Install the pinned dependency into sklearn-env by running:
python -m pip install -r requirements.txt

What Does This Installation Do?

The command reads requirements.txt through the active environment's Python interpreter. It installs scikit-learn 1.9.1 with the packages that release needs.

The terminal should finish the installation without an error. Your prompt should still begin with (sklearn-env).

Installation Did Not Finish?

Confirm that the terminal prompt begins with (sklearn-env). Check that requirements.txt contains the exact pin from the reference tab.

Help me debug the scikit-learn installation or share the terminal output with the NextWork community.

Before you check, which scikit-learn version do you expect the active environment to report?

  • Verify the installed package by running:
python -m pip show scikit-learn

What Does This Verification Prove?

The command reports the installed package version with its location. A path containing sklearn-env confirms that the package belongs to this project's isolated environment.

You should see Version: 1.9.1 in the output. You should also see an installation location inside sklearn-env.

Seeing the Wrong Version?

Check that the prompt begins with (sklearn-env) before running the verification. An installation location outside sklearn-env means another Python environment handled the command.

Help me verify scikit-learn inside sklearn-env

That is the lab locked down. Python meets the requirement with scikit-learn installed at the recorded version.

Your reproducible environment is ready. Next, you will run a high-accuracy baseline that hides a serious failure.

Run the High-Accuracy Baseline

Your reproducible scikit-learn environment is ready. Now you can test whether a polished metric reflects useful model behavior.

A most-frequent baseline gives you a fast result under class imbalance. You will compare its accuracy with a confusion matrix.

In this step, get ready to:
  • Generate a reproducible 90:10 classification dataset.
  • Fit a most-frequent dummy baseline.
  • Read the baseline's prediction pattern in a confusion matrix.
Build a reproducible dataset

Synthetic data keeps the experiment self-contained. Fixed random states make the generated samples reproducible.

A stratified holdout keeps the 90:10 class proportion across the training data and test data. This gives the baseline a consistent evaluation set.

  • Click the New File button in the VS Code Explorer sidebar.
  • Name the file baseline.py.

You should see an empty baseline.py tab in the editor.

  • Build the dataset by pasting this code into baseline.py:
from sklearn.datasets import make_classification
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score, confusion_matrix
from sklearn.model_selection import train_test_split


def build_dataset():
    X, y = make_classification(
        n_samples=1000,
        n_features=6,
        n_informative=4,
        n_redundant=0,
        n_clusters_per_class=1,
        weights=[0.9, 0.1],
        flip_y=0.0,
        class_sep=2.0,
        random_state=42,
    )
    return train_test_split(
        X,
        y,
        test_size=0.3,
        random_state=42,
        stratify=y,
    )

What does this code do?

  • The imports provide the dataset generator, baseline model, evaluation metrics, and split helper.
  • The make_classification() call creates 1,000 synthetic samples.
  • The weights=[0.9, 0.1] value creates the 90:10 class distribution.
  • The random_state=42 values make dataset generation reproducible.
  • The stratify=y value preserves the class proportions during the split.
  • Save baseline.py.
  • Confirm baseline.py appears beside requirements.txt in the Explorer sidebar.

That is the reproducible foundation in place: baseline.py now holds the dataset builder.

Missing the dataset file?

Check that baseline.py sits directly inside misleading-accuracy-lab. A nested file can make the later run command miss your script.

Confirm the editor tab does not show an unsaved-change indicator.

Help me find or save my baseline file.

Run the misleading baseline

The most-frequent strategy learns which class dominates the training data. It uses that class for every prediction.

The script prints two views of the same model behavior. The comparison tests whether the summary score tells the full story.

  • Add the model evaluation directly below build_dataset() in baseline.py by pasting this code:
def main():
    X_train, X_test, y_train, y_test = build_dataset()
    baseline = DummyClassifier(strategy="most_frequent")
    baseline.fit(X_train, y_train)
    predictions = baseline.predict(X_test)

    print("Most-frequent baseline")
    print(f"Accuracy: {accuracy_score(y_test, predictions):.3f}")
    print("Confusion matrix: true classes are rows")
    print(confusion_matrix(y_test, predictions))


if __name__ == "__main__":
    main()

What does this code do?

  • The build_dataset() call supplies four arrays for model fitting and evaluation.
  • The baseline model always selects the most frequent training class.
  • The predictions variable holds one predicted class for every test sample.
  • The first metric reports the share of correct predictions.
  • The matrix arranges true classes by row.
  • The matrix arranges predicted classes by column.
  • Save baseline.py.

✔️ Awesome, I've got everything!

Great. Your complete baseline script is ready for its first run.

ⓧ I'd like to double check the full code

  • Compare your complete baseline.py file with this reference:
from sklearn.datasets import make_classification
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score, confusion_matrix
from sklearn.model_selection import train_test_split


def build_dataset():
    X, y = make_classification(
        n_samples=1000,
        n_features=6,
        n_informative=4,
        n_redundant=0,
        n_clusters_per_class=1,
        weights=[0.9, 0.1],
        flip_y=0.0,
        class_sep=2.0,
        random_state=42,
    )
    return train_test_split(
        X,
        y,
        test_size=0.3,
        random_state=42,
        stratify=y,
    )


def main():
    X_train, X_test, y_train, y_test = build_dataset()
    baseline = DummyClassifier(strategy="most_frequent")
    baseline.fit(X_train, y_train)
    predictions = baseline.predict(X_test)

    print("Most-frequent baseline")
    print(f"Accuracy: {accuracy_score(y_test, predictions):.3f}")
    print("Confusion matrix: true classes are rows")
    print(confusion_matrix(y_test, predictions))


if __name__ == "__main__":
    main()
  • Before running the script, predict whether its high accuracy and prediction pattern will tell the same story.
  • Run the baseline from the activated VS Code terminal by using this command:
python baseline.py

What does the result reveal?

You should see Accuracy: 0.900 above the confusion matrix.

  • The first row shows 270 true class 0 samples predicted as class 0.
  • The second row shows 30 true class 1 samples predicted as class 0.
  • The minority-class prediction column contains only zeros.

You found the trap: the baseline earns 90 percent accuracy while detecting no minority cases.

Baseline script not running?

If Python reports a missing package, confirm the terminal prompt still shows (sklearn-env).

If Python cannot find the script, confirm the terminal is inside misleading-accuracy-lab.

Help me debug the baseline script.

Your baseline now proves that an attractive score can hide complete minority-class failure. Next, you will build an evaluation that gives each class a voice.

Evaluate Models Honestly

You caught the baseline's trap: its attractive accuracy came with zero minority-class detections. That result gives you evidence that one headline metric can conceal harmful model behavior.

This step adds balanced accuracy to measure recall across both classes. A confusion matrix keeps each type of prediction visible.

You will compare the baseline with class-weighted logistic regression. You will preserve the evidence in JSON and Markdown files.

In this step, get ready to:
  • Compare the most-frequent baseline with class-weighted logistic regression.
  • Record accuracy, balanced accuracy, and confusion matrices.
  • Confirm that repeated experiment runs produce the same evidence.
Build the model comparison

The baseline showed the problem. Your new evaluator needs a shared route from dataset creation to model scoring so both models face the same holdout data.

  • Use the new-file control in the left sidebar of Visual Studio Code to create evaluate.py inside misleading-accuracy-lab.
  • Add the experiment imports by pasting this code into evaluate.py:
import json
from pathlib import Path

from sklearn.datasets import make_classification
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    balanced_accuracy_score,
    confusion_matrix,
)
from sklearn.model_selection import train_test_split

What do these imports provide?

  • The standard library imports write machine-readable results to disk.
  • The dataset imports recreate the experiment from your baseline.
  • The model imports provide the dummy baseline and its class-weighted comparison.
  • The metric imports calculate three views of model behavior.
  • Save evaluate.py.
  • Confirm that evaluate.py appears beside baseline.py in the left sidebar.

Is the new file missing?

Check that you created evaluate.py inside the open misleading-accuracy-lab folder. A file created outside that folder will not share the project environment.

help me locate evaluate.py in my misleading-accuracy-lab folder.

  • Recreate the baseline's dataset and split by pasting this code below the imports:
def build_dataset():
    X, y = make_classification(
        n_samples=1000,
        n_features=6,
        n_informative=4,
        n_redundant=0,
        n_clusters_per_class=1,
        weights=[0.9, 0.1],
        flip_y=0.0,
        class_sep=2.0,
        random_state=42,
    )
    return train_test_split(
        X,
        y,
        test_size=0.3,
        random_state=42,
        stratify=y,
    )

Why reuse the same dataset?

  • The 90:10 class distribution preserves the imbalance that exposed the baseline failure.
  • The fixed random states reproduce the generated samples and holdout split.
  • The stratified split preserves the class proportions in the training and test data.
  • Save evaluate.py.
  • Compare build_dataset() with the same function in baseline.py to confirm every value matches.

Do the dataset functions differ?

Check the class weights and both random states first. A changed value would make the model comparison use different evidence from the original baseline.

help me compare the build_dataset functions.

  • Add one reusable scoring function by pasting this code below build_dataset():
def evaluate_model(model, X_train, X_test, y_train, y_test):
    model.fit(X_train, y_train)
    predictions = model.predict(X_test)
    return {
        "accuracy": float(accuracy_score(y_test, predictions)),
        "balanced_accuracy": float(
            balanced_accuracy_score(y_test, predictions)
        ),
        "confusion_matrix": confusion_matrix(
            y_test, predictions
        ).tolist(),
    }

What does the evaluator measure?

  • The function fits any supplied model against the shared training data.
  • Its predictions are evaluated against the untouched test labels.
  • The returned dictionary keeps accuracy, balanced accuracy, and the confusion matrix together.
  • Converting the matrix to a list makes it suitable for JSON output.
  • Save evaluate.py.
  • Confirm that evaluate_model() appears directly below build_dataset().

Seeing uneven indentation?

Check that every statement inside evaluate_model() uses one consistent indentation level. Keep the wrapped metric calls aligned as shown.

help me correct the indentation in evaluate_model().

  • Start the experiment entry point by pasting this code below evaluate_model():
def main():
    X_train, X_test, y_train, y_test = build_dataset()
    models = {
        "Most-frequent baseline": DummyClassifier(
            strategy="most_frequent"
        ),
        "Balanced logistic regression": LogisticRegression(
            class_weight="balanced",
            max_iter=1000,
        ),
    }

    results = {
        name: evaluate_model(
            model,
            X_train,
            X_test,
            y_train,
            y_test,
        )
        for name, model in models.items()
    }

How is the comparison kept fair?

  • Both models receive the same training and test arrays.
  • The dummy model preserves the most-frequent baseline strategy.
  • Class weighting gives the logistic model more incentive to learn the minority class.
  • The results comprehension applies the same evaluator to every model.
  • Save evaluate.py.
  • Confirm that the models dictionary contains exactly two named entries.

Is a model name missing?

Check the braces around the models dictionary. Each model name must map to its classifier instance.

help me repair the models dictionary.

  • Finish the initial terminal comparison by pasting this code directly below the results dictionary:
    for name, scores in results.items():
        print(name)
        print(f"  Accuracy: {scores['accuracy']:.3f}")
        print(
            "  Balanced accuracy: "
            f"{scores['balanced_accuracy']:.3f}"
        )
        print(f"  Confusion matrix: {scores['confusion_matrix']}")


if __name__ == "__main__":
    main()

What does the terminal output show?

  • Each model receives its own labeled result section.
  • Three decimal places make the two scalar metrics easy to compare.
  • The printed matrix keeps true-class rows and predicted-class columns available for inspection.
  • The final guard runs main() when you execute the file.
  • Save evaluate.py.
  • Before you run the comparison, predict which model will have the stronger balanced accuracy.
  • Run the initial comparison in the activated terminal:
python evaluate.py

What should you see?

You should see sections for Most-frequent baseline and Balanced logistic regression. Each section shows accuracy, balanced accuracy, and a confusion matrix.

The baseline still has attractive ordinary accuracy. Its balanced accuracy and matrix expose the missing minority predictions.

Does the comparison fail to run?

Check that the terminal still shows the activated sklearn-env environment. Confirm that every imported name matches the spelling used later in the file.

help me debug evaluate.py before report generation.

Generate reusable evidence

Terminal output proves the comparison works during one session. Saved reports make the evidence reviewable after the terminal closes.

  • Insert the start of write_markdown_report() above main() by pasting this code:
def write_markdown_report(results):
    lines = [
        "# Experiment Results",
        "",
        "| Model | Accuracy | Balanced accuracy |",
        "| --- | ---: | ---: |",
    ]

    for name, scores in results.items():
        lines.append(
            f"| {name} | {scores['accuracy']:.3f} | "
            f"{scores['balanced_accuracy']:.3f} |"
        )

How does the report begin?

  • The lines list stores the report one Markdown line at a time.
  • The opening rows define a table for the two scalar metrics.
  • The loop adds one formatted table row for each model.
  • Save evaluate.py.
  • Confirm that write_markdown_report() sits between evaluate_model() and main().

Is the report function nested?

The def write_markdown_report(results): line must start at the far left. An indented definition could place it inside evaluate_model().

help me place write_markdown_report correctly.

  • Add the confusion-matrix sections by pasting this code directly below the table loop:
    lines.extend(["", "## Confusion Matrices", ""])
    for name, scores in results.items():
        lines.extend(
            [
                f"### {name}",
                "",
                "Rows are true classes. Columns are predicted classes.",
                "",
                "```text",
                str(scores["confusion_matrix"]),
                "```",
                "",
            ]
        )

Why include the matrices?

  • Each model receives a separate confusion-matrix section.
  • The orientation sentence prevents readers from reversing true and predicted classes.
  • A fenced text block preserves the matrix layout in Markdown.
  • Save evaluate.py.
  • Run the comparison to confirm the new report-building code has valid syntax:
python evaluate.py

What does this check prove?

You should see the same two model comparisons as before. Reaching both result sections confirms that the newly inserted function does not prevent the script from loading.

Seeing a syntax error now?

Check the brackets around lines.extend(). Confirm that the code remains indented inside write_markdown_report().

help me fix the confusion-matrix report section.

  • Finish the decision section and file write by pasting this code below the confusion-matrix loop:
    lines.extend(
        [
            "## Decision",
            "",
            "Use balanced accuracy and the confusion matrix for the "
            "approval gate. Keep ordinary accuracy as context, not as "
            "the deciding metric.",
            "",
        ]
    )
    Path("RESULTS.md").write_text("\n".join(lines), encoding="utf-8")

What decision does the report preserve?

The report names balanced accuracy and the confusion matrix as approval evidence. Ordinary accuracy remains supporting context.

The final line joins the report sections and writes them to RESULTS.md with UTF-8 encoding.

  • Save evaluate.py.
  • Confirm that the report function ends immediately before main().

Is the decision text outside the function?

Keep the final lines.extend() call and the Path write indented inside write_markdown_report().

help me finish write_markdown_report().

  • In main(), locate the closing brace of the results dictionary.
  • Write both evidence files by inserting this code below that closing brace:
    Path("results.json").write_text(
        json.dumps(results, indent=2),
        encoding="utf-8",
    )
    write_markdown_report(results)

How are the two outputs different?

  • The JSON file preserves structured values for later automation.
  • The Markdown function produces a report for human review.
  • Both files come from the same in-memory results dictionary.
  • Save evaluate.py.
  • Confirm that the two write operations appear before the terminal-output loop.

Are the files written too early?

Place the write operations after the complete results dictionary. Code inserted inside the comprehension will break its structure.

help me place the evidence writes inside main().

  • Add the completion message below the final confusion-matrix print inside main() by pasting this line:
    print("Wrote results.json and RESULTS.md")

Why add a completion message?

This message confirms that the terminal comparison reached the end of the experiment. It also names the two files you should inspect.

  • Save evaluate.py.
  • Before you run the completed evaluator, predict which new files should appear in the left sidebar.
  • Generate the final evidence by running:
python evaluate.py

What should the completed run produce?

You should see both model comparisons followed by Wrote results.json and RESULTS.md. You should also see results.json and RESULTS.md in the left sidebar.

Open either file to confirm it contains accuracy, balanced accuracy, and confusion-matrix evidence for both models.

Are the evidence files missing?

Check that both write operations remain inside main(). Confirm that the terminal is running from the open misleading-accuracy-lab folder.

help me find why results.json or RESULTS.md was not created.

✔️ Awesome, I've got everything!

Great. Your evaluator now compares both models and saves the evidence in two reviewable formats.

ⓧ I'd like to double check the full code

Compare your saved evaluate.py with the complete file below:

import json
from pathlib import Path

from sklearn.datasets import make_classification
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    balanced_accuracy_score,
    confusion_matrix,
)
from sklearn.model_selection import train_test_split


def build_dataset():
    X, y = make_classification(
        n_samples=1000,
        n_features=6,
        n_informative=4,
        n_redundant=0,
        n_clusters_per_class=1,
        weights=[0.9, 0.1],
        flip_y=0.0,
        class_sep=2.0,
        random_state=42,
    )
    return train_test_split(
        X,
        y,
        test_size=0.3,
        random_state=42,
        stratify=y,
    )


def evaluate_model(model, X_train, X_test, y_train, y_test):
    model.fit(X_train, y_train)
    predictions = model.predict(X_test)
    return {
        "accuracy": float(accuracy_score(y_test, predictions)),
        "balanced_accuracy": float(
            balanced_accuracy_score(y_test, predictions)
        ),
        "confusion_matrix": confusion_matrix(
            y_test, predictions
        ).tolist(),
    }


def write_markdown_report(results):
    lines = [
        "# Experiment Results",
        "",
        "| Model | Accuracy | Balanced accuracy |",
        "| --- | ---: | ---: |",
    ]

    for name, scores in results.items():
        lines.append(
            f"| {name} | {scores['accuracy']:.3f} | "
            f"{scores['balanced_accuracy']:.3f} |"
        )

    lines.extend(["", "## Confusion Matrices", ""])
    for name, scores in results.items():
        lines.extend(
            [
                f"### {name}",
                "",
                "Rows are true classes. Columns are predicted classes.",
                "",
                "```text",
                str(scores["confusion_matrix"]),
                "```",
                "",
            ]
        )

    lines.extend(
        [
            "## Decision",
            "",
            "Use balanced accuracy and the confusion matrix for the "
            "approval gate. Keep ordinary accuracy as context, not as "
            "the deciding metric.",
            "",
        ]
    )
    Path("RESULTS.md").write_text("\n".join(lines), encoding="utf-8")


def main():
    X_train, X_test, y_train, y_test = build_dataset()
    models = {
        "Most-frequent baseline": DummyClassifier(
            strategy="most_frequent"
        ),
        "Balanced logistic regression": LogisticRegression(
            class_weight="balanced",
            max_iter=1000,
        ),
    }

    results = {
        name: evaluate_model(
            model,
            X_train,
            X_test,
            y_train,
            y_test,
        )
        for name, model in models.items()
    }

    Path("results.json").write_text(
        json.dumps(results, indent=2),
        encoding="utf-8",
    )
    write_markdown_report(results)

    for name, scores in results.items():
        print(name)
        print(f"  Accuracy: {scores['accuracy']:.3f}")
        print(
            "  Balanced accuracy: "
            f"{scores['balanced_accuracy']:.3f}"
        )
        print(f"  Confusion matrix: {scores['confusion_matrix']}")

    print("Wrote results.json and RESULTS.md")


if __name__ == "__main__":
    main()

How should you use this reference?

Check the order of the four functions first. Compare indentation and punctuation around any section that differs from your saved file.

Confirm reproducible results

A repeatable experiment should give the same evidence when its inputs and software stay fixed. Your dataset generation and train/test split both use fixed random states.

  • Before you repeat the experiment, predict whether either run will report different values.
  • Run the completed experiment twice in the activated terminal:
python evaluate.py
python evaluate.py

What should the repeated runs prove?

You should see the same reported values in both runs. The two evidence files are regenerated with those same values.

Minor floating-point differences can occur across different machines. Repeated runs in your unchanged local environment should remain stable.

  • Open results.json from the left sidebar.
  • Confirm that both model entries contain accuracy, balanced_accuracy, and confusion_matrix.
  • Open RESULTS.md from the left sidebar.
  • Confirm that its table and matrix sections support the decision to use imbalance-aware evidence.

Do the repeated values change?

Check that both random_state values remain 42. Confirm that the class weights and split settings match the completed file.

help me diagnose changing experiment results.

Your experiment now replaces a misleading headline score with evidence that shows per-class behavior. Next, you will turn these results into a concise research brief.

Write the Research Brief

Your evaluation now compares a dummy baseline with class-weighted logistic regression. Those outputs prove what happened in one run.

Raw metrics alone do not demonstrate research judgment. In this step, you will turn the experiment into a concise argument in README.md.

In this step, get ready to:
  • Document the research question plus the experiment method.
  • Connect the generated evidence to an evaluation decision.
  • Record the reproducibility choices plus the limitations.
Frame the research question and method

A research brief starts with a question that the evidence can answer. Its Markdown structure makes that reasoning easy to scan.

  • Create README.md inside the misleading-accuracy-lab folder with the new-file control in the VS Code file sidebar.
  • Add the research question plus the experiment method by pasting this content:
# Misleading Accuracy Lab

A reproducible experiment showing why ordinary accuracy can hide total
failure on a minority class, followed by a cloud-ready evaluation design.

## Research Question

Can a classifier appear successful under ordinary accuracy while failing
to detect every minority-class example?

## Method

- Generate a synthetic binary dataset with a 90:10 class distribution.
- Hold out 30 percent of the data with a reproducible stratified split.
- Establish a most-frequent dummy baseline.
- Compare it with class-weighted logistic regression.
- Evaluate both models with accuracy, balanced accuracy, and confusion
  matrices.

What Does This Section Establish?

  • The research question defines the failure that the experiment investigates.
  • The method records the class distribution plus the reproducible holdout.
  • The model comparison makes the baseline an explicit part of the evidence.
  • Save README.md.
  • Open the VS Code Markdown preview from the editor toolbar.

You should see the project title followed by the research question. The method should appear as a five-item list.

README Formatting Look Wrong?

Check that each heading starts with the same number of hash characters shown above. Confirm that every method item starts on its own line.

Ask for help if the preview still shows plain Markdown symbols: help me fix the Markdown formatting in my README preview

Connect the evidence to a decision

A useful brief explains how the result changes a decision. The next sections connect the generated metrics to the approval rule.

  • Move to the end of README.md.
  • Add the run instructions plus the interpretation by pasting this content:
## Run the Experiment

Activate the project environment, then run:

```bash
python baseline.py
python evaluate.py
```

The second command creates [RESULTS.md](RESULTS.md) and `results.json`.

## Interpretation

The baseline looks attractive under ordinary accuracy because most samples
belong to one class. Its confusion matrix exposes that it never detects the
minority class. Balanced accuracy makes that failure visible by giving each
class equal importance.

## Decision Rule

Use balanced accuracy and the confusion matrix as the cloud approval gate.
Keep ordinary accuracy as supporting context rather than the deciding metric.

How Does the Evidence Support the Decision?

  • Ordinary accuracy measures the overall proportion of correct predictions.
  • Balanced accuracy gives each class equal importance.
  • The confusion matrix exposes the baseline's missing minority-class predictions.
  • Save README.md.

Before you rerun the evaluation, do you expect the generated evidence to support the same decision?

  • Refresh the generated evidence by running this command in the activated terminal:
python evaluate.py

What Does This Command Confirm?

The command runs both model evaluations from the fixed dataset plus the fixed split. It rewrites results.json and RESULTS.md from the latest run.

You should see both model comparisons in the terminal. The final line confirms that results.json plus RESULTS.md were written.

  • Select RESULTS.md in the VS Code file sidebar.
  • Open its Markdown preview from the editor toolbar.
  • Compare the baseline evidence with the decision in README.md.

The baseline keeps attractive ordinary accuracy while its balanced accuracy plus confusion matrix reveal the minority-class failure. That evidence supports the decision rule in the brief.

Generated Evidence Missing?

Confirm that the terminal still shows the activated sklearn-env environment. Run the command from inside the misleading-accuracy-lab folder.

Ask for help if the script does not recreate both files: help me debug why evaluate.py is not generating its evidence files

Document reproducibility and limitations

A defensible conclusion records what makes the experiment repeatable. It also states where the evidence stops.

  • Return to README.md in the VS Code file sidebar.
  • Add the reproducibility details plus the limitations by pasting this content at the end:
## Reproducibility

Dataset generation and the train/test split use fixed random states. The
project pins scikit-learn in `requirements.txt`, and `evaluate.py` generates
both human-readable and machine-readable evidence.

## Cloud Architecture

See [architecture.md](architecture.md) for the evaluation pipeline and
architecture decision record.

## Limitations

- The dataset is synthetic rather than domain-specific.
- The experiment uses one holdout split rather than cross-validation.
- The architecture is provider-neutral and has not yet been deployed.
- A production decision would also require domain costs, calibration,
  fairness analysis, and drift thresholds.

Why Record These Boundaries?

Fixed random states plus the pinned scikit-learn dependency make the experiment repeatable. The limitations prevent the synthetic result from being presented as production proof.

The cloud architecture link creates the handoff to the provider-neutral design you will document next.

  • Save README.md.
  • Return to its Markdown preview from the editor toolbar.

You should see separate sections for reproducibility, cloud architecture, plus limitations. The final section should render as a four-item list.

Final Sections Not Rendering?

Confirm that each section has a blank line above its heading. Check that each limitation starts with a hyphen plus a space.

Ask for help if the preview still merges the sections: help me repair the final README sections

✔️ Awesome, I've got everything!

Your research brief is assembled. Save README.md before completing the final evidence check.

ⓧ I'd like to double check the full code

Compare your complete README.md with this reference file.

# Misleading Accuracy Lab

A reproducible experiment showing why ordinary accuracy can hide total
failure on a minority class, followed by a cloud-ready evaluation design.

## Research Question

Can a classifier appear successful under ordinary accuracy while failing
to detect every minority-class example?

## Method

- Generate a synthetic binary dataset with a 90:10 class distribution.
- Hold out 30 percent of the data with a reproducible stratified split.
- Establish a most-frequent dummy baseline.
- Compare it with class-weighted logistic regression.
- Evaluate both models with accuracy, balanced accuracy, and confusion
  matrices.

## Run the Experiment

Activate the project environment, then run:

```bash
python baseline.py
python evaluate.py
```

The second command creates [RESULTS.md](RESULTS.md) and `results.json`.

## Interpretation

The baseline looks attractive under ordinary accuracy because most samples
belong to one class. Its confusion matrix exposes that it never detects the
minority class. Balanced accuracy makes that failure visible by giving each
class equal importance.

## Decision Rule

Use balanced accuracy and the confusion matrix as the cloud approval gate.
Keep ordinary accuracy as supporting context rather than the deciding metric.

## Reproducibility

Dataset generation and the train/test split use fixed random states. The
project pins scikit-learn in `requirements.txt`, and `evaluate.py` generates
both human-readable and machine-readable evidence.

## Cloud Architecture

See [architecture.md](architecture.md) for the evaluation pipeline and
architecture decision record.

## Limitations

- The dataset is synthetic rather than domain-specific.
- The experiment uses one holdout split rather than cross-validation.
- The architecture is provider-neutral and has not yet been deployed.
- A production decision would also require domain costs, calibration,
  fairness analysis, and drift thresholds.

Before you test the evidence link, where do you expect it to open inside VS Code?

  • Scan the README.md preview for the Research Question, Method, Decision Rule, plus Limitations sections.
  • Select the RESULTS.md link in the Run the Experiment section.

VS Code should open the generated RESULTS.md file. That closes the evidence loop because your argument points directly to the generated proof.

Your research argument is ready. Next, you will build the provider-neutral architecture that turns its evaluation rule into a reviewable cloud workflow.

Publish the Evaluation Architecture

Your research brief already turns raw metrics into a defensible evaluation decision. A local experiment still lacks an operational path into model approval.

In this step, you will map that decision into a provider-neutral cloud architecture. You will publish the complete case study on GitHub so reviewers can inspect how the approval workflow operates.

In this step, get ready to:
  • Create a Mermaid flowchart with an architecture decision record.
  • Review each project file for credentials or private data.
  • Publish the complete project in a public GitHub repository.
Build the cloud evaluation design

The architecture extends your local decision rule into an automated workflow. It keeps evidence generation separate from model approval.

  • In the VS Code Explorer sidebar, create architecture.md inside misleading-accuracy-lab.
  • Add the evaluation pipeline by pasting this block into architecture.md:
# Cloud Evaluation Pipeline

```mermaid
flowchart LR
    A[Versioned experiment code] --> C[Scheduled evaluation job]
    B[Versioned evaluation data] --> C
    C --> D[Metrics and report store]
    C --> E[Candidate model artifact]
    D --> F{Balanced accuracy and confusion matrix gate}
    F -->|Pass| G[Approved model registry]
    F -->|Fail| H[Alert and investigation]
    D --> I[Audit dashboard]
    E --> G
```

What does this flow show?

This flow separates evidence generation from model approval. The gate reads stored metrics before a candidate reaches the registry.

Failed candidates go to investigation. The audit dashboard preserves a reviewable record.

  • Save architecture.md.
  • Open architecture.md in the VS Code Markdown preview.

You should see the Cloud Evaluation Pipeline heading at the top of the preview.

Diagram block missing from the preview?

Confirm the diagram fence starts with ```mermaid. Check that the closing fence contains three backticks.

Make sure the flowchart lines remain inside those fences.

help me check why my Mermaid block is missing from the architecture.md preview.

  • In the editor, place your cursor below the closing Mermaid fence.
  • Add the decision context with its security controls by pasting:
## Architecture Decision Record

### Context

A model can achieve high ordinary accuracy while completely missing a
minority class. An automated platform must preserve evidence and prevent one
attractive metric from approving a harmful candidate.

### Decision

Run evaluation as a short-lived job against versioned code and data. Store
the generated metrics and report before a policy gate checks balanced
accuracy and the confusion matrix. Only passing candidates enter the approved
model registry.

### Security

- Give the scheduler, evaluation job, and reviewer separate identities.
- Grant each identity only the data and artifact access it needs.
- Keep credentials and private evaluation data out of the code repository.
- Encrypt evaluation data, metrics, and model artifacts in transit and at
  rest when mapped to a cloud provider.

How does the decision record help?

The Context section captures the risk exposed by your experiment. The Decision section converts your evidence into an approval policy.

Separate identities limit each component's reach. Encryption protects the evaluation assets when this design is mapped to a provider.

  • Save architecture.md.
  • Return to its Markdown preview.

You should see the Architecture Decision Record heading with Context, Decision, plus Security sections.

Decision record headings not rendering?

Check that each heading starts at the beginning of its line. Keep one blank line above each heading.

Confirm the Security controls each begin with a hyphen.

help me fix the Markdown headings in my architecture decision record.

  • In the editor, place your cursor below the final Security control.
  • Complete the decision record by pasting:
### Reliability

- Assign every evaluation a unique run identifier.
- Preserve inputs, outputs, and the decision for later audit.
- Retry transient job failures without silently approving a model.
- Send failed evaluations to investigation rather than the approved registry.

### Cost Controls

- Use short-lived compute instead of an always-on evaluation server.
- Trigger work only for a schedule or a new candidate model.
- Apply retention limits to old model artifacts and detailed logs.

### Consequences

The design is portable and inexpensive to prototype, but provider-specific
identity, storage, scheduling, alerting, and registry services still need to
be selected before deployment.

What do these controls protect?

Run identifiers make each evaluation traceable. Retries prevent transient failures from becoming silent approvals.

Short-lived compute limits idle cost. Retention limits stop old artifacts from growing forever.

The Consequences section records the provider-specific work that remains before deployment.

  • Save architecture.md.
  • Return to its Markdown preview.

You should see Reliability, Cost Controls, plus Consequences below the Security section.

Final sections missing?

Confirm the Reliability heading sits below the complete Security list. Check that none of the new headings remain inside the Mermaid fence.

help me find why the final architecture decision record sections are missing.

✔️ Awesome, I've got everything!

Your cloud evaluation pipeline plus its complete architecture decision record are saved.

ⓧ I'd like to double check the full code

# Cloud Evaluation Pipeline

```mermaid
flowchart LR
    A[Versioned experiment code] --> C[Scheduled evaluation job]
    B[Versioned evaluation data] --> C
    C --> D[Metrics and report store]
    C --> E[Candidate model artifact]
    D --> F{Balanced accuracy and confusion matrix gate}
    F -->|Pass| G[Approved model registry]
    F -->|Fail| H[Alert and investigation]
    D --> I[Audit dashboard]
    E --> G
```

## Architecture Decision Record

### Context

A model can achieve high ordinary accuracy while completely missing a
minority class. An automated platform must preserve evidence and prevent one
attractive metric from approving a harmful candidate.

### Decision

Run evaluation as a short-lived job against versioned code and data. Store
the generated metrics and report before a policy gate checks balanced
accuracy and the confusion matrix. Only passing candidates enter the approved
model registry.

### Security

- Give the scheduler, evaluation job, and reviewer separate identities.
- Grant each identity only the data and artifact access it needs.
- Keep credentials and private evaluation data out of the code repository.
- Encrypt evaluation data, metrics, and model artifacts in transit and at
  rest when mapped to a cloud provider.

### Reliability

- Assign every evaluation a unique run identifier.
- Preserve inputs, outputs, and the decision for later audit.
- Retry transient job failures without silently approving a model.
- Send failed evaluations to investigation rather than the approved registry.

### Cost Controls

- Use short-lived compute instead of an always-on evaluation server.
- Trigger work only for a schedule or a new candidate model.
- Apply retention limits to old model artifacts and detailed logs.

### Consequences

The design is portable and inexpensive to prototype, but provider-specific
identity, storage, scheduling, alerting, and registry services still need to
be selected before deployment.
Check the files for safe publishing

A public repository exposes every uploaded file. A deliberate review keeps credentials plus private data out of your portfolio.

  • Review requirements.txt for passwords, tokens, API keys, or private data.
  • Review baseline.py for passwords, tokens, API keys, or private data.
  • Review evaluate.py for passwords, tokens, API keys, or private data.
  • Review results.json for passwords, tokens, API keys, or private data.
  • Review RESULTS.md for passwords, tokens, API keys, or private data.
  • Review README.md for passwords, tokens, API keys, or private data.
  • Review architecture.md for passwords, tokens, API keys, or private data.
  • Save all reviewed files.

That is the publishing risk handled. Your upload set now contains only experiment code, generated evidence, plus documentation.

Publish the project on GitHub

The public repository becomes the reviewable home for your experiment. Keeping it empty at creation avoids conflicts with the files already in misleading-accuracy-lab.

  • In your browser, sign in to GitHub.
  • On GitHub, select New repository.
  • Select your account under the owner control.
  • Enter misleading-accuracy-lab in the Repository name field.
  • Select Public as the repository visibility.
  • Leave the repository initialization choices unselected.
  • Click Create repository.

Good progress. Your empty public repository now has a clean destination for the reviewed files.

Why keep the repository empty?

Pre-populated files could duplicate the local documentation. An empty repository lets your reviewed project become the initial source.

Browser upload avoids an unverified local Git dependency. It keeps this step focused on presenting the architecture.

Why browser upload?

A local Git workflow would require additional setup. Browser upload uses the GitHub account you already have.

The upload control opens your browser's file picker. It accepts the local project files without a terminal workflow.

  • Select Add file on the empty repository page.
  • Click Upload files.
  • Select requirements.txt, baseline.py, evaluate.py, results.json, RESULTS.md, README.md, plus architecture.md in the file picker.
  • Enter Publish evaluation lab as the commit message.
  • Choose the branch flow that adds the files to the default branch.
  • Click Propose changes to complete the upload flow.

You should see the seven uploaded files in the repository.

Missing an uploaded file?

  • Return to Add file.
  • Select Upload files.
  • Check that the file picker includes the missing file from misleading-accuracy-lab.

help me identify which evaluation lab files are missing from my GitHub repository.

  • Before you inspect architecture.md, predict whether GitHub will display diagram source or a rendered system.
  • Open README.md from the repository file list.
  • Select the architecture.md link under Cloud Architecture.

GitHub renders the Mermaid flowchart as a connected system diagram. The architecture decision record appears directly below it.

Seeing diagram source instead?

Compare the opening Mermaid fence with the full architecture.md reference above. Confirm the flow starts with flowchart LR inside the fence.

Check that architecture.md was included in the completed upload flow.

help me debug why GitHub shows my Mermaid source instead of the rendered architecture diagram.

You made the evaluation decision reviewable from end to end. Your public repository now connects reproducible evidence to a cloud approval design.

Secret mission

Stress Test Distribution Shift

Your current evidence uses a 90:10 class distribution. This challenge changes the minority share to 3 percent so you can compare accuracy with balanced accuracy under a harder distribution. You will publish the result as evidence for your cloud approval gate.

Clean Up Your Resources

Clean Up Your Resources

Your experiment uses local compute. GitHub Free costs $ 0 USD per month.

Choose whether to keep your resources available, pause your work, or delete everything.

Resources you used:

  • Your local misleading-accuracy-lab folder stores the experiment plus its sklearn-env virtual environment.
  • Your public GitHub repository named misleading-accuracy-lab stores the committed project evidence.

Keep everything running

No action is needed. Choose this option if you want to run more experiments or keep sharing the evidence.

  • Your local misleading-accuracy-lab folder preserves the runnable experiment.
  • The sklearn-env environment keeps scikit-learn version 1.9.1 ready for another run.
  • The public GitHub repository keeps the generated evidence accessible from its existing URL.

Pause - I'll come back to this later

This option frees the disk space used by sklearn-env. Your committed evidence stays available in a private repository.

The requirements.txt pin makes the environment deletion reversible.

  • Locate sklearn-env inside misleading-accuracy-lab in Finder.
  • Delete sklearn-env in Finder.
  • Return to the misleading-accuracy-lab repository on GitHub.
  • Select Settings.
  • Scroll to Danger Zone.
  • Select Change repository visibility.
  • Acknowledge the visibility effects.
  • Select Make this repository private.

You should see that the repository is private. The local scripts remain inside misleading-accuracy-lab.

Delete - I don't want to use this again

Deleting both copies is permanent.

The local experiment disappears. The hosted evidence also disappears.

  • Return to the misleading-accuracy-lab repository on GitHub.
  • Select Settings.
  • Select General.
  • Scroll to Danger Zone.
  • Select Delete this repository.
  • Acknowledge the deletion warnings.
  • Type misleading-accuracy-lab as the repository name.
  • Select Delete this repository.

That is the hosted copy cleared. Your local folder is the last remaining copy.

  • Close VS Code.
  • Locate the misleading-accuracy-lab folder in Finder.
  • Delete the misleading-accuracy-lab folder in Finder.
  • Search for misleading-accuracy-lab in Finder to verify the deletion.

Cleanup complete. You should no longer see the local folder or the GitHub repository.

Nice Work!

Nice Work!

You did it! You built a reproducible AI evaluation case study that exposes misleading accuracy. Your published GitHub project connects experimental evidence to a reviewable cloud approval architecture.

You've learned how to:

  • Built a reproducible dummy baseline with scikit-learn. Exposed zero minority-class detections through a confusion matrix.
  • Compared the baseline with class-weighted logistic regression using balanced accuracy. Generated a machine-readable JSON file automatically. Produced a readable Markdown report from the same run.
  • Published a research brief in GitHub. Rendered a Mermaid cloud evaluation pipeline beside it. Captured its security controls in an architecture decision record. Documented its reliability safeguards there. Defined its cost controls there.
  • Completed the optional Secret Mission by testing the 90:10 class distribution against a 97:3 class distribution. Confirmed that balanced accuracy must accompany ordinary accuracy when class prevalence changes.

Ready to quiz yourself?