Trustworthy Experiment Decision Lab

Analyze a simulated A/B test and publish a guardrail-based launch decision.

Introduction

30 Second Summary

A promising product result can quickly become a launch recommendation. Hidden gaps in who was counted can turn that confidence into the wrong decision.

In this project, you will build a reproducible experiment decision lab in Google Colab for a simulated onboarding test. You will expose a false revenue win before producing a defensible launch recommendation for a public GitHub portfolio.

What You'll Build

Picture rerunning your public notebook and watching an attractive revenue result fail its validity check before the corrected evidence lands on an ITERATE recommendation.

By the end of this project, you'll have:

  • A reproducible notebook that generates the same source-like tables for every reviewer. Its sample ratio mismatch check catches a telemetry-only false win.
  • A corrected intention-to-treat analysis that restores all 20,000 randomized users. Its seven-day revenue estimate includes a 95% confidence interval.
  • A public GitHub portfolio where reviewers can inspect your decision graphic and executive README. They can compare raw estimates with CUPED-adjusted estimates behind your ITERATE recommendation.
  • Secret Mission: Test whether low-value and high-value users respond differently enough to justify a targeted rollout.

Are there any prerequisites?

You should have professional data-analysis experience with Python data manipulation and basic statistical inference. You also need a Mac, a Google account, and a GitHub account.

Before We Start

Before any analysis begins, lock in the launch decision this experiment readout must support. A sample ratio mismatch check must come before effect interpretation because an imbalanced analyzed population can create a false causal result.

Prepare a Reproducible Colab Runtime

Your launch question is locked in. A reviewer still needs to recreate the exact analysis environment before trusting any result.

A Google Colab notebook gives reviewers a hosted runtime. A pinned installation cell rebuilds that runtime from the notebook saved in Google Drive.

In this step, get ready to:
  • Create a Google Drive-backed notebook named trustworthy_experiment_lab.ipynb.
  • Install the pinned analytics packages before any imports.
  • Verify Python 3.12 or newer with NumPy 2.5.2, pandas 3.0.5, SciPy 1.18.1, plus Matplotlib 3.11.1.
Create the Colab notebook

The notebook holds both the analysis code and the instructions for recreating its environment. Saving it in Google Drive keeps that reproducible artifact tied to your account.

  • Open Google Colab in your browser.
  • Sign in with the Google account you confirmed earlier if Colab asks.
  • Click File in the top menu.
  • Select New notebook.
  • Click the notebook title at the top of the page.
  • Replace the existing title with trustworthy_experiment_lab.ipynb.
  • Press Enter to apply the new name.
  • Wait until Colab confirms the notebook is saved in Google Drive.

Your analysis now has a Drive-backed home that reviewers can reopen as a single notebook.

Pin the analytics packages

Version pins lock every reviewer to the same numerical stack. The installation cell belongs above every import cell.

  • Paste the pinned installer into the first code cell by copying the code below:
# Cell 1: Pin the reproducible environment.
%pip install -q "numpy==2.5.2" "pandas==3.0.5" "scipy==1.18.1" "matplotlib==3.11.1"

What does this cell do?

  • The %pip command installs the four packages inside the active notebook runtime.
  • Each equality pin selects one exact package version.
  • Running this cell first ensures that later imports use the versions checked by the notebook.
  • Run the first cell by selecting its run control.
  • Wait until the cell stops running.
  • Confirm the cell finishes without a package-installation error.
  • Wait until Colab confirms the latest notebook changes are saved in Google Drive.

That pins the exact analytics stack inside the notebook, ready for every fresh runtime.

Did the installation fail?

  • Compare the installation cell with the code above to catch a missing quote or version number.
  • Rerun the cell once after a temporary network interruption.

Ask for help diagnosing the Colab package installation output. You can also share the non-sensitive output in the NextWork community.

Verify the runtime

The verification cell converts environment assumptions into executable checks. A failed assertion stops the notebook before an incompatible runtime can affect the experiment analysis.

  • Add a second code cell directly below the installation cell.
  • Paste the environment verification code into that cell by copying the code below:
# Cell 2: Import and verify the environment.
import sys

import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import scipy
from scipy.stats import chisquare, norm

assert sys.version_info >= (3, 12), sys.version
assert np.__version__ == "2.5.2", np.__version__
assert pd.__version__ == "3.0.5", pd.__version__
assert scipy.__version__ == "1.18.1", scipy.__version__
assert matplotlib.__version__ == "3.11.1", matplotlib.__version__

print("Python:", sys.version.split()[0])
print("NumPy:", np.__version__)
print("pandas:", pd.__version__)
print("SciPy:", scipy.__version__)
print("Matplotlib:", matplotlib.__version__)

What does this code verify?

  • The imports make sys, matplotlib, plt, np, pd, scipy, chisquare, plus norm available to later cells.
  • The assertions stop execution when Python or a package does not match the required environment.
  • The print statements expose the active versions for reviewers to inspect.

Before you run the check, do you expect the runtime to satisfy every assertion?

  • Run the second code cell by selecting its run control.

You should see five version lines. They report Python 3.12 or newer, NumPy 2.5.2, pandas 3.0.5, SciPy 1.18.1, plus Matplotlib 3.11.1.

✔️ Every check passes

Your reproducibility gate is clear. Every required version matches the notebook.

The active runtime now contains every imported identifier needed by the next analysis cells.

  • Wait until Colab confirms the latest changes are saved in Google Drive.

ⓧ The Python check fails

The first assertion indicates that the hosted runtime is below Python 3.12.

Resetting the temporary runtime leaves your Drive-backed notebook untouched.

  • Click Runtime in the top menu.
  • Select Disconnect and delete runtime.
  • Run the first installation cell.
  • Run the second verification cell.

The Python line should now show 3.12 or newer.

ⓧ A package check fails

A package assertion means that the loaded version does not match its pin.

This reset removes only the temporary runtime. Your saved notebook remains in Google Drive.

  • Click Runtime in the top menu.
  • Select Disconnect and delete runtime.
  • Run the first installation cell.
  • Run the second verification cell.

All four package lines should now match their pinned versions.

Still failing an environment check?

  • Stop here until every assertion passes.

Ask for help with the failing Colab assertion. Then share the non-sensitive assertion output in the NextWork community if you need another pair of eyes.

The reference below shows the complete notebook state for this step.

✔️ Awesome, I've got everything!

Great. Both cells are saved in the correct order.

ⓧ I'd like to double check the full code

Your notebook should contain these two cells in this order.

# Cell 1: Pin the reproducible environment.
%pip install -q "numpy==2.5.2" "pandas==3.0.5" "scipy==1.18.1" "matplotlib==3.11.1"

# Cell 2: Import and verify the environment.
import sys

import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import scipy
from scipy.stats import chisquare, norm

assert sys.version_info >= (3, 12), sys.version
assert np.__version__ == "2.5.2", np.__version__
assert pd.__version__ == "3.0.5", pd.__version__
assert scipy.__version__ == "1.18.1", scipy.__version__
assert matplotlib.__version__ == "3.11.1", matplotlib.__version__

print("Python:", sys.version.split()[0])
print("NumPy:", np.__version__)
print("pandas:", pd.__version__)
print("SciPy:", scipy.__version__)
print("Matplotlib:", matplotlib.__version__)

Your Colab notebook now carries a reproducible analytics environment. Next, you will use it to produce the experiment's attractive first result.

Expose the Invalid Telemetry Result

Your verified Google Colab runtime gives reviewers a reproducible starting point. A polished experiment readout can still be wrong when its analyzed population no longer reflects random assignment.

This step deliberately creates a tempting telemetry-only result. You will challenge it with a sample ratio mismatch check before using the treatment effect.

In this step, get ready to:
  • Define the decision thresholds and reusable experiment estimators.
  • Generate deterministic assignment, billing, support, and exposure tables.
  • Run the telemetry-only analysis and test its allocation balance.
Define the estimators and validity check

The analysis needs one function for treatment effects and another for allocation validity. Shared constants keep the simulated scenario reproducible across every notebook run.

  • Add Cell 3 beneath the environment-verification cell by pasting this code:
# Cell 3: Define the decision policy and reusable estimators.
RNG_SEED = 20260829
N_USERS = 20_000
SRM_ALPHA = 0.0005
GUARDRAIL_TOLERANCE = 0.005
CONFIDENCE_LEVEL = 0.95


def difference_in_means(frame, metric):
    treatment = frame.loc[frame["variant"] == "treatment", metric].to_numpy(dtype=float)
    control = frame.loc[frame["variant"] == "control", metric].to_numpy(dtype=float)
    effect = np.mean(treatment) - np.mean(control)
    standard_error = np.sqrt(np.var(treatment, ddof=1) / len(treatment) + np.var(control, ddof=1) / len(control))
    critical_value = norm.ppf(0.975)
    return {"effect": effect, "standard_error": standard_error, "lower": effect - critical_value * standard_error, "upper": effect + critical_value * standard_error}


def srm_check(frame):
    observed = np.array([(frame["variant"] == "control").sum(), (frame["variant"] == "treatment").sum()])
    expected = np.repeat(observed.sum() / 2, 2)
    result = chisquare(f_obs=observed, f_exp=expected)
    return {"control": int(observed[0]), "treatment": int(observed[1]), "p_value": float(result.pvalue), "passed": bool(result.pvalue >= SRM_ALPHA)}

What does this code do?

  • The constants lock the random seed, population size, SRM threshold, guardrail tolerance, and confidence level.
  • The difference_in_means function uses NumPy to calculate the treatment-control difference and its standard error.
  • The normal critical value turns that uncertainty into a 95% confidence interval.
  • The srm_check function applies a chi-square test to the observed treatment and control counts.
  • Run Cell 3 using the play button beside the cell.
  • Wait for Google Colab to save the updated trustworthy_experiment_lab.ipynb notebook to Google Drive.

You will see Cell 3 finish without an exception. The runtime now holds the policy constants and both reusable functions.

Seeing an exception in Cell 3?

  • Run Cell 2 again so np, chisquare, and norm are available.
  • Compare the indentation inside difference_in_means and srm_check with the code above.
  • Ask for help with the failing cell.
Generate the source-like tables

A fixed seed gives every reviewer the same simulated users and outcomes. The four tables resemble separate product-data sources that an experiment analyst might receive.

  • Add Cell 4 beneath Cell 3 by pasting this code:
# Cell 4: Generate source-like tables for a randomized product experiment.
rng = np.random.default_rng(RNG_SEED)
user_id = np.arange(1, N_USERS + 1)
variant = np.repeat(np.array(["control", "treatment"]), N_USERS // 2)
rng.shuffle(variant)
is_treatment = (variant == "treatment").astype(int)
latent_value = rng.normal(0.0, 1.0, N_USERS)
pre_revenue = np.maximum(0.0, 20.0 + 10.0 * latent_value + rng.normal(0.0, 6.0, N_USERS))
revenue_7d = np.maximum(0.0, 5.0 + 0.70 * pre_revenue + 2.50 * is_treatment + rng.normal(0.0, 10.0, N_USERS))
support_probability = 0.04 + 0.04 * is_treatment
support_contact = (rng.random(N_USERS) < support_probability).astype(int)
assignment = pd.DataFrame({"user_id": user_id, "variant": variant, "pre_revenue": pre_revenue})
billing = pd.DataFrame({"user_id": user_id[revenue_7d > 0], "revenue_7d": revenue_7d[revenue_7d > 0]})
support_events = pd.DataFrame({"user_id": user_id[support_contact == 1], "support_contact": 1})
high_pre_value = pre_revenue > np.median(pre_revenue)
exposure_probability = np.where(is_treatment == 1, 0.99, np.where(high_pre_value, 0.62, 0.98))
exposure_log = pd.DataFrame({"user_id": user_id[rng.random(N_USERS) < exposure_probability]})
print("Assignment rows:", len(assignment))
print("Billing rows:", len(billing))
print("Support-event rows:", len(support_events))
print("Logged-exposure rows:", len(exposure_log))
print(assignment.head())

How are the tables structured?

  • The fixed RNG_SEED makes the simulated data deterministic.
  • The assignment table stores every randomized user with a treatment variant and pre-experiment revenue.
  • The billing and support_events tables represent trusted outcome sources built with Pandas.
  • The exposure_log table records a post-assignment telemetry event for a subset of users.
  • Run Cell 4 using the play button beside the cell.
  • Wait for Google Colab to save the updated notebook.

You will see Assignment rows: 20000 followed by three more row-count lines. You will also see a five-row preview of assignment.

Missing the table preview?

  • Confirm Cell 3 completed before you run Cell 4.
  • Check that the final line is print(assignment.head()).
  • Ask for help with the data-generation cell.
Run the telemetry-only analysis

The tempting shortcut uses the exposure log to decide which randomized users enter the analysis. It then attaches billing outcomes and calculates a decision-ready-looking estimate.

  • Add Cell 5 beneath Cell 4 by pasting this code:
# Cell 5: Run the attractive but invalid telemetry-only analysis.
naive = assignment.merge(exposure_log, how="inner", on="user_id", validate="one_to_one").merge(billing, how="left", on="user_id", validate="one_to_one").fillna({"revenue_7d": 0.0})
naive_result = difference_in_means(naive, "revenue_7d")
naive_srm = srm_check(naive)
print(f"Naive revenue effect: ${naive_result['effect']:.2f} per user")
print(f"Naive 95% CI: [${naive_result['lower']:.2f}, ${naive_result['upper']:.2f}]")
print("Observed users:", naive_srm["control"], "control and", naive_srm["treatment"], "treatment")
print(f"SRM p-value: {naive_srm['p_value']:.3g}")
print("SRM status:", "PASS" if naive_srm["passed"] else "SRM DETECTED: do not interpret the treatment effect.")

What does this analysis test?

  • The inner merge keeps users who appear in exposure_log.
  • The left merge attaches each retained user's billing outcome.
  • The difference_in_means function estimates the apparent revenue effect.
  • The srm_check function tests whether the analyzed allocation still matches the intended split.

Before you run Cell 5, do you expect the attractive revenue estimate to survive the allocation check?

  • Run Cell 5 using the play button beside the cell.
  • Wait for Google Colab to save the completed analysis cell.

You will see a positive naive revenue effect. The final line reads SRM status: SRM DETECTED: do not interpret the treatment effect..

Why is the attractive result invalid?

The SRM result shows that the analyzed treatment and control counts no longer match the intended randomized allocation. This proves that the telemetry filter changed the analysis population after assignment.

The incomplete exposure log excludes users unevenly across the variants. That selection can distort the outcome distribution, so the positive estimate stays quarantined from the launch decision.

Do you see a different SRM result?

  • Confirm Cell 4 uses RNG_SEED = 20260829.
  • Check that assignment uses an inner merge with exposure_log.
  • Ask for help comparing your output with the expected SRM warning.

The checkpoint below shows the cumulative notebook state through Cell 5.

✔️ Awesome, I've got everything!

That is the trap exposed. Your notebook now produces an attractive estimate and blocks its interpretation with the SRM check.

ⓧ I'd like to double check the full code

# Cell 1: Pin the reproducible environment.
%pip install -q "numpy==2.5.2" "pandas==3.0.5" "scipy==1.18.1" "matplotlib==3.11.1"

# Cell 2: Import and verify the environment.
import sys

import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import scipy
from scipy.stats import chisquare, norm

assert sys.version_info >= (3, 12), sys.version
assert np.__version__ == "2.5.2", np.__version__
assert pd.__version__ == "3.0.5", pd.__version__
assert scipy.__version__ == "1.18.1", scipy.__version__
assert matplotlib.__version__ == "3.11.1", matplotlib.__version__

print("Python:", sys.version.split()[0])
print("NumPy:", np.__version__)
print("pandas:", pd.__version__)
print("SciPy:", scipy.__version__)
print("Matplotlib:", matplotlib.__version__)

# Cell 3: Define the decision policy and reusable estimators.
RNG_SEED = 20260829
N_USERS = 20_000
SRM_ALPHA = 0.0005
GUARDRAIL_TOLERANCE = 0.005
CONFIDENCE_LEVEL = 0.95


def difference_in_means(frame, metric):
    treatment = frame.loc[frame["variant"] == "treatment", metric].to_numpy(dtype=float)
    control = frame.loc[frame["variant"] == "control", metric].to_numpy(dtype=float)
    effect = np.mean(treatment) - np.mean(control)
    standard_error = np.sqrt(np.var(treatment, ddof=1) / len(treatment) + np.var(control, ddof=1) / len(control))
    critical_value = norm.ppf(0.975)
    return {"effect": effect, "standard_error": standard_error, "lower": effect - critical_value * standard_error, "upper": effect + critical_value * standard_error}


def srm_check(frame):
    observed = np.array([(frame["variant"] == "control").sum(), (frame["variant"] == "treatment").sum()])
    expected = np.repeat(observed.sum() / 2, 2)
    result = chisquare(f_obs=observed, f_exp=expected)
    return {"control": int(observed[0]), "treatment": int(observed[1]), "p_value": float(result.pvalue), "passed": bool(result.pvalue >= SRM_ALPHA)}

# Cell 4: Generate source-like tables for a randomized product experiment.
rng = np.random.default_rng(RNG_SEED)
user_id = np.arange(1, N_USERS + 1)
variant = np.repeat(np.array(["control", "treatment"]), N_USERS // 2)
rng.shuffle(variant)
is_treatment = (variant == "treatment").astype(int)
latent_value = rng.normal(0.0, 1.0, N_USERS)
pre_revenue = np.maximum(0.0, 20.0 + 10.0 * latent_value + rng.normal(0.0, 6.0, N_USERS))
revenue_7d = np.maximum(0.0, 5.0 + 0.70 * pre_revenue + 2.50 * is_treatment + rng.normal(0.0, 10.0, N_USERS))
support_probability = 0.04 + 0.04 * is_treatment
support_contact = (rng.random(N_USERS) < support_probability).astype(int)
assignment = pd.DataFrame({"user_id": user_id, "variant": variant, "pre_revenue": pre_revenue})
billing = pd.DataFrame({"user_id": user_id[revenue_7d > 0], "revenue_7d": revenue_7d[revenue_7d > 0]})
support_events = pd.DataFrame({"user_id": user_id[support_contact == 1], "support_contact": 1})
high_pre_value = pre_revenue > np.median(pre_revenue)
exposure_probability = np.where(is_treatment == 1, 0.99, np.where(high_pre_value, 0.62, 0.98))
exposure_log = pd.DataFrame({"user_id": user_id[rng.random(N_USERS) < exposure_probability]})
print("Assignment rows:", len(assignment))
print("Billing rows:", len(billing))
print("Support-event rows:", len(support_events))
print("Logged-exposure rows:", len(exposure_log))
print(assignment.head())

# Cell 5: Run the attractive but invalid telemetry-only analysis.
naive = assignment.merge(exposure_log, how="inner", on="user_id", validate="one_to_one").merge(billing, how="left", on="user_id", validate="one_to_one").fillna({"revenue_7d": 0.0})
naive_result = difference_in_means(naive, "revenue_7d")
naive_srm = srm_check(naive)
print(f"Naive revenue effect: ${naive_result['effect']:.2f} per user")
print(f"Naive 95% CI: [${naive_result['lower']:.2f}, ${naive_result['upper']:.2f}]")
print("Observed users:", naive_srm["control"], "control and", naive_srm["treatment"], "treatment")
print(f"SRM p-value: {naive_srm['p_value']:.3g}")
print("SRM status:", "PASS" if naive_srm["passed"] else "SRM DETECTED: do not interpret the treatment effect.")

This reference contains the exact cumulative notebook code used to generate the source tables and expose the invalid telemetry result.

You have caught a positive result that fails its validity gate. Next, you will restore the randomized analysis population and recover a trustworthy revenue estimate.

Restore the Intention-to-Treat Population

Your Google Colab notebook has exposed a sample ratio mismatch in the telemetry-filtered population. That mismatch blocks the positive result from supporting a launch decision.

Intention-to-treat analysis keeps every randomized user in their assigned variant. You will rebuild the population from assignments before joining the trusted outcomes.

In this step, get ready to:
  • Restore every randomized user from the assignment table.
  • Join the trusted billing outcomes plus support events.
  • Compare the corrected estimate with the invalid naive estimate.
Choose the randomized anchor

The assignment table preserves the original randomized groups. The incomplete exposure log contains only a filtered subset.

  • Return to Cell 4 in the Google Colab notebook from earlier.
  • Locate the line that creates assignment.
  • Read the assignment row count printed beneath Cell 4.

You should see 20,000 assignment rows. This complete population becomes the anchor for the corrected analysis.

Why Start From Assignment?

Randomization happens in assignment. Starting there keeps users whose exposure telemetry is missing.

A missing billing row represents zero seven-day revenue in this simulation. A missing support-event row represents no recorded support contact.

Run the corrected analysis

A left join keeps every assigned user while adding any matching outcome. The zero-filled outcomes preserve the full analysis population.

  • In the notebook toolbar, use the code-cell control to insert a new cell beneath Cell 5.
  • Paste the following code into the new cell:
# Cell 6: Rebuild the full intention-to-treat population.
itt = assignment.merge(billing, how="left", on="user_id", validate="one_to_one").merge(support_events, how="left", on="user_id", validate="one_to_one").fillna({"revenue_7d": 0.0, "support_contact": 0.0})
itt_srm = srm_check(itt)
raw_result = difference_in_means(itt, "revenue_7d")
print("Assigned users:", itt_srm["control"], "control and", itt_srm["treatment"], "treatment")
print(f"SRM p-value: {itt_srm['p_value']:.3g}")
print("SRM status:", "PASS" if itt_srm["passed"] else "FAIL")
print(f"Corrected revenue effect: ${raw_result['effect']:.2f} per user")
print(f"Corrected 95% CI: [${raw_result['lower']:.2f}, ${raw_result['upper']:.2f}]")
print(f"Naive estimate overstated the effect by: ${naive_result['effect'] - raw_result['effect']:.2f}")

What Does This Code Do?

  • The itt population begins with every row in assignment.
  • The left joins add matching rows from billing plus support_events.
  • The fillna call encodes absent trusted events as zero outcomes.
  • The itt_srm result checks the restored variant counts.
  • The raw_result estimate reports the corrected revenue effect with its 95% confidence interval.

Before you run the cell, do you expect the SRM warning to remain once every assigned user is restored?

  • Run Cell 6 using the play control beside the cell.
  • Save trustworthy_experiment_lab.ipynb using Colab's notebook save control.

You will see 10,000 control users plus 10,000 treatment users. The output reports SRM status: PASS before printing the corrected revenue effect with its 95% confidence interval.

That closes the validity gap. Your estimate now represents all 20,000 randomized users.

Counts or SRM Status Look Wrong?

Check that itt starts from assignment. An inner join with exposure_log recreates the biased population.

Confirm that both outcome joins use how="left". Rerun Cells 3 through 5 first if the notebook reports that an earlier variable is unavailable.

Help me diagnose my corrected intention-to-treat analysis.

✔️ Awesome, I've got everything!

Your notebook now contains the complete intention-to-treat analysis cell.

ⓧ I'd like to double check the full code

# Cell 1: Pin the reproducible environment.
%pip install -q "numpy==2.5.2" "pandas==3.0.5" "scipy==1.18.1" "matplotlib==3.11.1"

# Cell 2: Import and verify the environment.
import sys

import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import scipy
from scipy.stats import chisquare, norm

assert sys.version_info >= (3, 12), sys.version
assert np.__version__ == "2.5.2", np.__version__
assert pd.__version__ == "3.0.5", pd.__version__
assert scipy.__version__ == "1.18.1", scipy.__version__
assert matplotlib.__version__ == "3.11.1", matplotlib.__version__

print("Python:", sys.version.split()[0])
print("NumPy:", np.__version__)
print("pandas:", pd.__version__)
print("SciPy:", scipy.__version__)
print("Matplotlib:", matplotlib.__version__)

# Cell 3: Define the decision policy and reusable estimators.
RNG_SEED = 20260829
N_USERS = 20_000
SRM_ALPHA = 0.0005
GUARDRAIL_TOLERANCE = 0.005
CONFIDENCE_LEVEL = 0.95


def difference_in_means(frame, metric):
    treatment = frame.loc[frame["variant"] == "treatment", metric].to_numpy(dtype=float)
    control = frame.loc[frame["variant"] == "control", metric].to_numpy(dtype=float)
    effect = np.mean(treatment) - np.mean(control)
    standard_error = np.sqrt(np.var(treatment, ddof=1) / len(treatment) + np.var(control, ddof=1) / len(control))
    critical_value = norm.ppf(0.975)
    return {"effect": effect, "standard_error": standard_error, "lower": effect - critical_value * standard_error, "upper": effect + critical_value * standard_error}


def srm_check(frame):
    observed = np.array([(frame["variant"] == "control").sum(), (frame["variant"] == "treatment").sum()])
    expected = np.repeat(observed.sum() / 2, 2)
    result = chisquare(f_obs=observed, f_exp=expected)
    return {"control": int(observed[0]), "treatment": int(observed[1]), "p_value": float(result.pvalue), "passed": bool(result.pvalue >= SRM_ALPHA)}

# Cell 4: Generate source-like tables for a randomized product experiment.
rng = np.random.default_rng(RNG_SEED)
user_id = np.arange(1, N_USERS + 1)
variant = np.repeat(np.array(["control", "treatment"]), N_USERS // 2)
rng.shuffle(variant)
is_treatment = (variant == "treatment").astype(int)
latent_value = rng.normal(0.0, 1.0, N_USERS)
pre_revenue = np.maximum(0.0, 20.0 + 10.0 * latent_value + rng.normal(0.0, 6.0, N_USERS))
revenue_7d = np.maximum(0.0, 5.0 + 0.70 * pre_revenue + 2.50 * is_treatment + rng.normal(0.0, 10.0, N_USERS))
support_probability = 0.04 + 0.04 * is_treatment
support_contact = (rng.random(N_USERS) < support_probability).astype(int)
assignment = pd.DataFrame({"user_id": user_id, "variant": variant, "pre_revenue": pre_revenue})
billing = pd.DataFrame({"user_id": user_id[revenue_7d > 0], "revenue_7d": revenue_7d[revenue_7d > 0]})
support_events = pd.DataFrame({"user_id": user_id[support_contact == 1], "support_contact": 1})
high_pre_value = pre_revenue > np.median(pre_revenue)
exposure_probability = np.where(is_treatment == 1, 0.99, np.where(high_pre_value, 0.62, 0.98))
exposure_log = pd.DataFrame({"user_id": user_id[rng.random(N_USERS) < exposure_probability]})
print("Assignment rows:", len(assignment))
print("Billing rows:", len(billing))
print("Support-event rows:", len(support_events))
print("Logged-exposure rows:", len(exposure_log))
print(assignment.head())

# Cell 5: Run the attractive but invalid telemetry-only analysis.
naive = assignment.merge(exposure_log, how="inner", on="user_id", validate="one_to_one").merge(billing, how="left", on="user_id", validate="one_to_one").fillna({"revenue_7d": 0.0})
naive_result = difference_in_means(naive, "revenue_7d")
naive_srm = srm_check(naive)
print(f"Naive revenue effect: ${naive_result['effect']:.2f} per user")
print(f"Naive 95% CI: [${naive_result['lower']:.2f}, ${naive_result['upper']:.2f}]")
print("Observed users:", naive_srm["control"], "control and", naive_srm["treatment"], "treatment")
print(f"SRM p-value: {naive_srm['p_value']:.3g}")
print("SRM status:", "PASS" if naive_srm["passed"] else "SRM DETECTED: do not interpret the treatment effect.")

# Cell 6: Rebuild the full intention-to-treat population.
itt = assignment.merge(billing, how="left", on="user_id", validate="one_to_one").merge(support_events, how="left", on="user_id", validate="one_to_one").fillna({"revenue_7d": 0.0, "support_contact": 0.0})
itt_srm = srm_check(itt)
raw_result = difference_in_means(itt, "revenue_7d")
print("Assigned users:", itt_srm["control"], "control and", itt_srm["treatment"], "treatment")
print(f"SRM p-value: {itt_srm['p_value']:.3g}")
print("SRM status:", "PASS" if itt_srm["passed"] else "FAIL")
print(f"Corrected revenue effect: ${raw_result['effect']:.2f} per user")
print(f"Corrected 95% CI: [${raw_result['lower']:.2f}, ${raw_result['upper']:.2f}]")
print(f"Naive estimate overstated the effect by: ${naive_result['effect'] - raw_result['effect']:.2f}")

The final output line compares the naive result with the corrected result. It quantifies how much the telemetry filter overstated the effect.

  • Compare the Naive revenue effect output from Cell 5 with the Corrected revenue effect output from Cell 6.
  • Use the checkpoint below to document why the telemetry-filtered estimate cannot support the launch decision.

Your analysis now has a valid population plus a corrected effect estimate. Next, you will apply CUPED to reduce the estimate's uncertainty.

Apply CUPED Variance Reduction

Your repaired intention-to-treat analysis now protects the full randomized population. Its revenue estimate still carries avoidable sampling noise.

CUPED uses pre-treatment behavior to reduce that noise while preserving the business metric. You will apply it with pre_revenue.

The notebook will measure the reduction in estimator variance. This comparison reveals how much precision the adjustment adds.

In this step, get ready to:
  • Estimate theta from the relationship between pre-experiment revenue and seven-day revenue.
  • Create revenue_cuped with a 95% confidence interval.
  • Compare the squared standard errors to measure estimator variance reduction.
Estimate the CUPED coefficient

The coefficient theta captures the predictive relationship between the pre-experiment covariate and the observed outcome. Dividing their covariance by pre-experiment variance sets the adjustment strength.

Why CUPED here?

The raw intention-to-treat estimate is valid. CUPED uses information collected before treatment to lower its uncertainty without changing treatment assignment.

Collecting more users could also improve precision. CUPED extracts more information from the experiment you already ran.

  • Add one code cell beneath Cell 6 using the code-cell control in the notebook toolbar.
Center the pre-treatment covariate

Centering pre_revenue around its mean keeps the adjustment balanced across the full population. The resulting revenue_cuped remains expressed as seven-day revenue per assigned user.

  • Confirm Cell 6 defines raw_result directly above the empty code cell.

You should see the corrected estimate from the full randomized population. Cell 7 uses its standard error as the baseline for the variance comparison.

Run the CUPED comparison
  • Paste the complete CUPED workflow into the empty code cell:
# Cell 7: Apply CUPED with the pre-experiment revenue covariate.
pre = itt["pre_revenue"].to_numpy(dtype=float)
outcome = itt["revenue_7d"].to_numpy(dtype=float)
theta = np.cov(outcome, pre, ddof=1)[0, 1] / np.var(pre, ddof=1)
itt["revenue_cuped"] = outcome - theta * (pre - np.mean(pre))
cuped_result = difference_in_means(itt, "revenue_cuped")
variance_reduction = 1.0 - (cuped_result["standard_error"] ** 2 / raw_result["standard_error"] ** 2)
print(f"CUPED theta: {theta:.3f}")
print(f"CUPED revenue effect: ${cuped_result['effect']:.2f} per user")
print(f"CUPED 95% CI: [${cuped_result['lower']:.2f}, ${cuped_result['upper']:.2f}]")
print(f"Estimator variance reduction: {variance_reduction:.1%}")

What does this code do?

  • The pre and outcome arrays hold the pre-treatment covariate and seven-day revenue.
  • The np.cov and np.var calculations produce theta.
  • The np.mean calculation centers pre-experiment revenue before the adjustment creates revenue_cuped.
  • The squared standard-error ratio compares cuped_result with raw_result.

Before you run the cell, do you expect the adjusted standard error to be larger or smaller than the raw one?

  • Run Cell 7 using the play control beside the code cell.

You will see the CUPED-adjusted revenue effect with its 95% confidence interval. The reported estimator variance reduction should be greater than zero.

  • Compare cuped_result["standard_error"] with raw_result["standard_error"] through the displayed variance reduction.
  • Confirm the displayed estimator variance reduction is positive.

That is the precision win: your valid estimate now carries less sampling uncertainty.

Cell 7 not producing the comparison?

  • Run Cell 6 again if the notebook reports that an earlier result is unavailable.
  • Check that revenue_cuped uses the same spelling in the assignment and the estimator call.
  • Use Help me debug my CUPED cell. if the variance comparison still does not run.

✔️ Awesome, I've got everything!

Cell 7 is complete. Your notebook now contains the adjusted estimate plus the variance comparison.

ⓧ I'd like to double check the full code

# Cell 1: Pin the reproducible environment.
%pip install -q "numpy==2.5.2" "pandas==3.0.5" "scipy==1.18.1" "matplotlib==3.11.1"

# Cell 2: Import and verify the environment.
import sys

import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import scipy
from scipy.stats import chisquare, norm

assert sys.version_info >= (3, 12), sys.version
assert np.__version__ == "2.5.2", np.__version__
assert pd.__version__ == "3.0.5", pd.__version__
assert scipy.__version__ == "1.18.1", scipy.__version__
assert matplotlib.__version__ == "3.11.1", matplotlib.__version__

print("Python:", sys.version.split()[0])
print("NumPy:", np.__version__)
print("pandas:", pd.__version__)
print("SciPy:", scipy.__version__)
print("Matplotlib:", matplotlib.__version__)

# Cell 3: Define the decision policy and reusable estimators.
RNG_SEED = 20260829
N_USERS = 20_000
SRM_ALPHA = 0.0005
GUARDRAIL_TOLERANCE = 0.005
CONFIDENCE_LEVEL = 0.95


def difference_in_means(frame, metric):
    treatment = frame.loc[frame["variant"] == "treatment", metric].to_numpy(dtype=float)
    control = frame.loc[frame["variant"] == "control", metric].to_numpy(dtype=float)
    effect = np.mean(treatment) - np.mean(control)
    standard_error = np.sqrt(
        np.var(treatment, ddof=1) / len(treatment)
        + np.var(control, ddof=1) / len(control)
    )
    critical_value = norm.ppf(0.975)
    return {
        "effect": effect,
        "standard_error": standard_error,
        "lower": effect - critical_value * standard_error,
        "upper": effect + critical_value * standard_error,
    }


def srm_check(frame):
    observed = np.array(
        [
            (frame["variant"] == "control").sum(),
            (frame["variant"] == "treatment").sum(),
        ]
    )
    expected = np.repeat(observed.sum() / 2, 2)
    result = chisquare(f_obs=observed, f_exp=expected)
    return {
        "control": int(observed[0]),
        "treatment": int(observed[1]),
        "p_value": float(result.pvalue),
        "passed": bool(result.pvalue >= SRM_ALPHA),
    }

# Cell 4: Generate source-like tables for a randomized product experiment.
rng = np.random.default_rng(RNG_SEED)
user_id = np.arange(1, N_USERS + 1)
variant = np.repeat(np.array(["control", "treatment"]), N_USERS // 2)
rng.shuffle(variant)
is_treatment = (variant == "treatment").astype(int)

latent_value = rng.normal(0.0, 1.0, N_USERS)
pre_revenue = np.maximum(
    0.0,
    20.0 + 10.0 * latent_value + rng.normal(0.0, 6.0, N_USERS),
)
revenue_7d = np.maximum(
    0.0,
    5.0
    + 0.70 * pre_revenue
    + 2.50 * is_treatment
    + rng.normal(0.0, 10.0, N_USERS),
)
support_probability = 0.04 + 0.04 * is_treatment
support_contact = (rng.random(N_USERS) < support_probability).astype(int)

assignment = pd.DataFrame(
    {
        "user_id": user_id,
        "variant": variant,
        "pre_revenue": pre_revenue,
    }
)
billing = pd.DataFrame(
    {
        "user_id": user_id[revenue_7d > 0],
        "revenue_7d": revenue_7d[revenue_7d > 0],
    }
)
support_events = pd.DataFrame(
    {
        "user_id": user_id[support_contact == 1],
        "support_contact": 1,
    }
)

high_pre_value = pre_revenue > np.median(pre_revenue)
exposure_probability = np.where(
    is_treatment == 1,
    0.99,
    np.where(high_pre_value, 0.62, 0.98),
)
exposure_log = pd.DataFrame(
    {"user_id": user_id[rng.random(N_USERS) < exposure_probability]}
)

print("Assignment rows:", len(assignment))
print("Billing rows:", len(billing))
print("Support-event rows:", len(support_events))
print("Logged-exposure rows:", len(exposure_log))
print(assignment.head())

# Cell 5: Run the attractive but invalid telemetry-only analysis.
naive = (
    assignment.merge(
        exposure_log,
        how="inner",
        on="user_id",
        validate="one_to_one",
    )
    .merge(
        billing,
        how="left",
        on="user_id",
        validate="one_to_one",
    )
    .fillna({"revenue_7d": 0.0})
)

naive_result = difference_in_means(naive, "revenue_7d")
naive_srm = srm_check(naive)

print(f"Naive revenue effect: ${naive_result['effect']:.2f} per user")
print(f"Naive 95% CI: [${naive_result['lower']:.2f}, ${naive_result['upper']:.2f}]")
print("Observed users:", naive_srm["control"], "control and", naive_srm["treatment"], "treatment")
print(f"SRM p-value: {naive_srm['p_value']:.3g}")
print(
    "SRM status:",
    "PASS" if naive_srm["passed"] else "SRM DETECTED: do not interpret the treatment effect.",
)

# Cell 6: Rebuild the full intention-to-treat population.
itt = assignment.merge(billing, how="left", on="user_id", validate="one_to_one").merge(support_events, how="left", on="user_id", validate="one_to_one").fillna({"revenue_7d": 0.0, "support_contact": 0.0})
itt_srm = srm_check(itt)
raw_result = difference_in_means(itt, "revenue_7d")
print("Assigned users:", itt_srm["control"], "control and", itt_srm["treatment"], "treatment")
print(f"SRM p-value: {itt_srm['p_value']:.3g}")
print("SRM status:", "PASS" if itt_srm["passed"] else "FAIL")
print(f"Corrected revenue effect: ${raw_result['effect']:.2f} per user")
print(f"Corrected 95% CI: [${raw_result['lower']:.2f}, ${raw_result['upper']:.2f}]")
print(f"Naive estimate overstated the effect by: ${naive_result['effect'] - raw_result['effect']:.2f}")

# Cell 7: Apply CUPED with the pre-experiment revenue covariate.
pre = itt["pre_revenue"].to_numpy(dtype=float)
outcome = itt["revenue_7d"].to_numpy(dtype=float)
theta = np.cov(outcome, pre, ddof=1)[0, 1] / np.var(pre, ddof=1)
itt["revenue_cuped"] = outcome - theta * (pre - np.mean(pre))
cuped_result = difference_in_means(itt, "revenue_cuped")
variance_reduction = 1.0 - (cuped_result["standard_error"] ** 2 / raw_result["standard_error"] ** 2)
print(f"CUPED theta: {theta:.3f}")
print(f"CUPED revenue effect: ${cuped_result['effect']:.2f} per user")
print(f"CUPED 95% CI: [${cuped_result['lower']:.2f}, ${cuped_result['upper']:.2f}]")
print(f"Estimator variance reduction: {variance_reduction:.1%}")

Your corrected estimate now carries less uncertainty. Next, you will combine that evidence with the support-contact guardrail to make the launch decision.

Publish the Guardrail-Based Decision

Your Google Colab notebook now has a valid intention-to-treat estimate. Its CUPED estimate reduces uncertainty around the same business metric.

A launch recommendation still needs the support-contact guardrail metric. In this step, you will apply the pre-declared rule before publishing the evidence in a public GitHub repository.

In this step, get ready to:
  • Apply the support-contact guardrail to produce the launch recommendation.
  • Save the estimate table plus the two-panel decision graphic.
  • Publish the notebook plus its supporting artifacts in a public GitHub repository.
Apply the guardrail decision rule

The primary revenue metric answers whether the treatment creates value. The support-contact guardrail checks whether that value comes with an unacceptable operational cost.

  • Add a new code cell directly below Cell 7 in the Google Colab notebook from earlier.
  • Build the first part of Cell 8 by pasting this code into the new cell:
# Cell 8: Evaluate the guardrail, make the decision, and save portfolio outputs.
guardrail_result = difference_in_means(itt, "support_contact")

if guardrail_result["effect"] > GUARDRAIL_TOLERANCE:
    decision = "ITERATE"
elif cuped_result["lower"] > 0:
    decision = "SHIP"
else:
    decision = "STOP"

print(f"Support-contact increase: {guardrail_result['effect']:.2%}")
print(f"Guardrail tolerance: {GUARDRAIL_TOLERANCE:.2%}")
print("Launch decision:", decision)

How does the decision rule work?

  • The first branch selects ITERATE when the support-contact increase exceeds GUARDRAIL_TOLERANCE.
  • The second branch selects SHIP when the guardrail passes. It also requires the lower CUPED confidence bound to exceed zero.
  • The final branch selects STOP when neither launch condition passes.

Before you run the cell, do you expect the rule to select SHIP, ITERATE, or STOP? Take a moment to decide.

  • Run Cell 8 with the play button beside the cell.

You will see a support-contact increase above the tolerance. The final line reads Launch decision: ITERATE.

Seeing a different decision?

Confirm that Cell 6 filled missing support_contact values with 0.0. Rerun Cells 6 through 8 in order after fixing any mismatch.

Ask for help checking the decision inputs.

The recommendation now reflects the trade-off your stakeholders face. Next, you will package the evidence behind it.

  • Return to Cell 8.
  • Add this estimate-table code directly below print("Launch decision:", decision):
estimates = pd.DataFrame(
    {
        "analysis": ["Naive telemetry", "ITT", "ITT + CUPED"],
        "effect": [
            naive_result["effect"],
            raw_result["effect"],
            cuped_result["effect"],
        ],
        "lower": [
            naive_result["lower"],
            raw_result["lower"],
            cuped_result["lower"],
        ],
        "upper": [
            naive_result["upper"],
            raw_result["upper"],
            cuped_result["upper"],
        ],
    }
)
estimates["margin"] = estimates["upper"] - estimates["effect"]
estimates.to_csv("effect_estimates.csv", index=False)

What does this code save?

  • The estimates DataFrame places the naive result beside the repaired ITT result. It also includes the CUPED-adjusted result.
  • The margin column stores the distance from each estimate to its upper confidence bound.
  • The final line saves the table as effect_estimates.csv for machine-readable review.
  • Run Cell 8 again.
  • Select the folder icon in Colab's left sidebar.
  • Confirm that effect_estimates.csv appears in the runtime file list.

CSV file missing?

Confirm that the to_csv() line remains inside Cell 8. A red cell error must be resolved before Colab reaches that line.

Ask for help finding the failed line.

  • Return to Cell 8.
  • Complete the cell by adding this plotting code directly below the to_csv() line:
x = np.arange(len(estimates))
fig, axes = plt.subplots(1, 2, figsize=(12, 4.5))
axes[0].errorbar(
    x,
    estimates["effect"].to_numpy(),
    yerr=estimates["margin"].to_numpy(),
    fmt="o",
    capsize=5,
    color="#2563eb",
)
axes[0].axhline(0, color="#111827", linewidth=1)
axes[0].set_xticks(x, estimates["analysis"].tolist(), rotation=15)
axes[0].set_ylabel("Revenue effect per assigned user")
axes[0].set_title("Estimate changes after validity repair")

axes[1].bar(
    ["Observed increase", "Tolerance"],
    [guardrail_result["effect"], GUARDRAIL_TOLERANCE],
    color=["#dc2626", "#9ca3af"],
)
axes[1].set_ylabel("Support-contact rate difference")
axes[1].set_title(f"Launch decision: {decision}")

fig.tight_layout()
fig.savefig("experiment_readout.png", dpi=160, bbox_inches="tight")
plt.show()

print("Saved effect_estimates.csv and experiment_readout.png")

How does the graphic support the decision?

  • The left panel compares how the effect estimate changes after repairing the population. It also shows the tighter CUPED interval.
  • The right panel compares the observed support-contact increase with the pre-declared tolerance.
  • The figure title carries the resulting ITERATE recommendation into the published artifact.

Before you rerun Cell 8, do you expect the revenue panel or the guardrail panel to determine the recommendation? Check your prediction against the graphic.

  • Run Cell 8 one final time.

You will see a two-panel graphic beneath the cell. Its right panel is titled Launch decision: ITERATE.

That is the decision artifact complete. Your notebook now explains why a positive primary effect does not trigger an immediate launch.

Graphic not displayed?

Check that fig, axes is created before either panel uses axes. Confirm that plt.show() remains near the end of Cell 8.

Ask for help checking the plotting section.

✔️ Awesome, I've got everything!

Your completed notebook should now contain Cells 1 through 8. Confirm that the latest Colab save has completed before downloading it.

ⓧ I'd like to double check the full code

# Cell 1: Pin the reproducible environment.
%pip install -q "numpy==2.5.2" "pandas==3.0.5" "scipy==1.18.1" "matplotlib==3.11.1"

# Cell 2: Import and verify the environment.
import sys

import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import scipy
from scipy.stats import chisquare, norm

assert sys.version_info >= (3, 12), sys.version
assert np.__version__ == "2.5.2", np.__version__
assert pd.__version__ == "3.0.5", pd.__version__
assert scipy.__version__ == "1.18.1", scipy.__version__
assert matplotlib.__version__ == "3.11.1", matplotlib.__version__

print("Python:", sys.version.split()[0])
print("NumPy:", np.__version__)
print("pandas:", pd.__version__)
print("SciPy:", scipy.__version__)
print("Matplotlib:", matplotlib.__version__)

# Cell 3: Define the decision policy and reusable estimators.
RNG_SEED = 20260829
N_USERS = 20_000
SRM_ALPHA = 0.0005
GUARDRAIL_TOLERANCE = 0.005
CONFIDENCE_LEVEL = 0.95


def difference_in_means(frame, metric):
    treatment = frame.loc[frame["variant"] == "treatment", metric].to_numpy(dtype=float)
    control = frame.loc[frame["variant"] == "control", metric].to_numpy(dtype=float)
    effect = np.mean(treatment) - np.mean(control)
    standard_error = np.sqrt(
        np.var(treatment, ddof=1) / len(treatment)
        + np.var(control, ddof=1) / len(control)
    )
    critical_value = norm.ppf(0.975)
    return {
        "effect": effect,
        "standard_error": standard_error,
        "lower": effect - critical_value * standard_error,
        "upper": effect + critical_value * standard_error,
    }


def srm_check(frame):
    observed = np.array(
        [
            (frame["variant"] == "control").sum(),
            (frame["variant"] == "treatment").sum(),
        ]
    )
    expected = np.repeat(observed.sum() / 2, 2)
    result = chisquare(f_obs=observed, f_exp=expected)
    return {
        "control": int(observed[0]),
        "treatment": int(observed[1]),
        "p_value": float(result.pvalue),
        "passed": bool(result.pvalue >= SRM_ALPHA),
    }

# Cell 4: Generate source-like tables for a randomized product experiment.
rng = np.random.default_rng(RNG_SEED)
user_id = np.arange(1, N_USERS + 1)
variant = np.repeat(np.array(["control", "treatment"]), N_USERS // 2)
rng.shuffle(variant)
is_treatment = (variant == "treatment").astype(int)

latent_value = rng.normal(0.0, 1.0, N_USERS)
pre_revenue = np.maximum(
    0.0,
    20.0 + 10.0 * latent_value + rng.normal(0.0, 6.0, N_USERS),
)
revenue_7d = np.maximum(
    0.0,
    5.0
    + 0.70 * pre_revenue
    + 2.50 * is_treatment
    + rng.normal(0.0, 10.0, N_USERS),
)
support_probability = 0.04 + 0.04 * is_treatment
support_contact = (rng.random(N_USERS) < support_probability).astype(int)

assignment = pd.DataFrame(
    {
        "user_id": user_id,
        "variant": variant,
        "pre_revenue": pre_revenue,
    }
)
billing = pd.DataFrame(
    {
        "user_id": user_id[revenue_7d > 0],
        "revenue_7d": revenue_7d[revenue_7d > 0],
    }
)
support_events = pd.DataFrame(
    {
        "user_id": user_id[support_contact == 1],
        "support_contact": 1,
    }
)

high_pre_value = pre_revenue > np.median(pre_revenue)
exposure_probability = np.where(
    is_treatment == 1,
    0.99,
    np.where(high_pre_value, 0.62, 0.98),
)
exposure_log = pd.DataFrame(
    {"user_id": user_id[rng.random(N_USERS) < exposure_probability]}
)

print("Assignment rows:", len(assignment))
print("Billing rows:", len(billing))
print("Support-event rows:", len(support_events))
print("Logged-exposure rows:", len(exposure_log))
print(assignment.head())

# Cell 5: Run the attractive but invalid telemetry-only analysis.
naive = (
    assignment.merge(
        exposure_log,
        how="inner",
        on="user_id",
        validate="one_to_one",
    )
    .merge(
        billing,
        how="left",
        on="user_id",
        validate="one_to_one",
    )
    .fillna({"revenue_7d": 0.0})
)

naive_result = difference_in_means(naive, "revenue_7d")
naive_srm = srm_check(naive)

print(f"Naive revenue effect: ${naive_result['effect']:.2f} per user")
print(f"Naive 95% CI: [${naive_result['lower']:.2f}, ${naive_result['upper']:.2f}]")
print("Observed users:", naive_srm["control"], "control and", naive_srm["treatment"], "treatment")
print(f"SRM p-value: {naive_srm['p_value']:.3g}")
print(
    "SRM status:",
    "PASS" if naive_srm["passed"] else "SRM DETECTED: do not interpret the treatment effect.",
)

# Cell 6: Rebuild the full intention-to-treat population.
itt = (
    assignment.merge(
        billing,
        how="left",
        on="user_id",
        validate="one_to_one",
    )
    .merge(
        support_events,
        how="left",
        on="user_id",
        validate="one_to_one",
    )
    .fillna({"revenue_7d": 0.0, "support_contact": 0.0})
)

itt_srm = srm_check(itt)
raw_result = difference_in_means(itt, "revenue_7d")

print("Assigned users:", itt_srm["control"], "control and", itt_srm["treatment"], "treatment")
print(f"SRM p-value: {itt_srm['p_value']:.3g}")
print("SRM status:", "PASS" if itt_srm["passed"] else "FAIL")
print(f"Corrected revenue effect: ${raw_result['effect']:.2f} per user")
print(f"Corrected 95% CI: [${raw_result['lower']:.2f}, ${raw_result['upper']:.2f}]")
print(f"Naive estimate overstated the effect by: ${naive_result['effect'] - raw_result['effect']:.2f}")

# Cell 7: Apply CUPED with the pre-experiment revenue covariate.
pre = itt["pre_revenue"].to_numpy(dtype=float)
outcome = itt["revenue_7d"].to_numpy(dtype=float)
theta = np.cov(outcome, pre, ddof=1)[0, 1] / np.var(pre, ddof=1)

itt["revenue_cuped"] = outcome - theta * (pre - np.mean(pre))
cuped_result = difference_in_means(itt, "revenue_cuped")
variance_reduction = 1.0 - (
    cuped_result["standard_error"] ** 2 / raw_result["standard_error"] ** 2
)

print(f"CUPED theta: {theta:.3f}")
print(f"CUPED revenue effect: ${cuped_result['effect']:.2f} per user")
print(f"CUPED 95% CI: [${cuped_result['lower']:.2f}, ${cuped_result['upper']:.2f}]")
print(f"Estimator variance reduction: {variance_reduction:.1%}")

# Cell 8: Evaluate the guardrail, make the decision, and save portfolio outputs.
guardrail_result = difference_in_means(itt, "support_contact")

if guardrail_result["effect"] > GUARDRAIL_TOLERANCE:
    decision = "ITERATE"
elif cuped_result["lower"] > 0:
    decision = "SHIP"
else:
    decision = "STOP"

print(f"Support-contact increase: {guardrail_result['effect']:.2%}")
print(f"Guardrail tolerance: {GUARDRAIL_TOLERANCE:.2%}")
print("Launch decision:", decision)

estimates = pd.DataFrame(
    {
        "analysis": ["Naive telemetry", "ITT", "ITT + CUPED"],
        "effect": [
            naive_result["effect"],
            raw_result["effect"],
            cuped_result["effect"],
        ],
        "lower": [
            naive_result["lower"],
            raw_result["lower"],
            cuped_result["lower"],
        ],
        "upper": [
            naive_result["upper"],
            raw_result["upper"],
            cuped_result["upper"],
        ],
    }
)
estimates["margin"] = estimates["upper"] - estimates["effect"]
estimates.to_csv("effect_estimates.csv", index=False)

x = np.arange(len(estimates))
fig, axes = plt.subplots(1, 2, figsize=(12, 4.5))
axes[0].errorbar(
    x,
    estimates["effect"].to_numpy(),
    yerr=estimates["margin"].to_numpy(),
    fmt="o",
    capsize=5,
    color="#2563eb",
)
axes[0].axhline(0, color="#111827", linewidth=1)
axes[0].set_xticks(x, estimates["analysis"].tolist(), rotation=15)
axes[0].set_ylabel("Revenue effect per assigned user")
axes[0].set_title("Estimate changes after validity repair")

axes[1].bar(
    ["Observed increase", "Tolerance"],
    [guardrail_result["effect"], GUARDRAIL_TOLERANCE],
    color=["#dc2626", "#9ca3af"],
)
axes[1].set_ylabel("Support-contact rate difference")
axes[1].set_title(f"Launch decision: {decision}")

fig.tight_layout()
fig.savefig("experiment_readout.png", dpi=160, bbox_inches="tight")
plt.show()

print("Saved effect_estimates.csv and experiment_readout.png")
Create the executive README

The notebook proves the analysis. The README.md file gives a reviewer the decision before they inspect any code.

Saving a Markdown file in TextEdit can feel fiddly because new documents may begin as rich text. Converting the document first keeps the repository file readable.

  • Press Cmd+Space to open Spotlight on your Mac.
  • Type TextEdit and press Enter.
  • Click File in the menu bar.
  • Click New.
  • Click Format in the menu bar.
  • Click Make Plain Text.
  • Add the first half of the executive README by pasting this content into the plain-text document:
# Trustworthy Experiment Decision Lab

## Executive decision

ITERATE. The corrected intention-to-treat revenue effect is positive, but the support-contact guardrail exceeds the pre-declared increase tolerance. The treatment should be revised and retested before launch.

![Experiment decision readout](experiment_readout.png)

## The trap

A telemetry-only population produced an attractive revenue result. A sample ratio mismatch check showed that the analyzed treatment and control populations no longer matched the randomized allocation. The result was withheld rather than interpreted.

## The repair

The valid analysis starts from every randomized assignment and left-joins outcomes from trusted billing and support-event sources. Missing billing or support events are encoded as zero outcomes, preserving the intention-to-treat population.

## The estimate

The notebook reports a difference in seven-day revenue per assigned user with a 95% confidence interval. CUPED uses pre-experiment revenue to reduce estimator variance while keeping treatment assignment untouched.

What does this section communicate?

The opening states the recommendation before describing the attractive result that failed validation. The repair section then explains why the intention-to-treat population supports a trustworthy estimate.

The image reference displays experiment_readout.png when GitHub renders the finished README.

  • Click File in the menu bar.
  • Click Save.
  • Enter README.md as the document name.
  • Choose Downloads from the Where menu.
  • Click Save.

The first half of README.md is now stored in your Downloads folder.

  • Return to the end of the TextEdit document.
  • Leave one blank line after the estimate paragraph.
  • Complete the README by pasting this content:
## The decision rule

- ITERATE when the support-contact increase exceeds 0.5 percentage points.
- SHIP when the guardrail passes and the CUPED confidence interval is above zero.
- STOP otherwise.

## Reproduce the analysis

Open `trustworthy_experiment_lab.ipynb`, run the cells from top to bottom, and compare your outputs with `effect_estimates.csv` and `experiment_readout.png`.

## Data disclosure

All data in this repository is synthetic and generated inside the notebook with a fixed random seed. No personal, customer, or company data is included.

Why include these final sections?

The explicit rule makes the recommendation auditable. A reviewer can see which condition selected ITERATE.

The reproduction instructions connect the notebook to its saved outputs. The disclosure confirms that the repository contains synthetic data.

  • Save README.md by pressing Cmd+S.
  • Confirm that the TextEdit title bar shows README.md.

Seeing README.md.txt?

Open TextEdit settings. Select Open and Save before clearing Add “.txt” extension to plain text files.

Save the document again as README.md. Ask for help correcting the filename.

✔️ Awesome, I've got everything!

Your README.md file should now contain the executive decision plus the complete reproduction and data-disclosure sections.

ⓧ I'd like to double check the full code

# Trustworthy Experiment Decision Lab

## Executive decision

ITERATE. The corrected intention-to-treat revenue effect is positive, but the support-contact guardrail exceeds the pre-declared increase tolerance. The treatment should be revised and retested before launch.

![Experiment decision readout](experiment_readout.png)

## The trap

A telemetry-only population produced an attractive revenue result. A sample ratio mismatch check showed that the analyzed treatment and control populations no longer matched the randomized allocation. The result was withheld rather than interpreted.

## The repair

The valid analysis starts from every randomized assignment and left-joins outcomes from trusted billing and support-event sources. Missing billing or support events are encoded as zero outcomes, preserving the intention-to-treat population.

## The estimate

The notebook reports a difference in seven-day revenue per assigned user with a 95% confidence interval. CUPED uses pre-experiment revenue to reduce estimator variance while keeping treatment assignment untouched.

## The decision rule

- ITERATE when the support-contact increase exceeds 0.5 percentage points.
- SHIP when the guardrail passes and the CUPED confidence interval is above zero.
- STOP otherwise.

## Reproduce the analysis

Open `trustworthy_experiment_lab.ipynb`, run the cells from top to bottom, and compare your outputs with `effect_estimates.csv` and `experiment_readout.png`.

## Data disclosure

All data in this repository is synthetic and generated inside the notebook with a fixed random seed. No personal, customer, or company data is included.
Publish the portfolio repository

GitHub renders an uploaded .ipynb notebook as a static page. Reviewers can inspect the saved outputs without running your Colab environment.

  • Switch back to the Google Colab notebook from earlier.
  • Click File in the top menu.
  • Click Download .ipynb to save trustworthy_experiment_lab.ipynb to your Mac.
  • Select the folder icon in Colab's left sidebar.
  • Open the options menu beside effect_estimates.csv.
  • Click Download.
  • Open the options menu beside experiment_readout.png.
  • Click Download.

Check your upload set

Your Downloads folder should contain trustworthy_experiment_lab.ipynb, README.md, effect_estimates.csv, plus experiment_readout.png.

These four files form one reviewable experiment readout.

  • Open GitHub in a new browser tab.
  • Sign in with your GitHub account.
  • Open the creation menu in the top-right corner.
  • Select New repository.
  • Enter trustworthy-experiment-lab as the repository name.
  • Select Public as the repository visibility.
  • Click Create repository.

Your empty public repository is ready for its first commit. The next upload publishes the full decision package together.

  • Click Add file.
  • Click Upload files.
  • Click choose your files.
  • Select trustworthy_experiment_lab.ipynb from your Downloads folder.
  • Select README.md from your Downloads folder.
  • Select effect_estimates.csv from your Downloads folder.
  • Select experiment_readout.png from your Downloads folder.
  • Enter Publish guardrail-based experiment decision as the commit message.

Before you publish the commit, which four filenames do you expect to see on the repository page? Check that expectation after GitHub finishes processing the upload.

  • Click Propose changes.

You will see the rendered README plus all four files in trustworthy-experiment-lab. Opening the notebook shows its saved cells and outputs as static HTML.

Missing a published file?

Return to the repository's upload flow when a file is absent. Confirm that each filename matches the names used by the notebook and README.

Ask for help checking the upload set.

Your analysis is now a public decision artifact. Reviewers can trace the false win through the validity repair to the final ITERATE recommendation.

Secret mission

Challenge the Average

The overall revenue effect may hide different responses across customer segments. Test whether a pre-treatment value split provides enough interaction evidence to justify a targeted rollout.

Clean Up Your Resources

Clean Up Your Resources

Your Google Colab notebook and public GitHub repository are free to keep, so there are no ongoing project costs. Choose whether to keep the portfolio project available, pause your work, or delete both saved copies.

Resources you used:

  • A Google Drive-backed trustworthy_experiment_lab.ipynb notebook containing the complete experiment analysis.
  • The public trustworthy-experiment-lab repository containing README.md, the rendered notebook, effect_estimates.csv, and experiment_readout.png.

Keep everything running

No action is needed. Choose this option if you want your notebook and public experiment readout to remain available for future analysis or portfolio reviews.

  • Keep trustworthy_experiment_lab.ipynb in Google Drive so you can rerun the synthetic analysis later.
  • Keep trustworthy-experiment-lab public so reviewers can inspect your decision process.
  • Return to the saved notebook whenever you want to test another pre-treatment segment.

Pause - I'll come back to this later

This option ends your active Colab session while preserving every saved artifact. The temporary runtime can be recycled without affecting the notebook in Google Drive.

  • Close the browser tab containing the active Colab notebook.
  • Leave trustworthy_experiment_lab.ipynb saved in Google Drive.
  • Leave trustworthy-experiment-lab available on GitHub.

Delete - I don't want to use this again

Permanent deletion can feel severe. Your saved portfolio copies disappear when each service finishes deletion.

  • In Google Drive, locate trustworthy_experiment_lab.ipynb in your file list.
  • Use the notebook's item menu to move it to trash.
  • Empty Google Drive trash to permanently remove the notebook.
  • Return to the trustworthy-experiment-lab repository on GitHub.
  • Open the repository's danger area within its settings.
  • Select the control that deletes the repository.
  • Enter trustworthy-experiment-lab when the confirmation prompt asks for the repository name.
  • Confirm the permanent repository deletion.

Deleting trustworthy-experiment-lab also removes its published README.md, rendered notebook, effect_estimates.csv, and experiment_readout.png.

Nice Work!

Nice Work!

You made it! Your public experiment decision lab now rejects a false win before making an uncertainty-aware launch recommendation.

You've learned how to:

  • Reject a misleading telemetry-only revenue win after a sample ratio mismatch check exposed the invalid analysis population.
  • Rebuild the full randomized population with intention-to-treat analysis. Estimate the revenue effect with a 95% confidence interval. Apply CUPED to reduce estimator variance.
  • Use a support-contact guardrail metric to convert a positive primary effect into an ITERATE recommendation. Publish the evidence in a public GitHub repository for reviewers.
  • Secret Mission: Test treatment-effect heterogeneity across pre-treatment value segments. Use a difference-in-effects 95% confidence interval to decide whether a targeted rollout is justified.

Ready to quiz yourself?