Analyze an E-Commerce A/B Test
Build a Colab notebook that runs a full A/B test analysis with a ship decision.
Introduction
30 Second Summary
Every time you buy something online, there's a good chance your checkout page was tested against a different version to see which one makes more people click "Buy." Companies make million-dollar decisions based on these experiments. The question is: how do you tell if a result is real or just random noise?
In this project, you will build a complete A/B test analysis notebook in Google Colab that takes raw experiment data, runs a statistical test, validates the experiment's integrity, and produces a clear ship/no-ship recommendation backed by evidence.
What You'll Build
When you run the final cell in your notebook, you'll see a formatted decision summary that tells a product manager exactly what happened in the experiment, whether the result is trustworthy, and what the team should do next.
By the end of this project, you'll have:
- A polished notebook that generates synthetic experiment data with a known ground truth, visualizes the conversion difference, and walks through the full analysis workflow you'd present in a product analytics interview.
- A validated experiment with a Sample Ratio Mismatch check confirming balanced group assignment and a statistical power analysis proving the test had enough users to detect the effect.
- A defensible ship/no-ship recommendation that combines statistical significance with practical significance to answer the real business question: is this lift big enough to actually matter?
- Secret Mission: Break down results by user segment to uncover hidden patterns that change the recommendation entirely.
Are there any prerequisites?
You'll need basic Python proficiency (loops, functions, and working with pandas DataFrames) and a fundamental understanding of statistics (what a mean, standard deviation, and proportion are). No local installs are required. Everything runs in Google Colab through your browser.
Before We Start
Before we dive in, let's lock in what you're about to build and why it matters. Taking a moment to frame the project helps you approach each step with purpose, not just paste code.
Set Up Your Colab Notebook
Before you can analyze an experiment, you need a workspace with all the statistical libraries ready to go. Configuring Python environments locally can eat up time on version conflicts and missing packages.
Google Colab gives you a cloud-based notebook with numpy, pandas, scipy, statsmodels, and more already pre-installed. In this step, you'll create the notebook and verify everything works.
In this step, get ready to:
- Create and name your Colab notebook.
- Import and verify all statistical libraries.
- Document the experiment context to frame the notebook professionally.
Create a new Colab notebook
Google Colab runs entirely in your browser. All you need is a Google account to get started.
- Open your browser and navigate to colab.research.google.com.
- Sign in with your Google account if prompted.
- Click New notebook in the popup dialog.
- Click on the notebook title at the top left (it will say something like Untitled0.ipynb).
- Rename it to AB_Test_Analysis_CartWave and press Enter.
You should see your notebook title update to AB_Test_Analysis_CartWave.ipynb at the top of the page.
Don't see the New notebook option?
If the popup doesn't appear, click File in the top menu bar, then New notebook. Make sure you're signed into your Google account. If you see a "recent notebooks" view instead, look for the New notebook button in the bottom left of the dialog.
Can't get a new notebook to open?
Add imports and verify your libraries
Your notebook already has an empty code cell waiting. This is where you'll import every library the analysis needs and confirm they load correctly.
- Click into the first empty code cell in your notebook and paste the following code:
import numpy as np
import pandas as pd
from scipy.stats import chi2_contingency
from statsmodels.stats.proportion import proportions_ztest, proportion_effectsize
from statsmodels.stats.power import NormalIndPower
import matplotlib.pyplot as plt
import seaborn as sns
# Print versions for reproducibility
print(f"numpy: {np.__version__}")
print(f"pandas: {pd.__version__}")
import scipy; print(f"scipy: {scipy.__version__}")
import statsmodels; print(f"statsmodels: {statsmodels.__version__}")What does this code do?
- numpy and pandas handle numerical arrays and tabular data.
- scipy.stats provides the chi-square test you'll use to check experiment validity.
- statsmodels supplies the two-proportion z-test and power analysis tools for your hypothesis test.
- matplotlib and seaborn produce the charts that visualize your results.
- The version prints at the end confirm everything loaded. They also make your notebook reproducible for anyone who reads it later.
Before you run this cell, what do you expect to see in the output? Think about what those print statements will produce.
- Run the cell by clicking the Play button on the left side of the cell, or press Shift+Enter.
You should see version numbers printed for numpy, pandas, scipy, and statsmodels, with no error messages above them. The exact version numbers will vary based on Colab's current runtime.
Seeing an import error?
All these libraries come pre-installed in Colab. If you see an import error, check for typos in the import lines. Pay close attention to capitalization and underscores in chi2_contingency and proportions_ztest.
If you accidentally deleted the cell, click + Code in the top left to add a new one and paste the code again.
Help me fix an import error in my Colab notebook.
Document the experiment context
A professional analysis notebook starts with context. Anyone reading your notebook should immediately understand what experiment was run, what the hypothesis was, and how you'll decide whether to ship.
- Click + Text in the top left toolbar to add a new markdown cell below your imports cell.
- Paste the following markdown into the new text cell:
# A/B Test Analysis: CartWave Checkout Redesign
## Experiment Context
- **Company:** CartWave (e-commerce)
- **Test:** New checkout page design vs. current design
- **Primary Metric:** Conversion rate (completed purchase / reached checkout)
- **Duration:** 14 days
- **Sample:** 20,000 users (10,000 per group)
- **Hypothesis:** The new checkout design will increase conversion rate by at least 1 percentage point (from ~12% to ~13%).
## Decision Criteria
- Statistical significance: p < 0.05 (two-sided)
- Practical significance: Lower bound of 95% CI for the lift > 1 percentage point
- Experiment validity: No Sample Ratio Mismatch, adequate statistical power (>80%)Why document the experiment up front?
Defining your decision criteria before seeing results prevents you from moving the goalposts after the fact. This is called pre-registration in experiment design.
The criteria here require three things to pass before you ship: statistical significance (p < 0.05), practical significance (the lift must exceed 1 percentage point), and experiment validity (clean data split and enough power).
- Click outside the markdown cell to render it. You should see a formatted heading, a bulleted list of experiment details, and the decision criteria displayed cleanly.
Seeing raw markdown instead of formatted text?
Double-check that you added a Text cell (not a Code cell). If you pasted into a code cell by mistake, cut the content, delete the cell, click + Text, and paste again.
My markdown cell isn't rendering properly in Colab.
✔️ Awesome, I've got everything!
Great. Make sure both cells are saved. Colab auto-saves to your Google Drive, so you're all set.
ⓧ I'd like to double check the full code
Here's what your notebook should contain at this point. Cell 1 (Code):
import numpy as np
import pandas as pd
from scipy.stats import chi2_contingency
from statsmodels.stats.proportion import proportions_ztest, proportion_effectsize
from statsmodels.stats.power import NormalIndPower
import matplotlib.pyplot as plt
import seaborn as sns
# Print versions for reproducibility
print(f"numpy: {np.__version__}")
print(f"pandas: {pd.__version__}")
import scipy; print(f"scipy: {scipy.__version__}")
import statsmodels; print(f"statsmodels: {statsmodels.__version__}")Cell 2 (Markdown):
# A/B Test Analysis: CartWave Checkout Redesign
## Experiment Context
- **Company:** CartWave (e-commerce)
- **Test:** New checkout page design vs. current design
- **Primary Metric:** Conversion rate (completed purchase / reached checkout)
- **Duration:** 14 days
- **Sample:** 20,000 users (10,000 per group)
- **Hypothesis:** The new checkout design will increase conversion rate by at least 1 percentage point (from ~12% to ~13%).
## Decision Criteria
- Statistical significance: p < 0.05 (two-sided)
- Practical significance: Lower bound of 95% CI for the lift > 1 percentage point
- Experiment validity: No Sample Ratio Mismatch, adequate statistical power (>80%)Your analysis workspace is ready. You have a named notebook, verified libraries, and documented criteria that will guide every decision in the analysis. Next up, you'll generate the experiment data and get your first look at the raw conversion numbers.
Generate and Explore the Experiment Data
Your notebook has the libraries loaded and the experiment framed. Now it's time to look at the actual data.
Before running any statistical tests, you need to see the raw numbers and get a feel for what the experiment produced. In this step you'll generate a realistic dataset, compute the conversion rates for each group, and visualize the difference. The chart will look promising. But looking promising is not the same as being real.
In this step, get ready to:
- Generate a 20,000-row synthetic dataset with a known conversion lift built in.
- Compute per-group conversion rates and the observed lift.
- Visualize control vs. treatment with a bar chart that shows the raw difference.
Write the data generation function
You'll create synthetic experiment data with a known ground truth baked in. This means you control the baseline conversion rate and the exact treatment effect, so later you can verify your analysis is working correctly.
- Add a new code cell below your markdown context cell and paste the following:
def generate_ab_data(n_per_group=10000, baseline_rate=0.12, treatment_effect=0.02, seed=42):
"""Generate realistic A/B test data with known ground truth."""
np.random.seed(seed)
# Control group
control = pd.DataFrame({
'user_id': range(1, n_per_group + 1),
'group': 'control',
'converted': np.random.binomial(1, baseline_rate, n_per_group)
})
# Treatment group
treatment = pd.DataFrame({
'user_id': range(n_per_group + 1, 2 * n_per_group + 1),
'group': 'treatment',
'converted': np.random.binomial(1, baseline_rate + treatment_effect, n_per_group)
})
return pd.concat([control, treatment], ignore_index=True)
# Generate the data
df = generate_ab_data()
print(f"Dataset shape: {df.shape}")
print(f"\nFirst few rows:")
df.head()What does this code do?
This function simulates what a real experiment logging system would produce.
- Each user gets a random 0 or 1 for converted, drawn from a binomial distribution with the group's true conversion probability.
- The control group converts at 12% (baseline_rate=0.12). The treatment group converts at 14% (baseline + 2 percentage point effect).
- Setting seed=42 makes the random draws reproducible. Every time you run this cell, you get the same dataset.
- Run the cell by pressing Shift+Enter.
You should see Dataset shape: (20000, 3) followed by a table showing the first five rows with columns user_id, group, and converted.
Seeing a NameError?
If you see NameError: name 'np' is not defined, your import cell (Cell 1) hasn't been run yet. Switch back to Cell 1 and run it first, then re-run this cell.
help me fix a NameError in my Colab notebook
Compute summary statistics
Now that you have raw data, you need to distill it into the numbers that matter: how many users converted in each group, and what's the difference.
- Add a new code cell below and paste the following:
# Calculate conversion rates per group
summary = df.groupby('group')['converted'].agg(['sum', 'count', 'mean'])
summary.columns = ['conversions', 'total_users', 'conversion_rate']
summary['conversion_rate_pct'] = summary['conversion_rate'] * 100
print("=" * 50)
print("EXPERIMENT SUMMARY")
print("=" * 50)
print(f"\nControl group:")
print(f" Users: {summary.loc['control', 'total_users']:,.0f}")
print(f" Conversions: {summary.loc['control', 'conversions']:,.0f}")
print(f" Conversion rate: {summary.loc['control', 'conversion_rate_pct']:.2f}%")
print(f"\nTreatment group:")
print(f" Users: {summary.loc['treatment', 'total_users']:,.0f}")
print(f" Conversions: {summary.loc['treatment', 'conversions']:,.0f}")
print(f" Conversion rate: {summary.loc['treatment', 'conversion_rate_pct']:.2f}%")
# Calculate lift
absolute_lift = summary.loc['treatment', 'conversion_rate'] - summary.loc['control', 'conversion_rate']
relative_lift = absolute_lift / summary.loc['control', 'conversion_rate']
print(f"\n{'─' * 50}")
print(f"Observed absolute lift: {absolute_lift * 100:+.2f} percentage points")
print(f"Observed relative lift: {relative_lift * 100:+.1f}%")
print(f"{'─' * 50}")What does this code do?
- groupby('group') splits the DataFrame into control and treatment, then agg computes the sum (conversions), count (users), and mean (conversion rate) in one step.
- absolute_lift is the raw difference in conversion rates (treatment minus control). This is what the business cares about.
- relative_lift expresses that difference as a percentage of the control rate. It answers "how much better is treatment, proportionally?"
- Run the cell by pressing Shift+Enter.
You should see a formatted summary showing both groups' conversion rates and the observed lift. The treatment group converts roughly 2 percentage points higher than control.
Seeing a KeyError on 'control'?
This means the df variable doesn't exist yet. Make sure you ran Cell 3 successfully before running this cell.
help me fix a KeyError in my summary statistics cell
Visualize the conversion difference
Numbers in a table are precise, but a chart makes the difference visceral. You'll create a bar chart with 95% confidence interval error bars so you can see both the lift and the uncertainty around it.
- Add a new code cell below and paste the following:
fig, ax = plt.subplots(figsize=(8, 5))
groups = ['Control', 'Treatment']
rates = [summary.loc['control', 'conversion_rate_pct'],
summary.loc['treatment', 'conversion_rate_pct']]
# Standard error for error bars
se_control = np.sqrt(summary.loc['control', 'conversion_rate'] * (1 - summary.loc['control', 'conversion_rate']) / summary.loc['control', 'total_users']) * 100
se_treatment = np.sqrt(summary.loc['treatment', 'conversion_rate'] * (1 - summary.loc['treatment', 'conversion_rate']) / summary.loc['treatment', 'total_users']) * 100
bars = ax.bar(groups, rates, yerr=[se_control * 1.96, se_treatment * 1.96],
capsize=10, color=['#4A90D9', '#7BC67E'], edgecolor='black', linewidth=0.8)
ax.set_ylabel('Conversion Rate (%)', fontsize=12)
ax.set_title('Checkout Conversion: Control vs. Treatment', fontsize=14, fontweight='bold')
ax.set_ylim(0, max(rates) * 1.4)
# Annotate bars
for bar, rate in zip(bars, rates):
ax.text(bar.get_x() + bar.get_width() / 2, bar.get_height() + 0.3,
f'{rate:.2f}%', ha='center', va='bottom', fontsize=12, fontweight='bold')
# Add lift annotation
ax.annotate(f'Lift: {absolute_lift * 100:+.2f} pp',
xy=(1, rates[1]), xytext=(1.3, rates[1] + 1),
fontsize=11, color='green', fontweight='bold',
arrowprops=dict(arrowstyle='->', color='green'))
plt.tight_layout()
plt.show()What does this chart show?
- Each bar represents the conversion rate for one group. The treatment bar is visibly taller.
- The error bars show 95% confidence intervals calculated from the standard error of each proportion. They give you a visual sense of uncertainty around each rate.
- The green "Lift" annotation calls out the raw difference in percentage points. This is the number you want to confirm is not just noise.
- Run the cell by pressing Shift+Enter.
Before you look at the output, predict: will the error bars overlap between the two groups, or will they be clearly separated?
You should see a bar chart with two bars. The treatment bar is taller, and a green arrow labels the lift. The error bars are relatively small because of the large sample size (10,000 per group).
Chart not showing?
If you see a blank output or text like <Figure size ...> instead of a chart, make sure plt.show() is the last line in your cell. In Colab, inline plotting is enabled by default, but calling plt.show() explicitly ensures the chart renders.
help me display a matplotlib chart in Google Colab
✔️ Awesome, I've got everything!
Great. Make sure all three cells (3, 4, and 5) show their output and that your notebook is saving automatically.
ⓧ I'd like to double check the full code
Scroll up through your notebook and confirm each cell matches. You should have Cell 3 (data generation), Cell 4 (summary statistics), and Cell 5 (visualization), each producing output when run.
Cell 3 should match:
def generate_ab_data(n_per_group=10000, baseline_rate=0.12, treatment_effect=0.02, seed=42):
"""Generate realistic A/B test data with known ground truth."""
np.random.seed(seed)
# Control group
control = pd.DataFrame({
'user_id': range(1, n_per_group + 1),
'group': 'control',
'converted': np.random.binomial(1, baseline_rate, n_per_group)
})
# Treatment group
treatment = pd.DataFrame({
'user_id': range(n_per_group + 1, 2 * n_per_group + 1),
'group': 'treatment',
'converted': np.random.binomial(1, baseline_rate + treatment_effect, n_per_group)
})
return pd.concat([control, treatment], ignore_index=True)
# Generate the data
df = generate_ab_data()
print(f"Dataset shape: {df.shape}")
print(f"\nFirst few rows:")
df.head()Cell 4 should match:
# Calculate conversion rates per group
summary = df.groupby('group')['converted'].agg(['sum', 'count', 'mean'])
summary.columns = ['conversions', 'total_users', 'conversion_rate']
summary['conversion_rate_pct'] = summary['conversion_rate'] * 100
print("=" * 50)
print("EXPERIMENT SUMMARY")
print("=" * 50)
print(f"\nControl group:")
print(f" Users: {summary.loc['control', 'total_users']:,.0f}")
print(f" Conversions: {summary.loc['control', 'conversions']:,.0f}")
print(f" Conversion rate: {summary.loc['control', 'conversion_rate_pct']:.2f}%")
print(f"\nTreatment group:")
print(f" Users: {summary.loc['treatment', 'total_users']:,.0f}")
print(f" Conversions: {summary.loc['treatment', 'conversions']:,.0f}")
print(f" Conversion rate: {summary.loc['treatment', 'conversion_rate_pct']:.2f}%")
# Calculate lift
absolute_lift = summary.loc['treatment', 'conversion_rate'] - summary.loc['control', 'conversion_rate']
relative_lift = absolute_lift / summary.loc['control', 'conversion_rate']
print(f"\n{'─' * 50}")
print(f"Observed absolute lift: {absolute_lift * 100:+.2f} percentage points")
print(f"Observed relative lift: {relative_lift * 100:+.1f}%")
print(f"{'─' * 50}")Cell 5 should match:
fig, ax = plt.subplots(figsize=(8, 5))
groups = ['Control', 'Treatment']
rates = [summary.loc['control', 'conversion_rate_pct'],
summary.loc['treatment', 'conversion_rate_pct']]
# Standard error for error bars
se_control = np.sqrt(summary.loc['control', 'conversion_rate'] * (1 - summary.loc['control', 'conversion_rate']) / summary.loc['control', 'total_users']) * 100
se_treatment = np.sqrt(summary.loc['treatment', 'conversion_rate'] * (1 - summary.loc['treatment', 'conversion_rate']) / summary.loc['treatment', 'total_users']) * 100
bars = ax.bar(groups, rates, yerr=[se_control * 1.96, se_treatment * 1.96],
capsize=10, color=['#4A90D9', '#7BC67E'], edgecolor='black', linewidth=0.8)
ax.set_ylabel('Conversion Rate (%)', fontsize=12)
ax.set_title('Checkout Conversion: Control vs. Treatment', fontsize=14, fontweight='bold')
ax.set_ylim(0, max(rates) * 1.4)
# Annotate bars
for bar, rate in zip(bars, rates):
ax.text(bar.get_x() + bar.get_width() / 2, bar.get_height() + 0.3,
f'{rate:.2f}%', ha='center', va='bottom', fontsize=12, fontweight='bold')
# Add lift annotation
ax.annotate(f'Lift: {absolute_lift * 100:+.2f} pp',
xy=(1, rates[1]), xytext=(1.3, rates[1] + 1),
fontsize=11, color='green', fontweight='bold',
arrowprops=dict(arrowstyle='->', color='green'))
plt.tight_layout()
plt.show()The chart tells a compelling story: the new checkout design appears to convert better. But "appears to" is doing a lot of work in that sentence. In the next step, you'll run a formal statistical test to find out whether this difference could have happened by pure chance.
Test if the Difference is Real
Your bar chart shows the treatment group converting roughly 2 percentage points higher than control. That looks promising. But a visual difference is not evidence.
The core question this step answers: is that 2 percentage point lift real, or could it be random noise? Without a proper statistical test, you would be making a million-dollar product decision on what might be a coin flip. You will run a two-proportion z-test to find out.
In this step, get ready to:
- Run a hypothesis test to determine if the conversion lift is statistically significant.
- Calculate a 95% confidence interval for the true size of the difference.
- Translate the statistical results into plain language a product manager can act on.
Frame the hypotheses and run the z-test
Before running any test, you need to state exactly what you are testing. A formal hypothesis test compares a null hypothesis (no difference) against an alternative (there is a difference).
- Add a new markdown cell below your bar chart cell. Paste the following text into it:
## Hypothesis Test: Two-Proportion Z-Test
- **Null Hypothesis (H0):** The conversion rate of the treatment group equals the conversion rate of the control group (p_treatment - p_control = 0)
- **Alternative Hypothesis (H1):** The conversion rates are different (p_treatment - p_control != 0)
- **Significance Level:** alpha = 0.05 (two-sided)What does this markdown cell do?
This cell documents your decision criteria before you see the result. It states that you will reject the null hypothesis only if the p-value falls below 0.05. Writing this down first prevents the temptation of moving the goalposts after seeing the data.
Now you will run the actual test. The proportions_ztest function from statsmodels compares two conversion rates and tells you how likely the observed difference would be if the true difference were zero.
- Add a new code cell below your markdown cell and paste the following:
# Extract values for the test
n_control = int(summary.loc['control', 'total_users'])
n_treatment = int(summary.loc['treatment', 'total_users'])
conversions_control = int(summary.loc['control', 'conversions'])
conversions_treatment = int(summary.loc['treatment', 'conversions'])
# Run the two-proportion z-test
count = np.array([conversions_treatment, conversions_control])
nobs = np.array([n_treatment, n_control])
z_stat, p_value = proportions_ztest(count, nobs)
print("=" * 50)
print("TWO-PROPORTION Z-TEST RESULTS")
print("=" * 50)
print(f"\nZ-statistic: {z_stat:.4f}")
print(f"P-value: {p_value:.6f}")
print(f"Significance level (alpha): 0.05")
print(f"\nResult: {'REJECT H0' if p_value < 0.05 else 'FAIL TO REJECT H0'}")What does this code do?
First, it pulls the conversion counts and sample sizes from the summary DataFrame you built in Step 2.
Then proportions_ztest takes the number of successes (count) and the total observations (nobs) for each group. It returns a z-statistic (how many standard errors the difference is from zero) and a p-value (the probability of seeing this difference if the null were true).
- Run the cell by pressing Shift+Enter.
You should see a z-statistic around 4 and a p-value well below 0.05. The result line prints REJECT H0, meaning the difference is statistically significant.
Seeing a NameError for summary?
The summary variable comes from Cell 4 (Step 2). Make sure you have run all previous cells in order. In Colab, click Runtime then Run all to re-execute everything from the top.
help me debug my NameError in the z-test cell
Calculate the 95% confidence interval
The p-value tells you the difference is real. But how big is it? A confidence interval gives you a range that captures the true difference 95% of the time.
- Add a new code cell below Cell 7 and paste the following:
# Calculate 95% confidence interval for the difference
p_treatment = conversions_treatment / n_treatment
p_control = conversions_control / n_control
diff = p_treatment - p_control
# Standard error of the difference (unpooled for CI)
se_diff = np.sqrt(p_treatment * (1 - p_treatment) / n_treatment +
p_control * (1 - p_control) / n_control)
# 95% CI
ci_lower = diff - 1.96 * se_diff
ci_upper = diff + 1.96 * se_diff
print(f"Observed difference: {diff * 100:.2f} percentage points")
print(f"95% Confidence Interval: [{ci_lower * 100:.2f}, {ci_upper * 100:.2f}] percentage points")
print(f"\nInterpretation: We are 95% confident that the true difference in")
print(f"conversion rates lies between {ci_lower * 100:.2f} and {ci_upper * 100:.2f} percentage points.")What does this code do?
It computes each group's conversion rate, then calculates the standard error of the difference using the unpooled formula. The unpooled formula is appropriate for the confidence interval because you are estimating each group's variance separately.
The 1.96 multiplier corresponds to the 95% confidence level in a normal distribution. The resulting interval tells you the plausible range for the true lift.
- Run the cell by pressing Shift+Enter.
You should see the observed difference (around 2 percentage points) and a confidence interval whose lower bound is above zero. This confirms the lift is not just noise.
CI lower bound is negative?
Double check that conversions_treatment comes from the treatment group and conversions_control comes from the control group. If the order is reversed in Cell 7, all downstream calculations will flip.
help me figure out why my confidence interval looks wrong
Add a plain-language interpretation
Statistical output means nothing if stakeholders cannot understand it. This cell translates the numbers into a sentence a product manager can act on.
- Add a new code cell below Cell 8 and paste the following:
if p_value < 0.05:
print("INTERPRETATION:")
print(f"The new checkout design produced a statistically significant")
print(f"improvement in conversion rate (p = {p_value:.4f}).")
print(f"\nThe treatment group converted at {p_treatment * 100:.2f}% vs.")
print(f"the control group at {p_control * 100:.2f}%, a lift of")
print(f"{diff * 100:.2f} percentage points.")
print(f"\nHowever, statistical significance alone does not mean we should ship.")
print(f"We still need to validate the experiment and assess practical significance.")
else:
print("INTERPRETATION:")
print(f"The observed difference is NOT statistically significant (p = {p_value:.4f}).")
print(f"We cannot rule out that the difference is due to random chance.")Why does this matter?
This cell makes two critical points. First, it restates the result in business terms (conversion rates, not z-statistics). Second, it sets up the next steps by explicitly warning that statistical significance alone is not enough to ship.
In a real interview or stakeholder presentation, this is the slide that lands. Numbers without narrative get ignored.
- Run the cell by pressing Shift+Enter.
You should see a formatted interpretation confirming the lift is significant, along with the key reminder that you still need to validate the experiment before recommending a ship.
Before you move on, take a moment. Based on what you have seen so far, do you think you have enough evidence to recommend shipping the new checkout design? What might still be missing?
✔️ Awesome, I've got everything!
Great. Make sure all cells from Cell 6 through Cell 9 are saved in your notebook.
ⓧ I'd like to double check the full code
Scroll up through this step and confirm you have the following cells in order:
- Cell 6 (markdown): States H0, H1, and alpha = 0.05.
- Cell 7 (code): Extracts values from summary, runs proportions_ztest, prints z-statistic, p-value, and REJECT/FAIL TO REJECT result.
- Cell 8 (code): Computes p_treatment, p_control, diff, se_diff, ci_lower, and ci_upper. Prints the 95% confidence interval.
- Cell 9 (code): Prints a plain-language interpretation using an if/else on p_value < 0.05.
You now have hard statistical evidence that the conversion lift is real. But a significant p-value only proves the effect is non-zero. It does not prove the effect is big enough to matter. Next up, you will validate whether the experiment itself was run correctly and whether it had enough users to trust the results.
Validate the Experiment Integrity
Your z-test showed a statistically significant result. But a significant p-value means nothing if the experiment itself was broken.
This step is the one most junior analysts skip, and the one that impresses interviewers the most. You'll test whether users were actually split evenly (or if something went wrong with the randomisation) and whether the experiment had enough users to reliably detect the effect.
In this step, get ready to:
- Run a Sample Ratio Mismatch check to verify users were split correctly.
- Calculate statistical power to confirm the experiment could detect the effect.
- Summarize the validation findings in a markdown cell.
Check for Sample Ratio Mismatch
A Sample Ratio Mismatch (SRM) happens when the number of users in each group doesn't match what the experiment intended. If CartWave meant to split users 50/50 but one group ended up with significantly more users, something went wrong with the randomisation. That would invalidate every result downstream.
- Add a new code cell below your interpretation cell (Cell 9). Run the SRM check by pasting this code:
from scipy.stats import chisquare
observed = np.array([n_control, n_treatment])
expected = np.array([10000, 10000])
chi2_result = chisquare(observed, f_exp=expected)
print(f"Chi-square statistic: {chi2_result.statistic:.4f}")
print(f"P-value: {chi2_result.pvalue:.4f}")
print(f"Result: {'PASS - No SRM detected' if chi2_result.pvalue > 0.05 else 'FAIL - SRM detected!'}")What does this code do?
This is a chi-square goodness-of-fit test. It compares the observed user counts in each group against the expected counts (10,000 each for a 50/50 split).
If the p-value is above 0.05, there's no evidence the split was biased. If it's below 0.05, something may have gone wrong with user assignment and the entire experiment could be invalid.
Why chisquare instead of chi2_contingency?
You might see chi2_contingency used in some tutorials. That function tests independence in a 2D table. Here you're comparing observed counts to expected counts in a single dimension (one row), so chisquare (goodness-of-fit) is the correct test.
- Run the cell by pressing Shift+Enter.
You should see a chi-square statistic of 0.0000 and a p-value of 1.0000. The result prints PASS - No SRM detected. Since you generated exactly 10,000 users per group, the split is perfectly balanced.
Seeing a NameError?
If you see an error about n_control or n_treatment not being defined, scroll up and re-run Cell 7 (the z-test cell) first. Those variables are defined there.
Help me debug my SRM check cell
Calculate statistical power
Even if the split was fair, the experiment might not have had enough users to reliably detect the effect. Statistical power is the probability of detecting a real effect when one exists. Industry standard is 80% power or higher.
- Add another code cell below Cell 10. Run the power analysis by pasting this code:
baseline_rate = 0.12
effect_size = proportion_effectsize(baseline_rate + 0.02, baseline_rate)
power_analysis = NormalIndPower()
required_n = power_analysis.solve_power(effect_size=effect_size, alpha=0.05, power=0.80, alternative='two-sided')
achieved_power = power_analysis.power(effect_size=effect_size, nobs1=n_control, alpha=0.05, alternative='two-sided')
print(f"Effect size (Cohen's h): {effect_size:.4f}")
print(f"Required sample per group: {int(np.ceil(required_n)):,}")
print(f"Achieved power: {achieved_power:.1%}")
print(f"Result: {'PASS - Adequately powered' if achieved_power >= 0.80 else 'WARNING - Underpowered'}")What does this code do?
- proportion_effectsize calculates Cohen's h, a standardised measure of the difference between two proportions. It translates your raw percentage-point lift into a scale that power calculations can use.
- solve_power answers: "How many users per group would I need to detect this effect with 80% power?"
- power answers: "Given the sample we actually had, what power did we achieve?" If it's above 80%, the experiment was adequately sized.
Why NormalIndPower and not TTestIndPower?
NormalIndPower is designed for z-tests on two independent samples. Since you're comparing proportions (not continuous means), the z-test is standard, and NormalIndPower is the matching power class. TTestIndPower is for continuous outcomes like revenue or time-on-page.
- Run the cell by pressing Shift+Enter.
You should see that the required sample per group is well below 10,000 and the achieved power is above 80%. The result prints PASS - Adequately powered. This confirms the experiment had more than enough users to detect a 2 percentage point lift.
Seeing WARNING - Underpowered?
Check that baseline_rate is set to 0.12 and the effect in proportion_effectsize is baseline_rate + 0.02. Also confirm that n_control equals 10000 by running print(n_control) in a scratch cell.
Help me debug my power analysis
Summarize the validation findings
A professional notebook documents its conclusions clearly. This markdown cell ties up the validation section so anyone reading the notebook knows the experiment is trustworthy before they see the final recommendation.
- Add a new markdown cell below Cell 11. Paste in this content:
## Validation Summary
Both checks passed:
1. **SRM Check:** No evidence of assignment bias (users split as intended)
2. **Power Analysis:** Experiment had sufficient power to detect the target effect
The experiment results can be trusted. We can proceed to the business decision.Why document validation in its own cell?
In an interview or portfolio review, readers skim notebooks quickly. A clear validation summary signals that you don't just run tests blindly. You check whether the experiment itself was sound before drawing conclusions.
- Run the cell by pressing Shift+Enter. You should see the formatted markdown with the two numbered findings and the conclusion sentence.
Before you scroll further, pause and think: if the SRM check had FAILED, what would your recommendation be, regardless of the z-test results?
The answer is you would NOT ship. A broken experiment means you can't trust any of the numbers. That's exactly why you run validation before the business decision.
✔️ Awesome, I've got everything!
Great. Double check that all three cells (10, 11, and 12) have been run and show their outputs.
ⓧ I'd like to double check the full code
Here are the three cells you added in this step. Confirm each one matches:
Cell 10 (SRM check):
from scipy.stats import chisquare
observed = np.array([n_control, n_treatment])
expected = np.array([10000, 10000])
chi2_result = chisquare(observed, f_exp=expected)
print(f"Chi-square statistic: {chi2_result.statistic:.4f}")
print(f"P-value: {chi2_result.pvalue:.4f}")
print(f"Result: {'PASS - No SRM detected' if chi2_result.pvalue > 0.05 else 'FAIL - SRM detected!'}")Cell 11 (power analysis):
baseline_rate = 0.12
effect_size = proportion_effectsize(baseline_rate + 0.02, baseline_rate)
power_analysis = NormalIndPower()
required_n = power_analysis.solve_power(effect_size=effect_size, alpha=0.05, power=0.80, alternative='two-sided')
achieved_power = power_analysis.power(effect_size=effect_size, nobs1=n_control, alpha=0.05, alternative='two-sided')
print(f"Effect size (Cohen's h): {effect_size:.4f}")
print(f"Required sample per group: {int(np.ceil(required_n)):,}")
print(f"Achieved power: {achieved_power:.1%}")
print(f"Result: {'PASS - Adequately powered' if achieved_power >= 0.80 else 'WARNING - Underpowered'}")Cell 12 (markdown):
## Validation Summary
Both checks passed:
1. **SRM Check:** No evidence of assignment bias (users split as intended)
2. **Power Analysis:** Experiment had sufficient power to detect the target effect
The experiment results can be trusted. We can proceed to the business decision.Both integrity checks passed. The user split was balanced and the experiment had more than enough power. Now you can trust the statistical results from Step 3. Next up, you'll combine everything into a final ship/no-ship recommendation.
Make the Ship/No-Ship Decision
In the previous step, you confirmed the experiment is trustworthy. The randomization worked, and the test had enough statistical power to detect the effect you were looking for.
A p-value below 0.05 tells you the lift is non-zero. It does not tell you the lift is big enough to justify the engineering cost of shipping a new design. In this step, you'll combine statistical significance with practical significance to produce a defensible ship/no-ship recommendation.
In this step, get ready to:
- Define a practical significance threshold (Minimum Detectable Effect) the business cares about.
- Build a decision framework that requires both statistical and practical significance.
- Generate a formatted Decision Summary with a clear SHIP or DO NOT SHIP recommendation.
Define the practical significance threshold
Statistical significance answers: "Is the effect real?" Practical significance answers a different question: "Is the effect big enough to matter?"
In the real world, shipping a redesign has costs. Engineering time, QA, potential regressions. The product team needs the conversion lift to clear a minimum bar before those costs are worth it.
What is a Minimum Detectable Effect?
The Minimum Detectable Effect (MDE) is the smallest improvement that justifies the business cost of shipping. For CartWave, the product team decided a lift of at least +1 percentage point is needed to justify the engineering investment.
Your decision framework will check whether the lower bound of the confidence interval exceeds this threshold. If even the pessimistic end of your estimate clears the bar, you can ship with confidence.
Build the ship/no-ship decision cell
This cell brings together everything you've built so far. It checks three criteria: statistical significance, practical significance, and experiment validity.
- Add a new code cell below Cell 12 and paste this code:
# Practical significance threshold: +1 percentage point minimum
mde_threshold = 0.01
stat_sig = p_value < 0.05
practical_sig = ci_lower > mde_threshold
experiment_valid = chi2_result.pvalue > 0.05 and achieved_power >= 0.80
print("=" * 50)
print("SHIP / NO-SHIP DECISION")
print("=" * 50)
print(f"\n--- Decision Criteria ---")
print(f" 1. Statistical significance (p < 0.05): {'PASS' if stat_sig else 'FAIL'} (p = {p_value:.4f})")
print(f" 2. Practical significance (CI lower > {mde_threshold*100:.0f}pp): {'PASS' if practical_sig else 'FAIL'} (CI lower = {ci_lower*100:.2f}pp)")
print(f" 3. Experiment validity: {'PASS' if experiment_valid else 'FAIL'}")
if stat_sig and practical_sig and experiment_valid:
print("\n RECOMMENDATION: SHIP")
print(f" The new checkout design increases conversion by {diff*100:.2f}pp (95% CI: [{ci_lower*100:.2f}, {ci_upper*100:.2f}]pp).")
print(f" At 10,000 checkouts/day, this translates to ~{int(diff * 10000)} additional conversions per day.")
elif stat_sig and not practical_sig and experiment_valid:
print("\n RECOMMENDATION: DO NOT SHIP")
print(f" Statistically significant but effect too small to justify engineering investment.")
else:
print("\n RECOMMENDATION: DO NOT SHIP")
print(f" The experiment did not meet all required criteria.")What does this code do?
- mde_threshold = 0.01 sets the business bar at +1 percentage point. Any lift below this is not worth shipping.
- stat_sig checks if the p-value is below 0.05 (the effect is real).
- practical_sig checks if the lower bound of the CI exceeds the MDE threshold (the effect is big enough).
- experiment_valid confirms both the SRM check and power analysis passed.
- The if/elif/else block maps the three possible outcomes: all criteria pass (SHIP), statistically significant but too small (DO NOT SHIP), or any other failure (DO NOT SHIP).
Run the decision cell and see your recommendation
Before you run Cell 13, think about what you expect. Given the p-value you saw in Step 3 and the confidence interval, do you think the recommendation will be SHIP or DO NOT SHIP?
- Run Cell 13 by pressing the play button or hitting Shift+Enter.
You should see a formatted decision summary with all three criteria showing PASS, and a clear RECOMMENDATION: SHIP with the observed lift, confidence interval, and an estimate of additional conversions per day.
Why does the recommendation say SHIP?
The experiment passes all three gates. The p-value is below 0.05 (the lift is real). The lower bound of the 95% CI exceeds 1 percentage point (the lift is big enough to matter). The experiment had no SRM and sufficient power (the results can be trusted).
This is the framework product analytics teams actually use. A result that is statistically significant but practically tiny would get a DO NOT SHIP. That distinction is what separates junior analysts from senior ones in interviews.
Seeing an error or unexpected output?
- If you see a NameError for variables like p_value or ci_lower, make sure you ran all previous cells (Cells 1 through 12) first. Use **Runtime** then **Run all** to re-run everything in order.
- If the recommendation says DO NOT SHIP, double check that your data generation in Cell 3 used the same seed and parameters from the earlier steps.
Need more help?
✔️ Awesome, I've got everything!
Great. Make sure your notebook is saved. You now have a complete, interview-ready A/B test analysis.
ⓧ I'd like to double check the full code
Here is the full Cell 13 code. Compare it against what you have:
mde_threshold = 0.01
stat_sig = p_value < 0.05
practical_sig = ci_lower > mde_threshold
experiment_valid = chi2_result.pvalue > 0.05 and achieved_power >= 0.80
print("=" * 50)
print("SHIP / NO-SHIP DECISION")
print("=" * 50)
print(f"\n--- Decision Criteria ---")
print(f" 1. Statistical significance (p < 0.05): {'PASS' if stat_sig else 'FAIL'} (p = {p_value:.4f})")
print(f" 2. Practical significance (CI lower > {mde_threshold*100:.0f}pp): {'PASS' if practical_sig else 'FAIL'} (CI lower = {ci_lower*100:.2f}pp)")
print(f" 3. Experiment validity: {'PASS' if experiment_valid else 'FAIL'}")
if stat_sig and practical_sig and experiment_valid:
print("\n RECOMMENDATION: SHIP")
print(f" The new checkout design increases conversion by {diff*100:.2f}pp (95% CI: [{ci_lower*100:.2f}, {ci_upper*100:.2f}]pp).")
print(f" At 10,000 checkouts/day, this translates to ~{int(diff * 10000)} additional conversions per day.")
elif stat_sig and not practical_sig and experiment_valid:
print("\n RECOMMENDATION: DO NOT SHIP")
print(f" Statistically significant but effect too small to justify engineering investment.")
else:
print("\n RECOMMENDATION: DO NOT SHIP")
print(f" The experiment did not meet all required criteria.")Secret mission
Segment-Level Analysis
Your overall analysis says SHIP. But what if the new checkout design helps new users while hurting returning users? Segment analysis uncovers hidden patterns that change the recommendation entirely.
Clean Up Your Resources
Clean Up Your Resources
This project runs entirely in Google Colab, so there are no ongoing costs or paid resources to worry about. Your notebook is saved automatically to your Google Drive.
Resources you used:
- Google Colab notebook AB_Test_Analysis_CartWave (stored in your Google Drive).
Keep everything running
No action needed. Choose this if you want to keep your analysis notebook accessible for interviews or portfolio sharing.
- Your notebook stays in Google Drive indefinitely at no cost.
- You can share the notebook link directly with hiring managers or add it to your portfolio.
- To find it later, open Google Drive and look in the Colab Notebooks folder.
Pause - I'll come back to this later
Close the Colab tab to free up browser memory. Your notebook is already saved to Google Drive automatically.
- Close the Colab browser tab.
- To resume later, open Google Drive and double-click AB_Test_Analysis_CartWave.ipynb to reopen it in Colab.
- You may need to re-run cells from the top since Colab does not persist runtime state between sessions.
Delete - I don't want to use this again
Remove the notebook from your Google Drive entirely.
- In Google Drive, navigate to the Colab Notebooks folder.
- Right-click AB_Test_Analysis_CartWave.ipynb and select Move to trash.
- To permanently delete it, open Trash in the left sidebar, right-click the file, and select Delete forever.
Nice Work!
Nice Work!
You just built a complete, interview-ready A/B test analysis notebook from scratch. From raw experiment data to a defensible ship/no-ship recommendation, you now have a portfolio piece that demonstrates the exact workflow hiring managers ask about in product analytics interviews.
You've learned how to:
- Run a two-proportion z-test and interpret p-values and confidence intervals in plain language a product manager can understand.
- Validate experiment integrity using a Sample Ratio Mismatch check and statistical power analysis to confirm results can be trusted.
- Combine statistical significance with practical significance into a decision framework that produces a clear ship or no-ship recommendation backed by business context.
- Secret Mission: Performed segment-level analysis with Bonferroni correction to uncover hidden patterns across user segments and refine the recommendation.
Ready to quiz yourself?