Build a Bayesian A/B Testing Analyzer

Introduction

30 Second Summary

Every time you shop online, chances are you're part of an experiment where different button colors, headlines, or layouts are competing for your click. The classic statistical approach can declare a "winner," but it cannot tell you the one thing that matters most: how much you stand to lose if you pick wrong.

In this project, you will build a Bayesian A/B testing analyzer in a Google Colab notebook. You will simulate a realistic website experiment, compute full posterior distributions for each variant, and create a decision engine that tells a product team exactly what to do.

What You'll Build

Run the final cell and your notebook produces a polished 2x2 dashboard you can screenshot for LinkedIn, complete with posterior curves, a Monte Carlo comparison, and a clear "Ship B" or "Keep Testing" verdict.

By the end of this project, you'll have:

• A posterior distribution plot showing both variants' conversion rates with shaded credible intervals, so you can point to exactly where the uncertainty lives.
• A sequential updating figure that visually proves how Bayesian beliefs sharpen from wide and overlapping at 10 visitors to clearly separated at 1000.
• A Monte Carlo decision engine that computes the probability one variant beats another, calculates expected loss for each choice, and outputs a business-ready "Ship" or "Keep Testing" recommendation.
• Secret Mission: extend your analyzer to handle three or more variants and automatically identify the overall winner.

Are there any prerequisites?

You should be comfortable with Python (functions, loops, NumPy arrays) and have a basic understanding of Bayes' Theorem. You also need a Google account to access Google Colab.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Set Up Your Colab Notebook

Every analysis project starts with a clean, verified environment. If a library import fails or a version mismatch slips through, you will spend time debugging plumbing instead of writing analysis code.

Google Colab provides NumPy, SciPy, and Matplotlib pre-installed. In this step, you will create your notebook, confirm all imports work, and frame the business question your analysis will answer.

In this step, get ready to:

• Create and title your Colab notebook.
• Import and verify all required libraries.
• Define the business scenario in a markdown cell.

Create and title your notebook

Your notebook is where every cell of analysis code and explanation will live. Let's create a fresh one and give it a descriptive name.

• In Google Colab, click File in the top-left menu bar.
• Click New notebook.
• A new notebook opens in a separate tab with a default name like Untitled0.ipynb.
• Click the notebook title in the top-left corner (next to the Colab logo) and rename it to Bayesian A/B Testing Analyzer.
• Press Enter to confirm the new name.

Now add a title markdown cell at the top of the notebook.

• Click the + Text button in the toolbar to add a text (markdown) cell.
• Type the following markdown into the cell:

# Bayesian A/B Testing Analyzer

A complete Bayesian analysis pipeline that computes posterior distributions, quantifies decision risk, and recommends whether to ship a website variant.

• Press Shift+Enter to render the markdown cell.

You should see a large heading and a one-line description rendered in your notebook.

Import libraries and set the random seed

Before writing analysis code, you need to confirm that all three libraries load without errors. You will also lock a random seed so every run produces identical simulated data.

• Click the + Code button in the toolbar to add a new code cell below your markdown cell.
• Paste the following code into the cell:

import numpy as np
from scipy import stats
import matplotlib.pyplot as plt

np.random.seed(42)

# Print versions
print(np.__version__)
import scipy; print(scipy.__version__)
import matplotlib; print(matplotlib.__version__)

What does this code do?

• The first three lines import NumPy (for numerical arrays), SciPy's stats module (for probability distributions), and Matplotlib (for plotting).
• np.random.seed(42) locks the random number generator so your simulated experiment produces the same results every time you run the notebook.
• The print statements confirm each library loaded successfully by displaying its version number.

Before you run this cell, what do you expect to see printed in the output?

• Run the cell by pressing Shift+Enter (or clicking the play button on the left of the cell).

You should see three version numbers printed to the output, one per line. No errors, no warnings.

Seeing an ImportError?

All three libraries come pre-installed in Google Colab. If you see an import error, make sure you are running this notebook in Google Colab (not a local Jupyter installation).

If the cell hangs without producing output, check that the Colab runtime is connected by looking at the green checkmark next to RAM / Disk in the top-right corner.

Help me fix my Google Colab import error.

✔️ Awesome, I've got everything!

Great. Double check you see all three version numbers printed with no errors, then move on.

ⓧ I'd like to double check the full code

Here is the complete code cell you should have so far:

import numpy as np
from scipy import stats
import matplotlib.pyplot as plt

np.random.seed(42)

# Print versions
print(np.__version__)
import scipy; print(scipy.__version__)
import matplotlib; print(matplotlib.__version__)

Define the business scenario

Every analysis needs a clear problem statement. Adding it as a markdown cell keeps your notebook readable and self-documenting for anyone who opens it later.

• Click + Text to add a new markdown cell below your code cell.
• Type the following markdown into the cell:

## Business Scenario

An e-commerce site is testing two button variants (A = blue, B = green) to see which gets more clicks. We want to know the probability that B is better than A, and how much we stand to lose if we pick wrong.

• Press Shift+Enter to render the markdown.

You should see a formatted heading and a paragraph describing the two-variant experiment.

Why write the scenario in the notebook?

A portfolio notebook needs to tell a story. When a hiring manager opens your link, the markdown cells explain the business context without them needing to read your code first.

Your workspace is locked and loaded. Next up, you will simulate a real A/B experiment and compute your very first posterior distribution.

Simulate the Experiment and Compute Your First Posterior

Your notebook has the tools imported and the business scenario defined. Before building the full analyzer, you need real data to work with and a visible proof that Bayesian updating actually works.

In this step, you will simulate a realistic A/B experiment, apply the Beta-Binomial conjugate model to compute posterior distributions, and plot the result. By the end, you will see a distribution curve peaking near the true conversion rate, proving the math is alive.

In this step, get ready to:

• Write a simulation function that generates conversion data for two variants.
• Compute posterior distributions using the Beta-Binomial conjugate update.
• Plot both posteriors on a single chart and observe how they compare.

Simulate the A/B experiment

In a real A/B test, you would collect data from live users over days or weeks. For this project, you will simulate that data instantly using NumPy's binomial random number generator. Each "visitor" flips a weighted coin, converting or not based on the true rate you set.

• Add a new code cell below your business scenario markdown cell.
• Write the simulation function and run it by pasting this code:

# --- Step 2: Simulate and Compute Posteriors ---

def simulate_ab_experiment(n_visitors_a, n_visitors_b, true_rate_a, true_rate_b):
    """Simulate an A/B experiment with binomial outcomes."""
    conversions_a = np.random.binomial(n_visitors_a, true_rate_a)
    conversions_b = np.random.binomial(n_visitors_b, true_rate_b)
    return conversions_a, conversions_b

# Run simulation
n_visitors_a = 1000
n_visitors_b = 1000
true_rate_a = 0.12
true_rate_b = 0.145

conversions_a, conversions_b = simulate_ab_experiment(
    n_visitors_a, n_visitors_b, true_rate_a, true_rate_b
)
print(f"Variant A: {conversions_a}/{n_visitors_a} conversions")
print(f"Variant B: {conversions_b}/{n_visitors_b} conversions")

What does this code do?

The function simulate_ab_experiment uses np.random.binomial to simulate visitor behavior. Each call generates a count of conversions from a given number of visitors at a given probability.

• Variant A has 1000 visitors with a 12% true conversion rate (blue button).
• Variant B has 1000 visitors with a 14.5% true conversion rate (green button).
• Because you set np.random.seed(42) in Step 1, your exact numbers will be reproducible.

• Run the cell by pressing Shift+Enter.

You should see output like Variant A: 112/1000 conversions and Variant B: 153/1000 conversions. Your exact numbers will be close to the true rates, but not identical due to randomness.

Seeing a NameError?

• Make sure you ran the import cell from Step 1 first. Colab requires cells to be executed in order.
• Check that your import cell includes import numpy as np and from scipy import stats.

Help me debug a NameError in my Colab notebook.

Compute the posterior distributions

You now have raw conversion counts, but counts alone do not tell you the true conversion rate. The Beta-Binomial conjugate model lets you go from raw data to a full probability distribution over each variant's true rate.

What is a conjugate prior?

A conjugate prior is a special type of prior distribution that, when combined with observed data, produces a posterior in the same family. For binary outcomes (convert or not), the Beta distribution is conjugate to the Binomial likelihood.

The update rule is simple. Start with a Beta(alpha, beta) prior, observe s successes and f failures, and the posterior is Beta(alpha + s, beta + f). Starting with Beta(1, 1) means you begin with no assumptions about the conversion rate.

• Add a new code cell below your simulation cell.
• Compute both posteriors by pasting this code:

# Prior: Beta(1, 1) - uninformative (uniform)
prior_alpha = 1
prior_beta = 1

# Posterior parameters: Beta(prior + successes, prior + failures)
alpha_a = prior_alpha + conversions_a
beta_a = prior_beta + (n_visitors_a - conversions_a)
alpha_b = prior_alpha + conversions_b
beta_b = prior_beta + (n_visitors_b - conversions_b)

# Create frozen distribution objects
posterior_a = stats.beta(alpha_a, beta_a)
posterior_b = stats.beta(alpha_b, beta_b)

What does this code do?

• prior_alpha = 1 and prior_beta = 1 create a Beta(1, 1) prior. This is a flat line representing total ignorance about the conversion rate.
• alpha_a = prior_alpha + conversions_a adds observed successes to the prior. beta_a adds observed failures (visitors who did not convert).
• stats.beta(alpha_a, beta_a) creates a frozen distribution object you can query for PDF values, means, random samples, and more.

• Run the cell by pressing Shift+Enter. The cell finishes without error, confirming your parameters are valid.

Visualize both posteriors

Numbers alone are hard to interpret. Plotting the full posterior distributions shows exactly where each variant's conversion rate likely lives and how much the two distributions overlap.

• Add a new code cell and create the visualization by pasting this code:

# Plot both posterior distributions
x = np.linspace(0, 0.25, 1000)
plt.figure(figsize=(10, 6))
plt.plot(x, posterior_a.pdf(x), label=f"Variant A (mean={posterior_a.mean():.4f})", color="blue")
plt.plot(x, posterior_b.pdf(x), label=f"Variant B (mean={posterior_b.mean():.4f})", color="green")
plt.axvline(posterior_a.mean(), color="blue", linestyle="--", alpha=0.5)
plt.axvline(posterior_b.mean(), color="green", linestyle="--", alpha=0.5)
plt.xlabel("Conversion Rate")
plt.ylabel("Density")
plt.title("Posterior Distributions of Conversion Rates")
plt.legend()
plt.show()

What does this code do?

• np.linspace(0, 0.25, 1000) creates 1000 evenly spaced points on the x-axis, representing possible conversion rates from 0% to 25%.
• posterior_a.pdf(x) evaluates the probability density at each point. Taller peaks mean the model is more confident about that conversion rate.
• plt.axvline draws a vertical dashed line at the posterior mean, giving a quick visual anchor for each variant's best estimate.

Before you run this cell, think about what you expect. Variant A has a true rate of 0.12 and Variant B has 0.145. Where do you predict each curve will peak?

• Run the cell by pressing Shift+Enter.

A plot appears showing two bell-shaped curves. The blue curve (Variant A) peaks near ~0.12 and the green curve (Variant B) peaks near ~0.145. Both have vertical dashed lines marking their posterior means.

Notice how the two curves separate visually but still overlap somewhat. This overlap is exactly why you cannot just eyeball the answer. You need to quantify it with probability.

Plot not showing?

• Make sure you ran the posterior computation cell (the one defining posterior_a and posterior_b) before this plotting cell. Variables do not persist between sessions.
• If you restarted the runtime, run all cells from the top using Runtime > Run all.

Help me fix my matplotlib plot in Colab.

✔️ Awesome, I've got everything!

Great. Double check you have three code cells below your business scenario markdown: the simulation, the posterior computation, and the plot.

ⓧ I'd like to double check the full code

Scroll through your notebook and confirm these three code cells are present in order:

• Cell 1: The simulate_ab_experiment function definition, the simulation call with 1000 visitors per variant, and the print statements showing conversion counts.
• Cell 2: The prior_alpha = 1 and prior_beta = 1 definitions, the four posterior parameter calculations (alpha_a, beta_a, alpha_b, beta_b), and the two frozen distribution objects (posterior_a, posterior_b).
• Cell 3: The plotting code with np.linspace, both plt.plot calls, both plt.axvline calls, axis labels, the title "Posterior Distributions of Conversion Rates", and plt.show().

You now have posterior distributions that capture everything the data tells you about each variant's true conversion rate. The curves look separated, but that visual gap is not a number a product manager can act on. Next up, you will animate how those posteriors evolve as data arrives, showing the core power of Bayesian inference in action.

Watch Beliefs Update in Real Time

Your posterior plot from Step 2 shows the final answer, but it hides the journey.

Sequential updating is the core insight of Bayesian inference: as data arrives one visitor at a time, your uncertainty shrinks. In this step, you will produce a multi-panel visualization that shows exactly how beliefs sharpen with more data.

In this step, get ready to:

• Generate sequential streams of binary visitor outcomes for both variants.
• Plot posterior distributions at five checkpoints (10, 50, 200, 500, 1000 visitors) showing beliefs narrowing over time.

Generate sequential binary data streams

In Step 2, you worked with aggregated conversion counts (e.g. 120 out of 1000). To simulate watching beliefs evolve, you need a stream of individual visitor outcomes: a 1 for each conversion and a 0 for each bounce.

• Add a new code cell below your existing plot. Write a data generation function and create the streams by pasting this code:

# --- Step 3: Sequential Updating ---

# Generate a stream of individual visitor outcomes (1 = converted, 0 = didn't)
def generate_sequential_data(n_total, true_rate):
    return np.random.binomial(1, true_rate, size=n_total)

# Create full streams for both variants
stream_a = generate_sequential_data(1000, true_rate_a)
stream_b = generate_sequential_data(1000, true_rate_b)

# Define the sample sizes where we'll snapshot our posteriors
checkpoints = [10, 50, 200, 500, 1000]

What does this code do?

generate_sequential_data uses np.random.binomial(1, true_rate, size=n_total) to produce an array of 1s and 0s. Each element represents one visitor either converting (1) or not (0).

The checkpoints list defines where you will "pause" and examine the posterior. This lets you see how much data the model needs before the two variants clearly separate.

• Run the cell. You should see no errors or output. The data is generated and stored in memory, ready for plotting.

Visualize posteriors at each checkpoint

Now you will loop through each checkpoint, slice the stream up to that point, compute a fresh posterior, and plot it. The result is a five-panel figure that tells the sequential updating story at a glance.

• Add a new code cell below and paste this plotting code:

# Create a row of 5 subplots, one per checkpoint
fig, axes = plt.subplots(1, 5, figsize=(20, 4))
x = np.linspace(0, 0.3, 500)

for idx, n in enumerate(checkpoints):
    # Count conversions up to this checkpoint
    conv_a_n = stream_a[:n].sum()
    conv_b_n = stream_b[:n].sum()

    # Compute posterior at this sample size
    post_a_n = stats.beta(prior_alpha + conv_a_n, prior_beta + (n - conv_a_n))
    post_b_n = stats.beta(prior_alpha + conv_b_n, prior_beta + (n - conv_b_n))

    # Plot both posteriors on this subplot
    axes[idx].plot(x, post_a_n.pdf(x), color="blue", label="A")
    axes[idx].plot(x, post_b_n.pdf(x), color="green", label="B")
    axes[idx].set_title(f"n = {n}")
    axes[idx].set_xlim(0, 0.3)
    axes[idx].legend(fontsize=8)
    axes[idx].set_xlabel("Conversion Rate")

axes[0].set_ylabel("Density")
plt.suptitle("How Beliefs Sharpen With Data", fontsize=14)
plt.tight_layout()
plt.show()

What does this code do?

• stream_a[:n].sum() slices the first n visitors and counts how many converted. This simulates having only seen that many visitors so far.
• At each checkpoint, a fresh Beta posterior is computed using the same conjugate update rule from Step 2: Beta(prior_alpha + conversions, prior_beta + non-conversions).
• Each subplot uses the same x-axis range [0, 0.3] so you can visually compare how wide or narrow the curves are across panels.

Before you run this cell, think about what you expect to see. At n=10, will the two posteriors be clearly separated or heavily overlapping? What about at n=1000?

• Run the cell.

A 1x5 subplot figure appears titled "How Beliefs Sharpen With Data". At n=10 the posteriors are wide and flat, completely overlapping. By n=1000 the curves are tall, narrow, and clearly separated.

Why does this matter for your portfolio?

This visualization proves you understand the learning process, not just the formula. It shows that with only 10 visitors you have no idea which variant is better, but by 1000 the answer is clear.

This is the sequential updating story that makes Bayesian inference powerful. Frequentist tests give you one answer at the end. Bayesian updating shows you the whole journey of uncertainty collapsing.

Not seeing the figure?

• Make sure you ran the previous cell first (the one defining generate_sequential_data and the streams). The plotting cell depends on stream_a, stream_b, and checkpoints existing in memory.
• If you see a NameError mentioning prior_alpha or prior_beta, re-run your Step 2 cell that defines those variables.
• help me debug my sequential updating plot

✔️ Awesome, I've got everything!

Great. Double check that your notebook is saved in Google Drive.

ⓧ I'd like to double check the full code

Your Step 3 code should contain these two blocks in order. Scroll up and confirm each is present:

• Cell 1: The generate_sequential_data function, followed by stream_a and stream_b generation, and the checkpoints list.
• Cell 2: The plt.subplots(1, 5, figsize=(20, 4)) figure, the for idx, n in enumerate(checkpoints) loop computing posteriors and plotting them, and the plt.suptitle("How Beliefs Sharpen With Data", fontsize=14) call at the end.

Your sequential updating figure tells a compelling visual story. But a product manager cannot ship a feature based on overlapping curves. Next up, you will turn those posteriors into a concrete number: the probability that B beats A, and exactly how much you stand to lose if you pick wrong.

Quantify the Decision with Monte Carlo

Your sequential updating visualization proves that beliefs sharpen with data. But a product manager staring at two overlapping curves cannot ship a feature.

They need a number: "What is the probability B is better?" and "What do we lose if we pick wrong?" In this step, you will sample from both posteriors and compute concrete decision metrics that answer both questions.

In this step, get ready to:

• Draw Monte Carlo samples from both posteriors and compute P(B > A).
• Calculate expected loss and 95% credible intervals for the difference.
• Build a decision function that outputs a clear recommendation.

Draw Monte Carlo samples and compute P(B > A)

Monte Carlo simulation turns posterior distributions into a straightforward comparison. Instead of integrating complex probability expressions, you draw many random samples from each posterior and simply count how often B beats A.

• Add a new code cell in your notebook below the sequential updating plot. Paste this code to draw 100,000 samples from each posterior and compute the probability that B beats A:

# --- Step 4: Monte Carlo Decision ---

# Draw samples from each variant's posterior
n_samples = 100_000
samples_a = posterior_a.rvs(n_samples)
samples_b = posterior_b.rvs(n_samples)

# P(B > A): fraction of samples where B's rate exceeds A's
prob_b_beats_a = np.mean(samples_b > samples_a)
print(f"P(B > A) = {prob_b_beats_a:.4f} ({prob_b_beats_a*100:.1f}%)")

What does this code do?

posterior_a.rvs(n_samples) draws 100,000 random conversion rates from variant A's posterior distribution. Each sample is one plausible "true rate" given the data you observed.

np.mean(samples_b > samples_a) compares the 100,000 pairs element-wise. The fraction where B's sample exceeds A's sample IS the probability that B is better. No integrals needed.

• Run the cell.
• You should see output like P(B > A) = 0.9XXX (9X.X%) with a value above 90%.

Seeing P(B > A) near 0.5?

Make sure you ran the Step 2 cells first so that posterior_a and posterior_b are defined with actual experiment data. If you restarted the runtime, run all cells from the top.

Help me debug why my P(B > A) value seems wrong.

Compute expected loss and credible intervals

Knowing B is probably better is not enough. A product team needs to know how much they stand to lose if they pick wrong. Expected loss answers that directly.

• Add a new code cell below your P(B > A) output. Paste this code to compute expected loss for each decision and the 95% credible interval:

# Expected loss: how much conversion rate you lose on average if you pick wrong
loss_a = np.mean(np.maximum(samples_b - samples_a, 0))
loss_b = np.mean(np.maximum(samples_a - samples_b, 0))
print(f"Expected loss (choosing A): {loss_a:.6f}")
print(f"Expected loss (choosing B): {loss_b:.6f}")

# 95% credible interval for the difference between B and A
diff_samples = samples_b - samples_a
ci_low, ci_high = np.percentile(diff_samples, [2.5, 97.5])
print(f"95% CI for (B - A): [{ci_low:.4f}, {ci_high:.4f}]")

What does expected loss mean?

loss_a measures the average conversion rate you sacrifice if you ship A but B was actually better. The np.maximum(..., 0) ensures you only count cases where you actually lose out.

The 95% credible interval tells you the range where the true difference between B and A almost certainly lives. If the interval does not contain zero, B is meaningfully better.

• Run the cell.
• You should see two expected loss values printed (loss for choosing A will be larger than loss for choosing B) and a credible interval with both bounds above zero.

Seeing identical loss values?

Check that samples_a and samples_b come from different posteriors. If you accidentally wrote posterior_a.rvs(n_samples) for both, they will be drawn from the same distribution.

Help me debug why my expected loss values look wrong.

Build the decision function

The final piece is a function that takes expected loss values and returns a business-ready recommendation. If the loss of choosing a variant is below a threshold (0.001, meaning less than 0.1% of conversion rate), it is safe to ship.

• Add a new code cell. Paste this code to define the decision function and print the final recommendation:

# Decision function: recommend based on expected loss threshold
def recommend(loss_a, loss_b, threshold=0.001):
    if loss_b < threshold:
        return "Ship B"
    elif loss_a < threshold:
        return "Ship A"
    else:
        return "Keep Testing"

recommendation = recommend(loss_a, loss_b)
print(f"\nRecommendation: {recommendation}")
print(f"\nInterpretation: If we choose B and we are wrong, we lose at most "
      f"{loss_b*100:.4f} percentage points of conversion on average.")

What does this code do?

The recommend() function checks whether the expected loss of the chosen variant is below 0.001 (0.1% of conversion rate). If choosing B costs you almost nothing even when you are wrong, the function returns "Ship B".

The threshold of 0.001 is a common industry standard for Bayesian A/B test decisions. It means you are willing to tolerate at most a 0.1 percentage point loss in conversion rate for making the wrong call.

Before you run this cell, think about what the recommendation will be. Given that P(B > A) was above 90% and the loss for choosing B was small, do you expect "Ship B", "Ship A", or "Keep Testing"?

• Run the cell.

You should see Recommendation: Ship B printed, along with the interpretation line showing how little you stand to lose.

Why expected loss instead of a p-value?

A frequentist p-value tells you whether an effect exists. Expected loss tells you how much it costs to be wrong. Product teams care about the cost of a bad decision, not whether the difference is "statistically significant" in an abstract sense.

This is the key advantage of the Bayesian approach. You can say "ship it" with a concrete dollar-amount justification, not just a threshold that nobody outside statistics understands.

Seeing "Keep Testing" instead of "Ship B"?

This can happen if the random seed was not set at the top of your notebook. Make sure the first code cell still has np.random.seed(42) and you ran all cells in order from the top. If you ran cells out of order, use **Runtime** then **Run all** to reset.

Help me understand why my recommendation says Keep Testing.

✔️ Awesome, I've got everything!

Great. Make sure all three code cells from this step are saved in your notebook. Your output should show P(B > A), both expected loss values, the 95% credible interval, and the "Ship B" recommendation.

ⓧ I'd like to double check the full code

Scroll through your notebook and confirm these code blocks are present in order:

• The imports and seed cell (import numpy as np, from scipy import stats, import matplotlib.pyplot as plt, np.random.seed(42)).
• The simulation cell with simulate_ab_experiment() and both posterior distributions plotted.
• The sequential updating cell with generate_sequential_data() and the 1x5 subplot figure.
• The Monte Carlo samples cell with n_samples = 100_000 and prob_b_beats_a printed.
• The expected loss cell with loss_a, loss_b, diff_samples, and the 95% CI printed.
• The decision cell with recommend() defined and Recommendation: Ship B in the output.

You now have hard numbers backing up what the curves suggested. Next up, you will combine everything into a single polished dashboard visualization that tells the full story at a glance.

Build the Final Dashboard Visualization

Your Monte Carlo analysis from Step 4 gives you the numbers: P(B > A), expected losses, and a clear recommendation. But those numbers live scattered across print statements in separate cells.

A portfolio piece needs a single, polished figure that tells the full story at a glance. In this step, you will build a 2x2 dashboard that combines all your analysis into one visualization you can screenshot for LinkedIn and link from your resume.

In this step, get ready to:

• Build a 2x2 subplot figure with posterior curves, sequential means, a difference histogram, and a summary panel.
• Annotate each panel with key metrics and labels for a stakeholder-ready presentation.
• Save the final dashboard as a PNG file.

Create the figure and fill the top-left panel

The top-left panel shows the core result: both posterior distributions with their 95% credible intervals shaded. This is the visual proof of where uncertainty lives for each variant.

• Add a new code cell at the bottom of your notebook.
• Create the 2x2 figure and fill the top-left panel by pasting this code:

# --- Step 5: Dashboard Visualization ---

fig, axes = plt.subplots(2, 2, figsize=(12, 10))

# Top-left: Posterior distributions with credible intervals
ax = axes[0, 0]
x = np.linspace(0, 0.25, 1000)
ax.plot(x, posterior_a.pdf(x), color="blue", label="Variant A")
ax.plot(x, posterior_b.pdf(x), color="green", label="Variant B")
ci_a = posterior_a.ppf([0.025, 0.975])
ci_b = posterior_b.ppf([0.025, 0.975])
ax.fill_between(x, 0, posterior_a.pdf(x), where=(x >= ci_a[0]) & (x <= ci_a[1]), alpha=0.2, color="blue")
ax.fill_between(x, 0, posterior_b.pdf(x), where=(x >= ci_b[0]) & (x <= ci_b[1]), alpha=0.2, color="green")
ax.set_title("Posterior Distributions")
ax.set_xlabel("Conversion Rate")
ax.set_ylabel("Density")
ax.legend()

What does this code do?

The plt.subplots(2, 2, figsize=(12, 10)) creates a 2-row, 2-column grid of axes you will fill one panel at a time.

The .ppf([0.025, 0.975]) method computes the 95% credible interval bounds from the posterior. The fill_between call shades only the region between those bounds, making the uncertainty band visually distinct.

• Run the cell.
• You should see a 2x2 grid with the top-left panel showing two bell-shaped curves (blue for A, green for B) with lightly shaded credible intervals. The other three panels are empty.

Only seeing an empty figure?

Make sure this new cell comes after all your Step 4 code. It depends on posterior_a and posterior_b existing from an earlier cell. Re-run all cells from the top if needed.

Help me debug my dashboard figure

Fill the top-right and bottom-left panels

The top-right panel shows how posterior means converge over time, reusing the sequential data from Step 3. The bottom-left panel shows the Monte Carlo difference histogram from Step 4 as a distribution.

• Add the top-right panel code directly below your existing code in the same cell:

# Top-right: Posterior means over time
ax = axes[0, 1]
means_a = []
means_b = []
ns = list(range(10, 1001, 10))
for n in ns:
    conv_a_n = stream_a[:n].sum()
    conv_b_n = stream_b[:n].sum()
    means_a.append((prior_alpha + conv_a_n) / (prior_alpha + prior_beta + n))
    means_b.append((prior_alpha + conv_b_n) / (prior_alpha + prior_beta + n))
ax.plot(ns, means_a, color="blue", label="Variant A")
ax.plot(ns, means_b, color="green", label="Variant B")
ax.axhline(true_rate_a, color="blue", linestyle=":", alpha=0.5)
ax.axhline(true_rate_b, color="green", linestyle=":", alpha=0.5)
ax.set_title("Posterior Means Over Time")
ax.set_xlabel("Number of Visitors")
ax.set_ylabel("Estimated Conversion Rate")
ax.legend()

What does this code do?

This loops through every 10th visitor count from 10 to 1000, computing the posterior mean at each point. The posterior mean formula (prior_alpha + conversions) / (prior_alpha + prior_beta + n) is the Beta distribution mean without needing to create a full distribution object each time.

The dotted horizontal reference lines (axhline) show the true conversion rates so you can see how the estimates converge toward ground truth.

• Run the cell. You should now see two panels filled: the posteriors on the left, and converging line charts on the right.
• Now add the bottom-left panel code below in the same cell:

# Bottom-left: Difference histogram
ax = axes[1, 0]
ax.hist(diff_samples, bins=100, density=True, alpha=0.7, color="purple")
ax.axvline(0, color="red", linestyle="--", linewidth=2)
ax.fill_between(np.linspace(0, diff_samples.max(), 100), 0, 0.1, alpha=0.1, color="green", label=f"P(B>A) = {prob_b_beats_a:.1%}")
ax.set_title("Distribution of (B - A)")
ax.set_xlabel("Difference in Conversion Rate")
ax.set_ylabel("Density")
ax.annotate(f"P(B > A) = {prob_b_beats_a:.1%}", xy=(0.02, ax.get_ylim()[1] * 0.8 if ax.get_ylim()[1] > 0 else 10), fontsize=11, fontweight="bold")

What does this code do?

This histograms the 100,000 Monte Carlo samples of (B minus A). The red dashed line at zero marks the boundary: everything to the right is evidence that B outperforms A.

The annotate call stamps the exact P(B > A) percentage directly onto the plot so a viewer does not need to read a separate table.

• Run the cell. Three of the four panels should now be filled. The bottom-right is still empty.

Complete the dashboard and save it

The final panel is a text summary box showing all key metrics in one place. This is what a product manager reads without needing to interpret the charts.

• Add the bottom-right summary panel, the figure title, and the save command at the bottom of the same cell:

# Bottom-right: Summary metrics
ax = axes[1, 1]
ax.axis("off")
summary_text = (
    f"BAYESIAN A/B TEST SUMMARY\n"
    f"{'='*35}\n\n"
    f"Variant A: {conversions_a}/{n_visitors_a} ({conversions_a/n_visitors_a:.1%})\n"
    f"Variant B: {conversions_b}/{n_visitors_b} ({conversions_b/n_visitors_b:.1%})\n\n"
    f"P(B > A): {prob_b_beats_a:.1%}\n"
    f"Expected Loss (choose A): {loss_a:.5f}\n"
    f"Expected Loss (choose B): {loss_b:.5f}\n"
    f"95% CI for (B-A): [{ci_low:.4f}, {ci_high:.4f}]\n\n"
    f"Recommendation: {recommendation}\n"
    f"{'='*35}"
)
ax.text(0.1, 0.5, summary_text, transform=ax.transAxes, fontsize=12, verticalalignment="center", fontfamily="monospace", bbox=dict(boxstyle="round", facecolor="lightyellow", alpha=0.8))

plt.suptitle("Bayesian A/B Test Results: Button Color Experiment", fontsize=14)
plt.tight_layout()
plt.savefig("bayesian_ab_results.png", dpi=150, bbox_inches="tight")
plt.show()

What does this code do?

The ax.axis("off") hides the axis lines and ticks so this panel acts as a clean text box. The f-string builds a formatted summary pulling in every metric you computed in Step 4.

The plt.savefig("bayesian_ab_results.png", dpi=150, bbox_inches="tight") saves a high-resolution copy of the figure to your Colab file system. This is the image you can download and use in a LinkedIn post or portfolio page.

Before you run this final cell, what do you expect to see in all four panels?

• Run the cell.

A polished 2x2 figure appears. Top-left shows both posterior curves with shaded credible intervals. Top-right shows the posterior means converging toward the true rates. Bottom-left shows the purple difference histogram with a P(B > A) annotation. Bottom-right displays the formatted summary box with all key metrics and your recommendation.

Where is the saved file?

In Colab, click the folder icon in the left sidebar to open the file browser. You will see bayesian_ab_results.png listed there. Right-click it and select Download to save it to your computer.

Figure looks cramped or overlapping?

Make sure plt.tight_layout() appears after plt.suptitle() in your code. The tight layout adjusts spacing to prevent overlap between subplots and the super title.

Help me fix my dashboard layout

✔️ Awesome, I've got everything!

Great. Double check you have saved your notebook.

ⓧ I'd like to double check the full code

Your dashboard cell should contain all four panel blocks you built above, in order. Scroll up and confirm each one is present:

• The fig, axes = plt.subplots(2, 2, figsize=(12, 10)) line and the top-left panel (posteriors with fill_between).
• The top-right panel (means_a and means_b line charts with axhline reference lines).
• The bottom-left panel (diff_samples histogram with annotated P(B > A)).
• The bottom-right panel (summary_text with ax.text()).
• The final four lines: plt.suptitle(), plt.tight_layout(), plt.savefig(), and plt.show().

Your Bayesian A/B testing analyzer is complete. You now have a polished, publication-ready notebook that computes posteriors, quantifies decisions, and presents everything in a single dashboard figure. Next up, the Secret Mission challenges you to extend this to three or more variants.

Secret mission

Handle Multiple Variants (A/B/C Testing)

Real experiments often test more than two variants. Can you extend your analyzer to handle three variants at once, compute the probability each is the overall best, and recommend a winner using generalized expected loss?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely in Google Colab, so there are no ongoing costs or cloud resources to worry about. Decide whether to keep your notebook accessible, download a local copy, or delete it entirely.

Resources you used:

• Google Colab notebook ("Bayesian A/B Testing Analyzer") stored in Google Drive.
• bayesian_ab_results.png file saved in the Colab session runtime (ephemeral, deleted when session ends).

Keep everything running

No action needed. Choose this if you want your notebook ready to share or revisit anytime.

• Your notebook persists in Google Drive indefinitely at no cost.
• To share it as a portfolio piece, click Share in the top-right corner of Colab and set access to Anyone with the link.
• The bayesian_ab_results.png file lives in Colab's ephemeral runtime storage. Download it before your session ends if you want to keep it. Click the folder icon in the left sidebar, right-click the file, and select Download.

Pause - I'll come back to this later

Download a local copy of your notebook so you have it even outside Google Drive.

• In Colab, click File in the top menu bar.
• Select Download, then Download .ipynb.
• Save the .ipynb file somewhere you can find it later.
• Download bayesian_ab_results.png from the left sidebar file browser before your session disconnects.

Delete - I don't want to use this again

Remove your notebook from Google Drive entirely.

• Open Google Drive at drive.google.com.
• Navigate to the Colab Notebooks folder.
• Right-click Bayesian A/B Testing Analyzer.ipynb and select Move to trash.
• Empty your trash to permanently delete the file.
• The bayesian_ab_results.png file is stored in Colab's ephemeral runtime. It is automatically deleted when your session ends. No action needed.

Nice Work!

Nice Work!

You just built a complete Bayesian A/B testing pipeline from scratch. Your notebook goes beyond p-values to give product teams actionable, probabilistic decisions backed by full posterior distributions and Monte Carlo evidence.

You've learned how to:

• Compute posterior distributions using the Beta-Binomial conjugate model and visualize how beliefs sharpen as data arrives through sequential updating.
• Quantify decisions with Monte Carlo simulation to produce P(B > A), expected loss, and 95% credible intervals that answer real business questions.
• Build a polished 2x2 dashboard visualization and a reusable recommend() function that outputs business-ready "Ship" or "Keep Testing" verdicts based on expected loss thresholds.
• Secret Mission: Extended the analyzer to handle three or more variants, computing probability of being best and generalized expected loss for each.

Ready to quiz yourself?