Targeted Rollout with Uplift Modeling

Build and validate an uplift model for a targeted checkout rollout policy.

Introduction

30 Second Summary

A checkout update can help first-time shoppers while frustrating loyal customers. One overall average can hide that split.

In this project, you will build a browser analysis in Google Colab that turns a randomized experiment into a targeted rollout policy. You will move from an A/B test summary to uplift modeling that earns trust on unseen data.

What You'll Build

Picture running one browser notebook that turns checkout results into a clear recommendation for exactly which customers should receive the new experience.

By the end of this project, you'll have:

  • An executive-ready A/B test summary that shows absolute lift plus relative lift. Its decision table reports a 95% confidence interval, a p-value, and an 80% power minimum detectable effect.
  • A segment analysis that makes the new-customer benefit visible. It also shows the harm to loyal customers that the overall average conceals.
  • A held-out CausalML policy that ranks customers by predicted incremental conversion. A Qini curve tests the ranking against random targeting. A policy table compares no rollout, full rollout, segment targeting, and model targeting.
  • An optional Secret Mission: Replace the zero-uplift targeting rule with a profit-aware break-even threshold that accounts for conversion value plus treatment cost.

Are there any prerequisites?

You should be comfortable writing Python plus interpreting A/B tests. The project teaches the CausalML workflow from the start.

Before We Start

Before the analysis begins, lock in the checkout decision your notebook must support. Your goal is to judge whether the treatment deserves a broad rollout or a targeted rollout when an overall A/B test result may conceal harm within customer segments.

Set Up the Colab Analysis Lab

Your checkout rollout decision depends on a randomized experiment that anyone can reproduce. A consistent browser lab keeps local Windows build differences out of the analysis.

Google Colab supplies a hosted runtime for your Python analysis. You will create the notebook before pinning every package to a tested version.

In this step, get ready to:
  • Make Google Chrome available on Windows.
  • Create targeted_rollout.ipynb on the default Python 3.12 runtime.
  • Verify the six pinned analysis packages by printing their exact versions.
Check Google Chrome

Chrome gives you a consistent route into the browser notebook. Windows search reveals whether it is already available.

  • Press the Windows key to open Windows search.
  • Type Google Chrome into the search field.
  • Press Enter to launch the browser.

✔️ Google Chrome opens

Chrome opens to a browser window. Your browser is ready for the Colab workflow.

ⓧ Google Chrome is unavailable

Chrome needs to be installed before you create the notebook. The official installer handles the Windows setup.

  • Visit the official Google Chrome download page.
  • Download the Windows installation file.
  • Run the downloaded installation file.
  • Select Run when Windows prompts you.
  • Select Yes to approve the Windows prompt.
  • Return to Windows search after installation.
  • Type Google Chrome into the search field.
  • Press Enter to launch the browser.

That first hurdle is clear. Chrome can now carry the full analysis without local build setup.

Chrome Does Not Open?

Confirm that the installer completed before retrying Windows search. Approve any pending Windows installation prompt.

Ask for targeted help with troubleshooting the Chrome installation.

Create the Colab notebook

A named notebook keeps every result in one shareable analysis. Colab stores the notebook in Google Drive while the hosted runtime performs the computation.

  • Visit Google Colab in Chrome.
  • Sign in with your Google account.
  • Create a new notebook from the Colab welcome screen.
  • Select the current notebook name in the top-left corner.
  • Type targeted_rollout.ipynb as the notebook name.
  • Press Enter to apply the name.
  • Keep the default hosted runtime unchanged.

You should see an empty notebook named targeted_rollout.ipynb. The default runtime provides Python 3.12 for the package installation.

Why Google Colab?

The hosted runtime removes local Windows compiler differences from the project. CausalML requires Python >=3.11.

The current default runtime uses Python 3.12. This gives the pinned packages a compatible browser environment.

Good progress. Your analysis now has a named notebook with a compatible runtime.

Cannot Create the Notebook?

Confirm that Chrome shows the Google account you intend to use. Reload Colab after completing any pending sign-in prompt.

Ask for help with creating the Colab notebook.

Install and verify the packages

Version pins make the notebook reproducible after a runtime reset. The installation cell places every analysis dependency into the current hosted runtime.

  • Select the first empty code cell in targeted_rollout.ipynb.
  • Paste the pinned installation command below into the cell:
!pip install -q causalml==0.18.0 statsmodels==0.15.0 scikit-learn==1.9.1 pandas==3.0.6 numpy==2.5.4 matplotlib==3.11.2

What Does This Installation Add?

  • CausalML supplies the uplift forest used for customer targeting.
  • Statsmodels supplies A/B inference plus statistical power calculations.
  • Scikit-learn supplies the held-out train and test split.
  • Pandas supplies the analysis tables.
  • NumPy supplies reproducible simulation operations.
  • Matplotlib supplies the experiment charts.
  • Run the installation cell by selecting its play control.
  • Wait for the cell to finish before adding another cell.

The cell's running indicator should stop without an installation failure. The pinned packages now belong to this temporary runtime.

Package Installation Failed?

Confirm that the Colab runtime is connected. Check your internet connection before running the installation cell again.

Ask for help with diagnosing the package installation.

Package metadata gives the runtime an exact way to report what was installed. This check catches missing packages plus silent version drift.

  • Add a new code cell beneath the installation cell.
  • Paste the following version-check code into the new cell:
# Cell 1: Verify the environment after running the pinned install command.
from importlib.metadata import version

PACKAGES = [
    "causalml",
    "statsmodels",
    "scikit-learn",
    "pandas",
    "numpy",
    "matplotlib",
]

for package in PACKAGES:
    print(package, version(package))

What Does This Code Do?

  • The PACKAGES list holds the six distribution names checked by the notebook.
  • The version() function reads each installed distribution's metadata.
  • The loop prints one package name plus its installed version on each line.

Before you run this check, do you expect every installed version to match its pin?

  • Run the version-check cell by selecting its play control.

✔️ I see all six pinned versions

The cell should print these six package and version pairs:

  • causalml 0.18.0.
  • statsmodels 0.15.0.
  • scikit-learn 1.9.1.
  • pandas 3.0.6.
  • numpy 2.5.4.
  • matplotlib 3.11.2.

ⓧ I see a different version

A different version means the runtime did not finish applying every pin. Rerunning the installation cell restores the tested package set.

  • Return to the pinned installation cell above.
  • Run the installation cell again.
  • Rerun the version-check cell after installation finishes.

You should now see all six pinned versions listed in the success tab.

ⓧ A package is missing

A missing package means the runtime does not contain the complete analysis environment. The pinned installation cell installs the full set together.

  • Confirm that the Colab runtime is connected.
  • Return to the pinned installation cell above.
  • Run the installation cell again.
  • Rerun the version-check cell after installation finishes.

You should now see one printed line for every package in PACKAGES.

Versions Still Do Not Match?

Confirm that the installation cell completed in the same connected runtime as the version check. A recycled runtime loses packages from the earlier session.

Ask for help with resolving the version mismatch.

✔️ Awesome, I've got everything!

Great. Keep the installation cell plus the version-check cell in targeted_rollout.ipynb.

ⓧ I'd like to double check the full code

# Cell 1: Verify the environment after running the pinned install command.
from importlib.metadata import version

PACKAGES = [
    "causalml",
    "statsmodels",
    "scikit-learn",
    "pandas",
    "numpy",
    "matplotlib",
]

for package in PACKAGES:
    print(package, version(package))

That's the analysis lab ready. Your notebook can now reproduce the tested package environment whenever you reconnect.

Your browser notebook now has a verified causal analysis environment. Next, you will generate the randomized checkout experiment plus its first conversion chart.

Build a Checkout Experiment

Your pinned Google Colab lab is ready for analysis. A rollout recommendation now needs a concrete randomized experiment behind it.

The notebook will generate 30,000 synthetic checkout records. The first chart reveals the treatment and control conversion rates.

In this step, get ready to:
  • Define the shared analysis foundation.
  • Generate a reproducible randomized checkout dataset.
  • Check the experiment's conversion signal against feature balance.
Define the analysis foundation

A fixed seed makes every notebook run generate the same experiment. The FEATURES list limits the later model to observable customer attributes.

  • Add a new code cell below the existing version-check cell in targeted_rollout.ipynb.
  • Paste the analysis imports and project constants into the new cell using this code:
# Cell 2: Imports and project constants.
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from causalml.inference.tree import UpliftRandomForestClassifier
from causalml.metrics import plot_qini, qini_score
from sklearn.model_selection import train_test_split
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import (
    confint_proportions_2indep,
    proportion_effectsize,
    test_proportions_2indep,
)

SEED = 20261013
N_CUSTOMERS = 30_000
FEATURES = [
    "new_customer",
    "mobile",
    "prior_purchases",
    "discount_sensitivity",
    "region_west",
]

What Does This Cell Define?

  • The imports prepare the notebook for tables, charts, inference, model training, and uplift evaluation.
  • SEED fixes the random number sequence so repeated runs reproduce the same records.
  • N_CUSTOMERS sets the experiment size to 30,000 customers.
  • FEATURES excludes true_uplift. This prevents hidden ground truth from leaking into the later model.
  • Run the imports and constants cell.
  • Confirm the cell finishes without an exception.
  • Allow targeted_rollout.ipynb to save to Google Drive.

Your analysis foundation is in place. Every later cell now uses the same sample size, random seed, and approved feature list.

Imports Did Not Finish?

Confirm that the pinned installation cell from the previous step completed in the current runtime. A recycled runtime needs the package installation cell to run again.

Ask for targeted help with diagnosing the failed notebook imports.

Generate the randomized experiment

The simulation first creates customer characteristics. It assigns each record independently to treatment or control.

  • Add a new code cell beneath the imports and constants cell.
  • Paste the customer feature generation logic into the new cell using this code:
# Cell 3: Generate a randomized checkout experiment.
rng = np.random.default_rng(SEED)
new_customer = rng.binomial(1, 0.40, size=N_CUSTOMERS)
mobile = rng.binomial(1, 0.60, size=N_CUSTOMERS)
prior_purchases = np.where(
    new_customer == 1,
    rng.poisson(0.6, size=N_CUSTOMERS),
    rng.poisson(6.0, size=N_CUSTOMERS),
)
discount_sensitivity = rng.beta(2.0, 3.0, size=N_CUSTOMERS)
region_west = rng.binomial(1, 0.30, size=N_CUSTOMERS)
treated = rng.binomial(1, 0.50, size=N_CUSTOMERS)
treatment = np.where(treated == 1, "treatment", "control")

How Are Customers Assigned?

  • The random generator creates customer attributes from fixed probability distributions.
  • prior_purchases follows different distributions for new customers and loyal customers.
  • treated gives every customer a 50% chance of receiving the checkout treatment.
  • treatment converts the binary assignment into readable group labels.
  • Run the experiment cell.
  • Confirm the cell completes without an exception.
  • Allow the updated targeted_rollout.ipynb notebook to save.

Customer Generation Failed?

Confirm that you ran the imports and constants cell first. Check that each feature line uses N_CUSTOMERS for its output size.

Ask for help with debugging the synthetic customer features.

Conversion depends on each customer's baseline probability. Treatment changes that probability through a hidden customer-level effect.

  • Find the line that defines treatment in the experiment cell.
  • Add the conversion outcome logic directly below that line using this code:
baseline_logit = (
    -2.0
    - 0.35 * new_customer
    + 0.18 * mobile
    + 0.05 * np.minimum(prior_purchases, 10)
    - 0.25 * discount_sensitivity
    + 0.10 * region_west
)
baseline_probability = 1 / (1 + np.exp(-baseline_logit))
true_uplift = np.where(new_customer == 1, 0.06, -0.02)
true_uplift = true_uplift + 0.01 * mobile * new_customer
conversion_probability = np.clip(
    baseline_probability + treated * true_uplift,
    0.01,
    0.95,
)
conversion = rng.binomial(1, conversion_probability, size=N_CUSTOMERS)

How Is Conversion Generated?

  • baseline_logit combines customer characteristics into an untreated conversion tendency.
  • true_uplift gives new customers a positive treatment effect. It gives loyal customers a negative treatment effect.
  • conversion_probability adds the hidden effect only when a customer receives treatment.
  • conversion turns each probability into a binary observed outcome.
  • Run the updated experiment cell.
  • Confirm the cell completes without an exception.
  • Allow the updated targeted_rollout.ipynb notebook to save.

Conversion Generation Failed?

Check the indentation inside baseline_logit and conversion_probability. Confirm that the earlier part of the cell defines every referenced customer feature.

Ask for help with debugging the conversion simulation.

A single table now needs to hold the assignment, observed outcome, customer features, and hidden learning signal. The printed shape provides the first concrete reproducibility check.

  • Find the line that defines conversion in the experiment cell.
  • Add the experiment table and preview directly below that line using this code:
df = pd.DataFrame(
    {
        "customer_id": np.arange(1, N_CUSTOMERS + 1),
        "treatment": treatment,
        "conversion": conversion,
        "new_customer": new_customer,
        "customer_segment": np.where(new_customer == 1, "New", "Loyal"),
        "mobile": mobile,
        "prior_purchases": prior_purchases,
        "discount_sensitivity": discount_sensitivity,
        "region_west": region_west,
        "true_uplift": true_uplift,
    }
)

print(df.shape)
print(df.head())

What Does the Table Contain?

  • df stores one row per synthetic customer.
  • customer_segment labels new customers separately from loyal customers.
  • true_uplift remains in the table as hidden ground truth. It remains excluded from FEATURES.
  • The final two lines print the table dimensions and its first five records.
  • Run the completed data-generation cell.
  • Confirm the first output reads (30000, 10).
  • Confirm the preview contains the treatment assignment and conversion outcome columns.
  • Allow targeted_rollout.ipynb to save to Google Drive.

Good progress. Your notebook now holds a reproducible checkout experiment with 30,000 randomized customer records.

Shape Does Not Match?

Confirm that the table dictionary contains all ten columns from customer_id through true_uplift. Check that N_CUSTOMERS still equals 30_000.

Ask for help with finding the DataFrame shape mismatch.

Inspect conversion and balance

Conversion rates show the experiment's overall outcome signal. A bar chart makes the treatment comparison visible before formal inference begins.

  • Add a new code cell beneath the data-generation cell.
  • Paste the conversion summary and chart into the new cell using this code:
# Cell 4: Early visible win and randomization check.
rates = df.groupby("treatment")["conversion"].agg(["count", "sum", "mean"])
print(rates)

plt.figure(figsize=(6, 4))
plt.bar(rates.index, rates["mean"], color=["#6c757d", "#2a9d8f"])
plt.ylabel("Conversion rate")
plt.title("Checkout experiment: overall conversion")
plt.ylim(0, rates["mean"].max() * 1.20)
plt.show()

What Does This Chart Show?

  • rates reports the customer count, conversion count, and conversion mean for each assignment group.
  • The bar height represents each group's observed conversion rate.
  • The fixed colors keep the control and treatment groups visually distinct.

Before you run this cell, which group do you expect to have the taller conversion bar?

  • Run the conversion summary cell.
  • Confirm the output includes one row for control.
  • Confirm the output includes one row for treatment.

You will see a two-bar conversion chart beneath the rates table. The treatment bar is higher because the positive effect among new customers outweighs the negative effect among loyal customers in the aggregate.

  • Allow the chart cell in targeted_rollout.ipynb to save.

Chart Did Not Render?

Confirm that the data-generation cell ran in the current runtime. Check that rates uses the conversion column from df.

Ask for help with debugging the conversion chart.

Random assignment should also produce similar average feature values across groups. The balance table checks that the treatment label did not systematically separate customers by their observed characteristics.

  • Find the plt.show() line in the conversion summary cell.
  • Add the randomization balance check directly below that line using this code:
balance = df.groupby("treatment")[FEATURES].mean().T
balance["absolute_difference"] = (
    balance["treatment"] - balance["control"]
).abs()
print(balance)

How Does the Balance Check Work?

  • The grouped means compare every modeled customer feature across assignment groups.
  • The transpose places each feature on its own row for quick scanning.
  • absolute_difference measures the gap between treatment and control without preserving its direction.

Before you rerun the two cells, do you expect the printed shape or randomized assignments to change?

  • Rerun the completed data-generation cell.
  • Run the completed experiment-summary cell.
  • Allow the completed targeted_rollout.ipynb notebook to save.

The data cell prints (30000, 10) again. The fixed seed reproduces the same first rows.

The summary cell displays a two-bar conversion chart. Its final table shows closely matched treatment and control feature means with small absolute differences.

Balance Table Is Missing?

Confirm that the balance code sits below plt.show() in the same cell. Check that FEATURES contains the five observable feature names.

Ask for help with debugging the randomization balance check.

The comparison below lets you check every notebook cell created so far against the cumulative project code.

✔️ Awesome, I've got everything!

Your notebook now contains the version check, analysis foundation, randomized experiment, conversion chart, and balance table.

ⓧ I'd like to double check the full code

# Cell 1: Verify the environment after running the pinned install command.
from importlib.metadata import version

PACKAGES = [
    "causalml",
    "statsmodels",
    "scikit-learn",
    "pandas",
    "numpy",
    "matplotlib",
]

for package in PACKAGES:
    print(package, version(package))

# Cell 2: Imports and project constants.
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from causalml.inference.tree import UpliftRandomForestClassifier
from causalml.metrics import plot_qini, qini_score
from sklearn.model_selection import train_test_split
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import (
    confint_proportions_2indep,
    proportion_effectsize,
    test_proportions_2indep,
)

SEED = 20261013
N_CUSTOMERS = 30_000
FEATURES = [
    "new_customer",
    "mobile",
    "prior_purchases",
    "discount_sensitivity",
    "region_west",
]

# Cell 3: Generate a randomized checkout experiment.
rng = np.random.default_rng(SEED)
new_customer = rng.binomial(1, 0.40, size=N_CUSTOMERS)
mobile = rng.binomial(1, 0.60, size=N_CUSTOMERS)
prior_purchases = np.where(
    new_customer == 1,
    rng.poisson(0.6, size=N_CUSTOMERS),
    rng.poisson(6.0, size=N_CUSTOMERS),
)
discount_sensitivity = rng.beta(2.0, 3.0, size=N_CUSTOMERS)
region_west = rng.binomial(1, 0.30, size=N_CUSTOMERS)
treated = rng.binomial(1, 0.50, size=N_CUSTOMERS)
treatment = np.where(treated == 1, "treatment", "control")

baseline_logit = (
    -2.0
    - 0.35 * new_customer
    + 0.18 * mobile
    + 0.05 * np.minimum(prior_purchases, 10)
    - 0.25 * discount_sensitivity
    + 0.10 * region_west
)
baseline_probability = 1 / (1 + np.exp(-baseline_logit))
true_uplift = np.where(new_customer == 1, 0.06, -0.02)
true_uplift = true_uplift + 0.01 * mobile * new_customer
conversion_probability = np.clip(
    baseline_probability + treated * true_uplift,
    0.01,
    0.95,
)
conversion = rng.binomial(1, conversion_probability, size=N_CUSTOMERS)

df = pd.DataFrame(
    {
        "customer_id": np.arange(1, N_CUSTOMERS + 1),
        "treatment": treatment,
        "conversion": conversion,
        "new_customer": new_customer,
        "customer_segment": np.where(new_customer == 1, "New", "Loyal"),
        "mobile": mobile,
        "prior_purchases": prior_purchases,
        "discount_sensitivity": discount_sensitivity,
        "region_west": region_west,
        "true_uplift": true_uplift,
    }
)

print(df.shape)
print(df.head())

# Cell 4: Early visible win and randomization check.
rates = df.groupby("treatment")["conversion"].agg(["count", "sum", "mean"])
print(rates)

plt.figure(figsize=(6, 4))
plt.bar(rates.index, rates["mean"], color=["#6c757d", "#2a9d8f"])
plt.ylabel("Conversion rate")
plt.title("Checkout experiment: overall conversion")
plt.ylim(0, rates["mean"].max() * 1.20)
plt.show()

balance = df.groupby("treatment")[FEATURES].mean().T
balance["absolute_difference"] = (
    balance["treatment"] - balance["control"]
).abs()
print(balance)

How to Use This Reference

Each numbered cell comment marks one notebook cell. Compare the reference with targeted_rollout.ipynb from top to bottom.

Your reproducible checkout experiment now exposes an aggregate conversion signal with balanced randomized groups. Next, you will quantify that signal with lift, uncertainty, significance, and statistical power.

Make the Aggregate Rollout Call

The treatment and control chart in Google Colab suggests that the checkout treatment improves average conversion.

A rollout decision needs more than two bar heights. You will measure the lift from the A/B test before checking its uncertainty and sensitivity.

In this step, get ready to:
  • Calculate the overall treatment and control conversion rates.
  • Quantify lift with a score confidence interval and a two-sided hypothesis test.
  • Judge whether the aggregate evidence supports a broad rollout.
Build the A/B summary function

The first part of ab_summary() separates the randomized groups before calculating their conversion rates. It also gives statsmodels the success counts needed for a score test.

  • Add a code cell beneath the balance output in the notebook from earlier.
  • Start the reusable summary function by pasting this first section into the new cell:
# Cell 5: Conventional A/B inference.
def ab_summary(frame):
    treated_rows = frame[frame["treatment"] == "treatment"]
    control_rows = frame[frame["treatment"] == "control"]

    treatment_successes = int(treated_rows["conversion"].sum())
    control_successes = int(control_rows["conversion"].sum())
    treatment_n = len(treated_rows)
    control_n = len(control_rows)
    treatment_rate = treatment_successes / treatment_n
    control_rate = control_successes / control_n

    test_result = test_proportions_2indep(
        count1=treatment_successes,
        nobs1=treatment_n,
        count2=control_successes,
        nobs2=control_n,
        method="score",
        compare="diff",
        alternative="two-sided",
        correction=True,
    )

How Does the Test Work?

  • The group filters isolate the observed outcomes for treatment and control.
  • The success counts record how many customers converted in each group.
  • The sample sizes turn those counts into treatment and control conversion rates.
  • The score test asks whether the observed rate difference is distinguishable from zero.
  • Keep the cursor directly below the closing parenthesis in the same code cell.
  • Complete the function and print its first summary by pasting this code below the first section:
    ci_low, ci_high = confint_proportions_2indep(
        count1=treatment_successes,
        nobs1=treatment_n,
        count2=control_successes,
        nobs2=control_n,
        method="score",
        compare="diff",
        alpha=0.05,
        correction=True,
    )

    return pd.Series(
        {
            "control_rate": control_rate,
            "treatment_rate": treatment_rate,
            "absolute_lift": treatment_rate - control_rate,
            "relative_lift": treatment_rate / control_rate - 1,
            "ci_low": ci_low,
            "ci_high": ci_high,
            "p_value": test_result.pvalue,
            "treatment_n": treatment_n,
            "control_n": control_n,
        }
    )


overall = ab_summary(df)

print(overall.to_frame("overall_experiment").T)

What Does the Summary Return?

  • The confidence interval describes a range of treatment effects compatible with the randomized result.
  • The p_value comes from a two-sided hypothesis test against zero difference.
  • The absolute lift measures the conversion-rate difference in percentage-point terms.
  • The relative lift scales the same difference against the control conversion rate.

Before you run this cell, do you think the uncertainty measures will support the improvement suggested by the chart?

  • Run the completed code cell using its run control.

You should see one row containing both conversion rates, both lift measures, the confidence interval, the p-value, and both sample sizes.

Good progress. Your bar chart now has a statistical summary that measures the size and uncertainty of the average effect.

The Summary Cell Failed?

Check that the second snippet sits inside the same code cell as the first snippet. Its indented lines must remain inside ab_summary().

Confirm that df still exists by rerunning the earlier data-generation cells if the hosted runtime restarted.

Ask for targeted help with debugging the aggregate A/B summary.

Add power and MDE context

Statistical significance evaluates the observed result. Statistical power asks which effect size this experiment was designed to detect reliably.

The minimum detectable effect translates 80% power into an absolute conversion-rate difference on the control scale.

  • Find this block at the bottom of the current code cell:
overall = ab_summary(df)

print(overall.to_frame("overall_experiment").T)

Why Expand This Block?

The current block prints the observed result without measuring the experiment's detection threshold. The expanded block adds that threshold before printing the final row.

  • Replace the current block with this power calculation:
overall = ab_summary(df)

power_model = NormalIndPower()
ratio = overall["control_n"] / overall["treatment_n"]
required_effect_size = power_model.solve_power(
    effect_size=None,
    nobs1=overall["treatment_n"],
    alpha=0.05,
    power=0.80,
    ratio=ratio,
    alternative="two-sided",
)
control_angle = np.arcsin(np.sqrt(overall["control_rate"]))
detectable_treatment_rate = np.sin(
    control_angle + abs(required_effect_size) / 2
) ** 2
mde_80_power = detectable_treatment_rate - overall["control_rate"]
observed_effect_size = proportion_effectsize(
    overall["treatment_rate"], overall["control_rate"]
)

overall["observed_effect_size"] = observed_effect_size
overall["mde_80_power"] = mde_80_power
print(overall.to_frame("overall_experiment").T)

What Does the Power Calculation Add?

  • The sample-size ratio accounts for small differences between the randomized group sizes.
  • NormalIndPower.solve_power() solves for the standardized effect associated with 80% power.
  • proportion_effectsize() records the standardized size of the observed conversion difference.
  • mde_80_power converts the solved effect back to the control conversion-rate scale.
Make the aggregate rollout call

Before you run the expanded cell, do you think the power calculation will reverse the positive lift or add context to it?

  • Run the expanded aggregate inference cell using its run control.

You should see one row containing absolute_lift, relative_lift, ci_low, ci_high, p_value, and mde_80_power.

  • Compare treatment_rate with control_rate to confirm the direction of the average effect.
  • Read absolute_lift as the conversion-rate difference created by treatment.
  • Check whether the confidence interval crosses zero.
  • Use p_value to assess the evidence against zero average effect.
  • Compare absolute_lift with mde_80_power to judge the observed effect against the experiment's detection threshold.

The Intended Shortfall

The positive aggregate evidence makes a broad rollout look defensible. The result summarises the treatment effect across all 30,000 customers.

Every field describes an average. This row cannot reveal whether one customer group benefits while another is harmed.

✔️ Awesome, I've got everything!

Your aggregate inference cell is complete. Make sure the latest output row remains visible in the notebook.

ⓧ I'd like to double check the full code

# Cell 1: Verify the environment after running the pinned install command.
from importlib.metadata import version

PACKAGES = [
    "causalml",
    "statsmodels",
    "scikit-learn",
    "pandas",
    "numpy",
    "matplotlib",
]

for package in PACKAGES:
    print(package, version(package))

# Cell 2: Imports and project constants.
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from causalml.inference.tree import UpliftRandomForestClassifier
from causalml.metrics import plot_qini, qini_score
from sklearn.model_selection import train_test_split
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import (
    confint_proportions_2indep,
    proportion_effectsize,
    test_proportions_2indep,
)

SEED = 20261013
N_CUSTOMERS = 30_000
FEATURES = [
    "new_customer",
    "mobile",
    "prior_purchases",
    "discount_sensitivity",
    "region_west",
]

# Cell 3: Generate a randomized checkout experiment.
rng = np.random.default_rng(SEED)
new_customer = rng.binomial(1, 0.40, size=N_CUSTOMERS)
mobile = rng.binomial(1, 0.60, size=N_CUSTOMERS)
prior_purchases = np.where(
    new_customer == 1,
    rng.poisson(0.6, size=N_CUSTOMERS),
    rng.poisson(6.0, size=N_CUSTOMERS),
)
discount_sensitivity = rng.beta(2.0, 3.0, size=N_CUSTOMERS)
region_west = rng.binomial(1, 0.30, size=N_CUSTOMERS)
treated = rng.binomial(1, 0.50, size=N_CUSTOMERS)
treatment = np.where(treated == 1, "treatment", "control")

baseline_logit = (
    -2.0
    - 0.35 * new_customer
    + 0.18 * mobile
    + 0.05 * np.minimum(prior_purchases, 10)
    - 0.25 * discount_sensitivity
    + 0.10 * region_west
)
baseline_probability = 1 / (1 + np.exp(-baseline_logit))
true_uplift = np.where(new_customer == 1, 0.06, -0.02)
true_uplift = true_uplift + 0.01 * mobile * new_customer
conversion_probability = np.clip(
    baseline_probability + treated * true_uplift,
    0.01,
    0.95,
)
conversion = rng.binomial(1, conversion_probability, size=N_CUSTOMERS)

df = pd.DataFrame(
    {
        "customer_id": np.arange(1, N_CUSTOMERS + 1),
        "treatment": treatment,
        "conversion": conversion,
        "new_customer": new_customer,
        "customer_segment": np.where(new_customer == 1, "New", "Loyal"),
        "mobile": mobile,
        "prior_purchases": prior_purchases,
        "discount_sensitivity": discount_sensitivity,
        "region_west": region_west,
        "true_uplift": true_uplift,
    }
)

print(df.shape)
print(df.head())

# Cell 4: Early visible win and randomization check.
rates = df.groupby("treatment")["conversion"].agg(["count", "sum", "mean"])
print(rates)

plt.figure(figsize=(6, 4))
plt.bar(rates.index, rates["mean"], color=["#6c757d", "#2a9d8f"])
plt.ylabel("Conversion rate")
plt.title("Checkout experiment: overall conversion")
plt.ylim(0, rates["mean"].max() * 1.20)
plt.show()

balance = df.groupby("treatment")[FEATURES].mean().T
balance["absolute_difference"] = (
    balance["treatment"] - balance["control"]
).abs()
print(balance)

# Cell 5: Conventional A/B inference.
def ab_summary(frame):
    treated_rows = frame[frame["treatment"] == "treatment"]
    control_rows = frame[frame["treatment"] == "control"]

    treatment_successes = int(treated_rows["conversion"].sum())
    control_successes = int(control_rows["conversion"].sum())
    treatment_n = len(treated_rows)
    control_n = len(control_rows)
    treatment_rate = treatment_successes / treatment_n
    control_rate = control_successes / control_n

    test_result = test_proportions_2indep(
        count1=treatment_successes,
        nobs1=treatment_n,
        count2=control_successes,
        nobs2=control_n,
        method="score",
        compare="diff",
        alternative="two-sided",
        correction=True,
    )
    ci_low, ci_high = confint_proportions_2indep(
        count1=treatment_successes,
        nobs1=treatment_n,
        count2=control_successes,
        nobs2=control_n,
        method="score",
        compare="diff",
        alpha=0.05,
        correction=True,
    )

    return pd.Series(
        {
            "control_rate": control_rate,
            "treatment_rate": treatment_rate,
            "absolute_lift": treatment_rate - control_rate,
            "relative_lift": treatment_rate / control_rate - 1,
            "ci_low": ci_low,
            "ci_high": ci_high,
            "p_value": test_result.pvalue,
            "treatment_n": treatment_n,
            "control_n": control_n,
        }
    )


overall = ab_summary(df)

power_model = NormalIndPower()
ratio = overall["control_n"] / overall["treatment_n"]
required_effect_size = power_model.solve_power(
    effect_size=None,
    nobs1=overall["treatment_n"],
    alpha=0.05,
    power=0.80,
    ratio=ratio,
    alternative="two-sided",
)
control_angle = np.arcsin(np.sqrt(overall["control_rate"]))
detectable_treatment_rate = np.sin(
    control_angle + abs(required_effect_size) / 2
) ** 2
mde_80_power = detectable_treatment_rate - overall["control_rate"]
observed_effect_size = proportion_effectsize(
    overall["treatment_rate"], overall["control_rate"]
)

overall["observed_effect_size"] = observed_effect_size
overall["mde_80_power"] = mde_80_power
print(overall.to_frame("overall_experiment").T)

This reference combines the unchanged environment, experiment, balance check, and completed aggregate inference cell.

The Final Row Did Not Print?

Confirm that the power block replaced the earlier two-line block at the bottom of the same cell. A duplicate print line can leave the notebook showing an outdated row first.

Rerun the earlier cells if the runtime no longer contains df or the imported statsmodels functions.

Ask for help with debugging the power and MDE calculation.

Your aggregate evidence now supports an initial rollout call. Next, you will test whether that average conceals opposite effects across customer groups.

Find the Hidden Customer Split

Your Google Colab analysis made a broad rollout look reasonable. Now you will test whether that average holds across customer tenure.

Segment-level inference applies the same score-based confidence interval to smaller customer groups. It keeps the inference method constant while the customer frames change.

In this step, get ready to:
  • Apply ab_summary() separately to New customers and Loyal customers.
  • Plot each segment's absolute lift with a 95% confidence interval.
  • Record the rollout decision supported by the segment evidence.
Calculate lift for each customer segment

The existing ab_summary() function gives every segment the same statistical treatment. This makes the resulting lift estimates directly comparable.

  • Use the notebook's add-cell control to insert a code cell directly below the aggregate inference cell.
  • Build the segment table by pasting this code into the new cell:
# Cell 6: Segment analysis reveals the aggregate limitation.
segment_results = pd.DataFrame(
    {
        segment: ab_summary(segment_frame)
        for segment, segment_frame in df.groupby("customer_segment", sort=False)
    }
).T
print(segment_results)

What Does This Code Do?

  • The groupby() operation separates the experiment by customer_segment.
  • The dictionary comprehension sends each segment frame into ab_summary().
  • The transpose places New customers and Loyal customers on separate rows.
  • The printed table includes each segment's lift estimate, interval, and p-value.
  • Save targeted_rollout.ipynb using the notebook's save control.

Before you run this cell, do you expect both customer segments to support the aggregate rollout decision?

  • Select the run control on the left side of the new cell.

The New row should show positive absolute_lift. The Loyal row should show negative absolute_lift.

You uncovered opposite checkout effects for new customers and loyal customers.

Segment Table Not Appearing?

Confirm that the earlier experiment cells ran in the current hosted runtime. The df data frame must contain customer_segment before this cell can group the rows.

Confirm that the aggregate cell defined ab_summary() before you run the segment cell.

Ask for help with diagnosing the segment table.

Plot the segment confidence intervals

The table carries the exact estimates. A Matplotlib chart makes the direction of each effect visible against a zero-lift reference line.

  • Use the notebook's add-cell control to insert another code cell below the segment table.
  • Build the segment lift chart by pasting this code into the new cell:
segment_order = ["New", "Loyal"]
segment_plot = segment_results.loc[segment_order]
lower_error = segment_plot["absolute_lift"] - segment_plot["ci_low"]
upper_error = segment_plot["ci_high"] - segment_plot["absolute_lift"]

plt.figure(figsize=(7, 4))
plt.bar(
    segment_order,
    segment_plot["absolute_lift"],
    yerr=np.vstack([lower_error, upper_error]),
    capsize=6,
    color=["#2a9d8f", "#e76f51"],
)
plt.axhline(0, color="black", linewidth=1)
plt.ylabel("Absolute conversion lift")
plt.title("The average hides opposite customer effects")
plt.show()

What Does This Code Do?

  • The segment_order list fixes the chart order as New followed by Loyal.
  • The lower_error and upper_error values convert each confidence interval into bar-chart error lengths.
  • The horizontal line marks zero absolute lift. Bars on opposite sides of that line imply different rollout consequences.
  • Save targeted_rollout.ipynb using the notebook's save control.

Before you run this cell, where do you expect the two bars to sit relative to the zero line?

  • Select the run control on the left side of the chart cell.

You should see the New bar above zero. You should see the Loyal bar below zero.

Each bar should include its 95% confidence-interval error bars. The chart now exposes the limitation of the aggregate result.

Chart Missing or Misaligned?

Confirm that the segment table cell completed successfully. The chart depends on segment_results from that cell.

Check that segment_order contains the exact labels stored in customer_segment.

Ask for help with fixing the segment lift chart.

✔️ Awesome, I've got everything!

Your notebook now contains the segment inference table and the confidence-interval chart.

ⓧ I'd like to double check the full code

  • Compare the first six cells in targeted_rollout.ipynb with this cumulative reference:
# Cell 1: Verify the environment after running the pinned install command.
from importlib.metadata import version

PACKAGES = [
    "causalml",
    "statsmodels",
    "scikit-learn",
    "pandas",
    "numpy",
    "matplotlib",
]

for package in PACKAGES:
    print(package, version(package))

# Cell 2: Imports and project constants.
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from causalml.inference.tree import UpliftRandomForestClassifier
from causalml.metrics import plot_qini, qini_score
from sklearn.model_selection import train_test_split
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import (
    confint_proportions_2indep,
    proportion_effectsize,
    test_proportions_2indep,
)

SEED = 20261013
N_CUSTOMERS = 30_000
FEATURES = [
    "new_customer",
    "mobile",
    "prior_purchases",
    "discount_sensitivity",
    "region_west",
]

# Cell 3: Generate a randomized checkout experiment.
rng = np.random.default_rng(SEED)
new_customer = rng.binomial(1, 0.40, size=N_CUSTOMERS)
mobile = rng.binomial(1, 0.60, size=N_CUSTOMERS)
prior_purchases = np.where(
    new_customer == 1,
    rng.poisson(0.6, size=N_CUSTOMERS),
    rng.poisson(6.0, size=N_CUSTOMERS),
)
discount_sensitivity = rng.beta(2.0, 3.0, size=N_CUSTOMERS)
region_west = rng.binomial(1, 0.30, size=N_CUSTOMERS)
treated = rng.binomial(1, 0.50, size=N_CUSTOMERS)
treatment = np.where(treated == 1, "treatment", "control")

baseline_logit = (
    -2.0
    - 0.35 * new_customer
    + 0.18 * mobile
    + 0.05 * np.minimum(prior_purchases, 10)
    - 0.25 * discount_sensitivity
    + 0.10 * region_west
)
baseline_probability = 1 / (1 + np.exp(-baseline_logit))
true_uplift = np.where(new_customer == 1, 0.06, -0.02)
true_uplift = true_uplift + 0.01 * mobile * new_customer
conversion_probability = np.clip(
    baseline_probability + treated * true_uplift,
    0.01,
    0.95,
)
conversion = rng.binomial(1, conversion_probability, size=N_CUSTOMERS)

df = pd.DataFrame(
    {
        "customer_id": np.arange(1, N_CUSTOMERS + 1),
        "treatment": treatment,
        "conversion": conversion,
        "new_customer": new_customer,
        "customer_segment": np.where(new_customer == 1, "New", "Loyal"),
        "mobile": mobile,
        "prior_purchases": prior_purchases,
        "discount_sensitivity": discount_sensitivity,
        "region_west": region_west,
        "true_uplift": true_uplift,
    }
)

print(df.shape)
print(df.head())

# Cell 4: Early visible win and randomization check.
rates = df.groupby("treatment")["conversion"].agg(["count", "sum", "mean"])
print(rates)

plt.figure(figsize=(6, 4))
plt.bar(rates.index, rates["mean"], color=["#6c757d", "#2a9d8f"])
plt.ylabel("Conversion rate")
plt.title("Checkout experiment: overall conversion")
plt.ylim(0, rates["mean"].max() * 1.20)
plt.show()

balance = df.groupby("treatment")[FEATURES].mean().T
balance["absolute_difference"] = (
    balance["treatment"] - balance["control"]
).abs()
print(balance)

# Cell 5: Conventional A/B inference.
def ab_summary(frame):
    treated_rows = frame[frame["treatment"] == "treatment"]
    control_rows = frame[frame["treatment"] == "control"]

    treatment_successes = int(treated_rows["conversion"].sum())
    control_successes = int(control_rows["conversion"].sum())
    treatment_n = len(treated_rows)
    control_n = len(control_rows)
    treatment_rate = treatment_successes / treatment_n
    control_rate = control_successes / control_n

    test_result = test_proportions_2indep(
        count1=treatment_successes,
        nobs1=treatment_n,
        count2=control_successes,
        nobs2=control_n,
        method="score",
        compare="diff",
        alternative="two-sided",
        correction=True,
    )
    ci_low, ci_high = confint_proportions_2indep(
        count1=treatment_successes,
        nobs1=treatment_n,
        count2=control_successes,
        nobs2=control_n,
        method="score",
        compare="diff",
        alpha=0.05,
        correction=True,
    )

    return pd.Series(
        {
            "control_rate": control_rate,
            "treatment_rate": treatment_rate,
            "absolute_lift": treatment_rate - control_rate,
            "relative_lift": treatment_rate / control_rate - 1,
            "ci_low": ci_low,
            "ci_high": ci_high,
            "p_value": test_result.pvalue,
            "treatment_n": treatment_n,
            "control_n": control_n,
        }
    )


overall = ab_summary(df)

power_model = NormalIndPower()
ratio = overall["control_n"] / overall["treatment_n"]
required_effect_size = power_model.solve_power(
    effect_size=None,
    nobs1=overall["treatment_n"],
    alpha=0.05,
    power=0.80,
    ratio=ratio,
    alternative="two-sided",
)
control_angle = np.arcsin(np.sqrt(overall["control_rate"]))
detectable_treatment_rate = np.sin(
    control_angle + abs(required_effect_size) / 2
) ** 2
mde_80_power = detectable_treatment_rate - overall["control_rate"]
observed_effect_size = proportion_effectsize(
    overall["treatment_rate"], overall["control_rate"]
)

overall["observed_effect_size"] = observed_effect_size
overall["mde_80_power"] = mde_80_power
print(overall.to_frame("overall_experiment").T)

# Cell 6: Segment analysis reveals the aggregate limitation.
segment_results = pd.DataFrame(
    {
        segment: ab_summary(segment_frame)
        for segment, segment_frame in df.groupby("customer_segment", sort=False)
    }
).T
print(segment_results)

segment_order = ["New", "Loyal"]
segment_plot = segment_results.loc[segment_order]
lower_error = segment_plot["absolute_lift"] - segment_plot["ci_low"]
upper_error = segment_plot["ci_high"] - segment_plot["absolute_lift"]

plt.figure(figsize=(7, 4))
plt.bar(
    segment_order,
    segment_plot["absolute_lift"],
    yerr=np.vstack([lower_error, upper_error]),
    capsize=6,
    color=["#2a9d8f", "#e76f51"],
)
plt.axhline(0, color="black", linewidth=1)
plt.ylabel("Absolute conversion lift")
plt.title("The average hides opposite customer effects")
plt.show()
Update the rollout decision

The segment evidence rules out a universal rollout. The checkout treatment should not ship to everyone.

The fixed tenure split demonstrates heterogeneous treatment effects using one customer feature. The FEATURES list contains combinations that a two-segment rule may miss.

  • Compare the two segment directions with the aggregate absolute_lift from the previous step.
  • Use the checkpoint below to record the rollout decision supported by the chart.

Your segment evidence has stopped the full rollout. Next, you will train an uplift model to rank richer combinations of customer features on held-out data.

Train and Validate an Uplift Policy

Your segment analysis exposed opposite checkout effects. The fixed tenure rule proves that customer response varies.

The next decision needs a CausalML uplift forest to rank combinations represented in FEATURES. It must earn trust on untouched rows from the randomized experiment before it can guide a targeted rollout.

In this step, get ready to:
  • Fit an uplift forest using a training split.
  • Evaluate customer ranking with a held-out Qini curve.
  • Compare four rollout policies with held-out policy value.
Train the uplift forest on a holdout split

An uplift forest estimates how treatment changes conversion for different feature combinations. Using separate splits keeps the final ranking test independent from model fitting.

  • Create Cell 7 below the segment analysis by pasting this split and model configuration:
# Cell 7: Train the uplift model on one split and score only the holdout.
train, test = train_test_split(
    df,
    test_size=0.30,
    random_state=111,
    stratify=df["treatment"],
)
train = train.copy()
test = test.copy()

uplift_model = UpliftRandomForestClassifier(
    control_name="control",
    n_estimators=30,
    max_features=5,
    random_state=42,
    max_depth=4,
    min_samples_leaf=200,
    min_samples_treatment=100,
    n_reg=100,
    evaluationFunction="KL",
    normalization=True,
    honesty=True,
    estimation_sample_size=0.50,
    n_jobs=None,
)

What Does This Code Do?

  • The split reserves 30% of the randomized rows for testing.
  • Treatment stratification preserves the experiment allocation across both splits.
  • The copied DataFrames let the notebook add predictions without changing df.
  • The forest configuration limits tree complexity while preserving treatment-specific splits.
  • Complete Cell 7 by adding this fitting and prediction code below the model configuration:
uplift_model.fit(
    X=train[FEATURES].to_numpy(),
    treatment=train["treatment"].to_numpy(),
    y=train["conversion"].to_numpy(),
)
test["uplift_score"] = uplift_model.predict(
    test[FEATURES].to_numpy()
).ravel()

print(
    test.groupby("customer_segment")["uplift_score"].agg(
        ["count", "mean", "median"]
    )
)

How Is the Holdout Protected?

  • The fit() call receives only rows from train.
  • The predict() call assigns treatment deltas only to test.
  • The segment summary provides a quick check on whether the predicted rankings retain the direction revealed earlier.

Before you run Cell 7, do you expect the two customer segments to receive similar uplift-score distributions?

  • Run Cell 7 using the play button beside the cell.

You'll see one row for each customer segment. The output reports count, mean and median uplift scores.

That is the hardest fitting work complete. Your model now scores only customers it did not train on.

Model Cell Did Not Finish?

Confirm that the earlier imports cell ran in the current hosted runtime. Check that df still contains the experiment rows.

Compare the fit() keyword names with the code above. Positional treatment inputs do not match this project.

Ask for targeted help with diagnosing the uplift forest cell.

Evaluate the held-out Qini curve

A Qini curve tests whether higher model scores correspond to more useful treatment targeting. The held-out comparison checks ranking quality against random targeting.

  • Create Cell 8 below the model output by pasting this held-out evaluation:
# Cell 8: Evaluate ranking quality on untouched randomized data.
qini_data = pd.DataFrame(
    {
        "conversion": test["conversion"].to_numpy(),
        "is_treated": (test["treatment"] == "treatment").astype(int).to_numpy(),
        "uplift_score": test["uplift_score"].to_numpy(),
    }
)
qini = qini_score(
    qini_data,
    outcome_col="conversion",
    treatment_col="is_treated",
    normalize=False,
)
print("Held-out Qini score:", qini["uplift_score"])

plot_qini(
    qini_data,
    outcome_col="conversion",
    treatment_col="is_treated",
    normalize=False,
    figsize=(7, 5),
)
plt.title("Held-out Qini curve")
plt.show()

What Does This Evaluation Measure?

  • The qini_data DataFrame combines observed conversion with randomized treatment assignment.
  • The uplift_score column supplies the model ranking produced on held-out customers.
  • The qini_score() result measures the area between the model curve and random targeting.
  • The plot_qini() chart makes the ranking advantage visible across increasing target shares.

Before you run Cell 8, do you expect model-ranked targeting to outperform random targeting across the strongest-ranked customers?

  • Run Cell 8 using the play button beside the cell.

You'll see a printed held-out Qini score. You'll also see the model curve compared with random targeting.

A curve above the random benchmark over useful target shares shows that the forest ranks responsive customers better than chance. The held-out score keeps that evidence separate from training.

Qini Curve Missing?

Rerun Cell 7 before Cell 8 so test contains uplift_score.

Confirm that qini_data includes the three columns shown above. The treatment indicator must contain numeric zeroes and ones.

Ask for help with diagnosing the held-out Qini evaluation.

Compare rollout policies on held-out customers

A useful ranking still needs a rollout rule. Inverse-probability evaluation estimates each policy from randomized rows whose observed treatment matches that policy's chosen action.

  • Create Cell 9 below the Qini chart by pasting this policy evaluator and policy set:
# Cell 9: Compare rollout policies with inverse-probability evaluation.
def policy_value(frame, target_actions, assignment_probability=0.50):
    observed_actions = (frame["treatment"] == "treatment").astype(int).to_numpy()
    target_actions = np.asarray(target_actions, dtype=int)
    matched = observed_actions == target_actions
    weighted_outcomes = (
        frame["conversion"].to_numpy() * matched / assignment_probability
    )
    return weighted_outcomes.mean()


policies = {
    "No rollout": np.zeros(len(test), dtype=int),
    "Full rollout": np.ones(len(test), dtype=int),
    "New customers only": test["new_customer"].to_numpy(),
    "Positive predicted uplift": (test["uplift_score"] > 0).astype(int).to_numpy(),
}

How Does Policy Value Work?

  • The function checks whether each randomized action matches the action proposed by a policy.
  • Matched outcomes receive an inverse-probability weight based on the 0.50 assignment probability.
  • The four policies represent no treatment, universal treatment, fixed segment targeting and model targeting.
  • Complete the first part of Cell 9 by adding this comparison table below policies:
policy_table = pd.DataFrame(
    {
        "estimated_conversion": {
            name: policy_value(test, actions) for name, actions in policies.items()
        },
        "target_share": {
            name: np.mean(actions) for name, actions in policies.items()
        },
    }
)
policy_table["incremental_vs_control_pp"] = 100 * (
    policy_table["estimated_conversion"]
    - policy_table.loc["No rollout", "estimated_conversion"]
)
policy_table = policy_table.sort_values("estimated_conversion", ascending=False)
print(policy_table)

What Will the Table Compare?

  • The estimated_conversion column reports held-out policy value.
  • The target_share column reports the fraction of customers selected for treatment.
  • The incremental_vs_control_pp column expresses each estimated gain in percentage points above no rollout.
  • Sorting places the highest-valued held-out policy at the top.

Before you run the comparison, which policy do you expect to balance conversion gain with a selective target share?

  • Run Cell 9 using the play button beside the cell.

You'll see four rows named No rollout, Full rollout, New customers only and Positive predicted uplift.

The first row identifies the strongest estimated conversion policy on the untouched test set. Its target share shows how selective that rollout would be.

Policy Table Looks Incomplete?

Confirm that all four entries remain inside the policies dictionary. Check that the comparison uses test instead of train.

Rerun Cell 8 if the model-targeting row cannot access uplift_score.

Ask for help with diagnosing the policy comparison.

The table supports a decision with measured outcomes. The final memo packages that evidence for an executive audience.

  • Finish Cell 9 by adding this memo code below the policy table:
best_policy = policy_table["estimated_conversion"].idxmax()
target_share = policy_table.loc[best_policy, "target_share"]
loyal_lift = segment_results.loc["Loyal", "absolute_lift"]
new_lift = segment_results.loc["New", "absolute_lift"]

memo = f"""
EXECUTIVE ROLLOUT MEMO
Decision: Use the highest-valued held-out policy, currently '{best_policy}'.
Evidence: Overall lift is {overall['absolute_lift']:.2%}, but segment effects diverge.
New-customer lift: {new_lift:.2%}.
Loyal-customer lift: {loyal_lift:.2%}.
Held-out Qini score: {qini['uplift_score']:.3f}.
Recommended target share: {target_share:.1%}.
Guardrail: Validate this policy in a fresh randomized rollout before production use.
"""
print(memo)

What Does the Memo Capture?

  • The memo names the policy with the highest held-out estimated conversion.
  • It contrasts the overall result with the New and Loyal segment lifts.
  • It includes the held-out Qini score plus the recommended target share.
  • The guardrail requires a fresh randomized rollout before production use.

Before you rerun Cell 9, which pieces of evidence should stop the recommendation from becoming a blanket rollout?

  • Rerun Cell 9 using the play button beside the cell.

You'll see the policy table followed by EXECUTIVE ROLLOUT MEMO. The recommendation names the highest-valued held-out policy.

The memo ends with a fresh randomized-rollout guardrail. You have turned an average A/B result into a validated targeting recommendation.

Memo Did Not Print?

Confirm that the policy table printed successfully before the memo code runs. The memo depends on policy_table, segment_results and qini.

Check that the triple quotes surround the entire memo. Keep each formatted value inside its matching braces.

Ask for help with diagnosing the executive memo.

✔️ Awesome, I've got everything!

Great. Your three new cells now match the complete held-out modeling reference.

ⓧ I'd like to double check the full code

# Cells 1 through 6 remain exactly as previously created.

# Cell 7: Train the uplift model on one split and score only the holdout.
train, test = train_test_split(
    df,
    test_size=0.30,
    random_state=111,
    stratify=df["treatment"],
)
train = train.copy()
test = test.copy()

uplift_model = UpliftRandomForestClassifier(
    control_name="control",
    n_estimators=30,
    max_features=5,
    random_state=42,
    max_depth=4,
    min_samples_leaf=200,
    min_samples_treatment=100,
    n_reg=100,
    evaluationFunction="KL",
    normalization=True,
    honesty=True,
    estimation_sample_size=0.50,
    n_jobs=None,
)
uplift_model.fit(
    X=train[FEATURES].to_numpy(),
    treatment=train["treatment"].to_numpy(),
    y=train["conversion"].to_numpy(),
)
test["uplift_score"] = uplift_model.predict(
    test[FEATURES].to_numpy()
).ravel()

print(
    test.groupby("customer_segment")["uplift_score"].agg(
        ["count", "mean", "median"]
    )
)

# Cell 8: Evaluate ranking quality on untouched randomized data.
qini_data = pd.DataFrame(
    {
        "conversion": test["conversion"].to_numpy(),
        "is_treated": (test["treatment"] == "treatment").astype(int).to_numpy(),
        "uplift_score": test["uplift_score"].to_numpy(),
    }
)
qini = qini_score(
    qini_data,
    outcome_col="conversion",
    treatment_col="is_treated",
    normalize=False,
)
print("Held-out Qini score:", qini["uplift_score"])

plot_qini(
    qini_data,
    outcome_col="conversion",
    treatment_col="is_treated",
    normalize=False,
    figsize=(7, 5),
)
plt.title("Held-out Qini curve")
plt.show()

# Cell 9: Compare rollout policies with inverse-probability evaluation.
def policy_value(frame, target_actions, assignment_probability=0.50):
    observed_actions = (frame["treatment"] == "treatment").astype(int).to_numpy()
    target_actions = np.asarray(target_actions, dtype=int)
    matched = observed_actions == target_actions
    weighted_outcomes = (
        frame["conversion"].to_numpy() * matched / assignment_probability
    )
    return weighted_outcomes.mean()


policies = {
    "No rollout": np.zeros(len(test), dtype=int),
    "Full rollout": np.ones(len(test), dtype=int),
    "New customers only": test["new_customer"].to_numpy(),
    "Positive predicted uplift": (test["uplift_score"] > 0).astype(int).to_numpy(),
}

policy_table = pd.DataFrame(
    {
        "estimated_conversion": {
            name: policy_value(test, actions) for name, actions in policies.items()
        },
        "target_share": {
            name: np.mean(actions) for name, actions in policies.items()
        },
    }
)
policy_table["incremental_vs_control_pp"] = 100 * (
    policy_table["estimated_conversion"]
    - policy_table.loc["No rollout", "estimated_conversion"]
)
policy_table = policy_table.sort_values("estimated_conversion", ascending=False)
print(policy_table)

best_policy = policy_table["estimated_conversion"].idxmax()
target_share = policy_table.loc[best_policy, "target_share"]
loyal_lift = segment_results.loc["Loyal", "absolute_lift"]
new_lift = segment_results.loc["New", "absolute_lift"]

memo = f"""
EXECUTIVE ROLLOUT MEMO
Decision: Use the highest-valued held-out policy, currently '{best_policy}'.
Evidence: Overall lift is {overall['absolute_lift']:.2%}, but segment effects diverge.
New-customer lift: {new_lift:.2%}.
Loyal-customer lift: {loyal_lift:.2%}.
Held-out Qini score: {qini['uplift_score']:.3f}.
Recommended target share: {target_share:.1%}.
Guardrail: Validate this policy in a fresh randomized rollout before production use.
"""
print(memo)

Secret mission

Add a Profit Threshold

Replace the zero-uplift policy boundary with a break-even threshold based on conversion value and treatment cost. Compare every held-out policy by estimated net value per customer.

Clean Up Your Resources

Clean Up Your Resources

This project costs $0 in Google Colab's free-of-charge hosted runtime. Your notebook persists in Google Drive until you delete it.

Resources you used:

  • The targeted_rollout.ipynb notebook stored in Google Drive.
  • A temporary Google Colab hosted runtime containing the pinned analysis libraries.

Keep everything running

No action is needed. This option fits if you plan to rerun or share the targeted rollout analysis.

  • Leave targeted_rollout.ipynb in Google Drive.
  • Keep working in the current Colab session while its runtime remains connected.
  • Rerun the pinned installation cell after a future runtime deletion or recycle.

Your notebook remains available from Google Drive. Its experiment code can rebuild every chart and policy table.

Pause - I'll come back to this later

Release the temporary hosted runtime to end the current session. Your notebook remains stored in Google Drive.

  • Switch back to the Google Colab notebook from earlier.
  • Select Runtime from the notebook menu.
  • Select Disconnect and delete runtime.

The hosted runtime is disconnected. Your targeted_rollout.ipynb notebook remains available in Google Drive.

Delete - I don't want to use this again

This is the only irreversible option. It fits when you no longer want the notebook or its temporary runtime.

  • Return to the Google Colab notebook from earlier.
  • Select Runtime from the notebook menu.
  • Select Disconnect and delete runtime.
  • Go to Google Drive using the Google account from this project.
  • Select targeted_rollout.ipynb in the Drive file list.
  • Use the selected file's removal control to send the notebook to Drive trash.
  • Open Drive trash from the left sidebar.
  • Empty Drive trash.

The notebook no longer appears in Google Drive. The temporary runtime and its installed packages are also gone.

Nice Work!

Nice Work!

You did it! Your Google Colab notebook now turns a randomized checkout experiment into a validated targeting policy.

You've learned how to:

  • Build a reproducible checkout experiment with 30,000 synthetic customers. Summarize absolute lift, relative lift, uncertainty, significance, power, and minimum detectable effect.
  • Expose opposite treatment effects for new customers versus loyal customers. Use segment confidence intervals to challenge the aggregate rollout decision.
  • Train an uplift random forest with CausalML. Validate its customer ranking with a held-out Qini evaluation. Compare rollout policies before producing an executive recommendation.
  • Secret Mission: Add a profit-aware targeting threshold based on the economic break-even point. Compare net value per customer across the original policies plus the profit-aware policy.

Ready to quiz yourself?